Monitoring device for industrial operations

Description

[0001] The invention relates to the field of industrial operations monitoring.

[0002] An industrial operation is defined as any operation on an industrial site that takes place in a constrained environment, and in which a finite and known number of events are expected to occur, defining the "normality" of the operation. For example, this could be a type of material conveyor with a specific caliber, a container that receives material brought to a melting point or very high temperature and must not overflow, an injection system that must move vertically without horizontal travel, crusher jaws that must interlock to reduce the particle size of the introduced elements, an area that workers must not cross during the execution of an action, etc. Any situation outside the normality can be classified as an "incident".

[0003] In the field of industrial operations, several types of controls exist to ensure safety, smooth operation, or to prevent accidents during an industrial process. These controls can be human, automated, or semi-automated (in which case a human monitors a parameter detected automatically).

[0004] Many incidents involving equipment and personnel could be avoided with continuous visual monitoring. With limited human resources, such monitoring is impossible. Automated solutions are therefore the most desirable.

[0005] Automated video surveillance analytics solutions are proven technologies that have existed since the 1990s and are used in industry for compliance and quality control. However, existing automated visual inspection devices generally require configuration to specify what should be detected, or at least to have examples of what should be detected. This is known as a supervised approach to incident detection.

[0006] Unfortunately, in many cases the nature of the incident is not well known, and often even its cause is unknown, or must be identified very early in order to prevent the incident.

[0007] Defining morphological parameters to detect anomalies can therefore prove overly complex, as can searching for example images of what needs to be detected. Conventional video surveillance systems are thus unsuitable for incidents whose nature is difficult to define a priori. For example, if we consider a hopper blocked by branches, how can we configure a video surveillance system to detect the branches?

[0008] The Applicant's research has shown that video stream anomaly detection devices rarely rely on an unsupervised approach.

[0009] Advances in image analysis using deep learning methods allow for the automatic characterization of images and / or video streams, which are processed as a sequence of images. This is the case, for example, with certain sorting or quality control machines. However, these machines still require training on what to detect. Furthermore, they generally require significant computing power, and therefore active cooling of the computing resources that implement them, which is incompatible with industrial environments that are often very dusty.

[0010] The paper Nawaratne Rashmika et al, "Spatiotemporal Anomaly Detection Using Deep Learning for Real-Time Video Surveillance", IEEE Transactions On Industrial Informatics, vol. 16, n. 1, 29 August 2019 (29 August 2019), pages 393-402, describes an intelligent real-time video surveillance system using unsupervised deep learning.

[0011] Therefore, no system provides a satisfactory solution for the automated monitoring of industrial operations.

[0012] The invention improves the situation. To this end, it proposes an industrial operation monitoring system comprising a substantially sealed housing containing an image and / or video acquisition source and a computer. This device is configured to generate monitoring images from the image and / or video acquisition source. The computer is passively cooled and is configured to execute a normality training sequence for a scene at an industrial operation site to be monitored. The output data from this sequence is then used by the industrial operation monitoring system.The computer is arranged, for a given scene of an industrial operation site to be monitored, to implement the normality learning sequence using a pre-trained transfer-style convolutional neural network and surveillance images taken from said given scene and associated with an acquisition duration chosen according to the industrial operation to be monitored, to select as output data of the normality learning sequence reference matrices taken from some at least of the Gram matrices of at least one convolution layer of said neural network obtained after processing all the surveillance images associated with the chosen acquisition duration.The computer is arranged, for a given scene of an industrial operation site to be monitored, to implement the monitoring sequence by receiving monitoring images obtained from said given scene, providing them to said convolutional neural network, and returning a monitoring value indicating a condition of normality or abnormality of said industrial operation depending on the processing of these monitoring images by said convolutional neural network and the output data of the normality learning sequence.

[0013] This device is particularly advantageous because it can be used in any industrial environment. Furthermore, it operates unsupervised, making it suitable for use in many previously unmanageable situations. Finally, since the training environment is the actual industrial operating site being monitored, it is particularly inexpensive to implement and configure, and operates according to the "set it and forget it" paradigm.

[0014] The invention is described in the attached set of claims.

[0015] According to various embodiments, the invention may have one or more of the following characteristics: The computer is arranged to implement the monitoring sequence by retrieving for each monitoring image the Gram matrices produced by the neural network, and to determine the monitoring value by comparing them to the reference matrices using a measurement and comparing the result of this measurement to a normality threshold. The computer is arranged, for a given monitoring image of the monitoring sequence, to implement said measurement by calculating the squared difference between each Gram matrix of the given retrieved monitoring image and the reference matrix to which it corresponds. The computer is arranged to implement a convolutional neural network of the VGG-19 type, in which the Gram matrices are selected from the output of the first convolution layer.

[0016] The invention also relates to a method for monitoring industrial operations on a scene at an industrial operating site comprising the following operations: a) receive surveillance images of said scene, b) implement a normality learning sequence using a pre-trained transfer-style convolutional neural network and surveillance images taken from said scene and associated with an acquisition duration chosen according to the industrial operation to be monitored, and select as output data of the normality learning sequence reference matrices taken from at least some of the Gram matrices of at least one convolutional layer of said neural network obtained after processing all the surveillance images associated with the chosen acquisition duration, c) after operation b), implement a surveillance sequence by receiving new surveillance images obtained from said scene and providing them to said convolutional neural network,and d) return a monitoring value indicating a normality or abnormality condition of said industrial operation based on the processing of new monitoring images by said convolutional neural network and output data from operation b).

[0017] According to various embodiments, the invention may have one or more of the following characteristics: operation c) includes c1) retrieving for each surveillance image the Gram matrices produced by the neural network, and, c2) determining the surveillance value by comparing the matrices of operation c1) to the reference matrices using a measure, and comparing the result of this measure to a normality threshold, operation c2) includes, for a given surveillance image of operation c1), the use of a measure calculating the squared difference between each Gram matrix of the given retrieved surveillance image and the reference matrix to which it corresponds, and the convolutional neural network is of the VGG-19 type, and in which operation b) and c1) selects the Gram matrices at the output of the first convolution layer.

[0018] The invention also relates to a computer program comprising instructions for executing the process according to the invention, a data storage medium on which such a computer program is recorded, and a computer system comprising a processor coupled to a memory, the memory having recorded such a computer program.

[0019] Other features and advantages of the invention will become clearer upon reading the following description, drawn from illustrative and non-limiting examples taken from the drawings shown: there figure 1 Figure 1 represents a generic diagram of a device according to the invention; Figure 2 represents an example of the operating loop of the device. figure 1 , there figure 3 represents an example of the implementation of a neural network used in the device of the figure 1 , there figure 4 represents an implementation of the invention on an industrial operating site scene in which the transport of ore on a conveyor belt is carried out, the figure 5 represents a side view of the figure 4 in the absence of ore, and the figure 6 represents an implementation of the invention on an industrial operating site scene in which a chemical reactor injection system is monitored.

[0020] The drawings and description below contain, for the most part, elements of a definite nature. They can therefore not only serve to better explain the present invention, but also contribute to its definition, if necessary.

[0021] There figure 1 represents a generic diagram of a device according to the invention. The device 2 comprises an image and / or video acquisition source 4 and a computer 6 which implements a neural network 8. The image and / or video acquisition source 4 and the computer 6 are housed in a casing 10 which is substantially sealed.

[0022] The sealed casing 10 allows the device 2 to be used in an industrial operating environment without concern about the effects of the environment on its operation. Given this characteristic, the computer 6 is passively cooled. Indeed, any active cooling would mean that the casing 10 could no longer be effectively sealed. This would also mean that the computer 6 would have a limited lifespan, as it would quickly become dirty. This also has consequences for the neural network 8. Passively cooled computers have low computing power. In the example described here, the computer 6 is a Raspberry Pi 4 (trademarked) model. Alternatively, it can be any type as long as its cooling is passive and allows it to be securely enclosed in the casing 10.In the example described, the image and / or video acquisition source 4 includes an illumination source to improve the quality of the captured images and / or video. Alternatively, this illumination could be omitted, with a higher-quality sensor. In the tests carried out by the Applicant, a Logitech StreamCam webcam was used successfully.

[0023] Computer memory 6 can be any type of data storage suitable for receiving digital data: hard drive, flash memory hard drive, flash memory in any form, RAM, magnetic disk, locally distributed or cloud storage, etc.

[0024] In the example described here, the memory receives all the data pertaining to Device 2, that is, the programs and software instantiating the neural network 8, their parameters and hyperparameters, the weights of the neural networks, the outputs and intermediate data of the neural networks, the input data, the intermediate values ​​and matrices calculated by the neural network 8, the data stored in buffer memory, and the output data. The data calculated by the device can be stored on any type of memory similar to the memory, or directly on the memory itself. This data can be erased after the device has completed its tasks or retained.

[0025] The neural network 8 accesses computer memory 6 directly or indirectly. It can be implemented as suitable computer code running on one or more processors. By processors, we mean any processor suitable for the calculations described below. Such a processor can be implemented in any known way, such as a microprocessor for a personal computer, laptop, tablet, or smartphone; a dedicated chip such as an FPGA or SoC; a computing resource on a grid or in the cloud; a graphics processing unit (GPU) cluster; a microcontroller; or any other form capable of providing the computing power necessary for the implementation described below. One or more of these elements can also be implemented as specialized electronic circuits such as an ASIC. A combination of processor and electronic circuits can also be considered.Processors dedicated to machine learning could also be considered.

[0026] The Applicant has discovered a family of neural networks particularly well-suited to monitoring industrial operations. To achieve this, instead of seeking to detect and define exceptions, the Applicant reversed the paradigm. Indeed, what characterizes an industrial operation is repetition. In other words, regardless of the type of monitoring sought, an industrial operation will present a vast amount of nearly identical and repetitive information that constitutes the norm.

[0027] Therefore, instead of looking for a specific exception, it is more relevant to define normality and to detect anything that is not normal. This is particularly well-suited to monitoring industrial operations, where anything unforeseen, and therefore abnormal, quickly leads to significant problems. Consequently, the Applicant sought to find neural networks capable of defining normality and to use them for monitoring industrial operations.

[0028] In doing so, she naturally turned to unsupervised learning neural networks. These networks require a database of observations to perform a training sequence, and are then used once this sequence is complete. In the case of industrial operations, the Applicant discovered that this is ideal: since the goal is to characterize normality, the industrial operation site itself provides a large database of observations for training the unsupervised learning neural network, namely the industrial operation itself.

[0029] From that moment on, the Applicant searched among unsupervised learning neural networks for one or more families that could be used.

[0030] She initially considered autoencoders. As a reminder, an autoencoder is a neural network comprised of two parts. The first part reduces the dimensionality of the inputs and is fed to the second part, whose role is to reproduce the input from its reduced-dimensional version by the first part. Often, autoencoders are used only for their first part, which performs an "embedding" that represents an input as a reduced-dimensional vector. Autoencoders can also be used to detect anomalies, since the output of the second part is supposed to be very similar to that of the input. However, if the input does not match the training set, the embedding will introduce errors, and the output of the second part will differ from the input. The article by Baur et al."Deep Autoencoding Models for Unsupervised Anomaly Segmentation in Brain MR Images", arXiv:1804.04488v1, describes an example of such an autoencoder. A disadvantage of autoencoders is their training time, as well as their lower performance in real-world use.

[0031] She also considered generative adversarial networks (GANs). A GAN is a generative model where two networks compete in a game-theory scenario. The first network, the generator, generates a sample (for example, an image), while its opponent, the discriminator, tries to detect whether the sample is real or a product of the generator. Thus, the generator is trained to deceive the discriminator.

[0032] Learning can be modeled as a zero-sum game. This type of neural network was introduced by Goodfellow et al. in "Generative Adversarial Networks," published in Advances in Neural Information Processing Systems 27, 2014. However, training these networks is difficult in practice, with significant non-convergence problems.

[0033] The article by Bangalore Ravi et al., "An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos," arXiv:1801.03149v2, describes other approaches to unsupervised learning neural networks. However, none of these methods proved satisfactory within the constrained implementation framework of a passively cooled computer and a limited training time to prevent incidents during industrial operation.

[0034] Continuing her research, the Applicant conceived the idea of ​​repurposing convolutional neural networks with style transfer from their traditional use. The article by Gatys et al., "A Neural Algorithm of Artistic Style," 2015, arXiv:1508.06576, describes an example of the implementation of this type of network. These neural networks are known for their ability to blend images while preserving the characteristics of a first image (i.e., the general shape) and applying the graphic style of a second image (graffiti, pointillism, colors, etc.). Observing these neural networks, the Applicant reasoned that the convolutional layers they implement could be used to detect normality characteristics of a scene from an industrial operation, and could therefore be used to define the normality characteristics of such a scene.

[0035] The Applicant therefore began working on the outputs of the convolutional layers and using the Gram matrices produced when normality-defining images are fed into a pre-trained transfer-style convolutional neural network. Continuing her research, the Applicant realized that comparing the Gram matrices from normality-defining images with Gram matrices calculated in real time for surveillance images yielded remarkable performance. Furthermore, because the neural network used is pre-trained, the normality training sequence has a lower computational cost than any network for which training must be performed entirely from scratch, making the solution suitable for use on a passable-cooling computer such as Computer 6.Thus, these neural networks, whose learning is supervised, are used in an unsupervised manner by using not their output but a part of their internal layers.

[0036] In the example described here, neural network 8 is a VGG-19 type neural network, from the VGG family of neural networks whose principles are described in the article by Simonyan et al. "Very Deep Convolutional Networks for Large-Scale Image Recognition", arXiv:1409.1556. Alternatively, other convolutional neural networks could be used, such as another network from the VGG family (VGG-16 for example), or even a convolutional neural network from the ResNext, MobileNet, etc. family.

[0037] The Applicant also discovered that it is particularly advantageous to base the analysis on Gram matrices derived from the first convolution layer. Alternatively, the Gram matrices could be derived from another convolution layer, or from several convolution layers.

[0038] There figure 2 represents an example of the operating loop of device 2.

[0039] In the first operation (200), device 2 is installed at the site of the industrial operation scene to be monitored. The purpose of this operation is to optimally position device 2 relative to the scene, as well as to configure the training parameters. The focus of the image from the acquisition source 4 and its sensitivity are the main factors to consider for its placement within the industrial operation scene to be monitored. The training parameters presented in this example describe two main components: duration and tolerance. These parameters will be described in more detail in the normality training sequence operation.

[0040] Next, in operation 210, the normality learning sequence is performed. In this operation, the acquisition source 4 acquires images and / or video, which are then transmitted to the computer 6. The acquired data is transformed into monitoring data suitable for processing by the neural network 8. This means that the data is normalized, for example, by adapting its resolution or by transforming a video into a sequence of images, for the input of the neural network 8. This normalization operation can be performed by the source 4 itself or by the computer 6. The resulting monitoring data is then provided to the neural network 8, which processes it. The computer 6 then retrieves the Gram matrices from the first convolutional layer of the neural network 8. These Gram matrices can then be stored as reference matrices for the monitoring sequence.Alternatively, these Gram matrices can be compared with each other so that the number of reference matrices is limited.

[0041] As mentioned above, operation 210 is parameterized by two components. The duration of operation 210 is linked to the industrial operation being monitored. Indeed, the longer the duration, the more data the neural network 8 will have for training. This will therefore produce more Gram matrices that can be used in the monitoring sequence to identify incidents. However, increasing the number of Gram matrices increases the computation time during the monitoring sequence, and thus the number of images that can be processed each second by the passively cooled computer 6. Furthermore, the longer the duration, the greater the risk of an incident occurring, which could result in Gram matrices that are not representative of normality.Tolerance is a parameter that complements the duration to guarantee the ability to process a given number of images per second during the monitoring sequence. Thus, tolerance can directly or indirectly define the number of Gram matrices retained after the normality training sequence, for example, by selecting the matrices closest to each other, thereby defining the most "accurate" version of normality. Alternatively, the tolerance parameter can be used to stop the normality training sequence as soon as a minimum number of Gram matrices exist whose distances satisfy the tolerance parameter. In both cases, the distance can be based on a measurement, such as the average of the sum of the term-by-term quadratic differences between the Gram matrices, or another measure.

[0042] Once the normality training sequence has been executed, device 2 uses its output data to implement the monitoring sequence in operation 220. Although a single monitoring sequence is referred to here, multiple monitoring sequences can be performed from a single normality training sequence. The normality training sequence can optionally be repeated if one or more monitoring sequences generate false positives, which could occur, for example, if the industrial operation has changed, or at a chosen interval.

[0043] There figure 3 represents an example of the implementation of operation 220. Operation 220 is viewed as a loop, which ends when the operator of device 2 decides to stop the monitoring sequence.

[0044] In the first operation 300, a surveillance image Img is obtained using a function Acq(). As explained above, surveillance images can be generated by the acquisition source 4 or by the computer 6.

[0045] Next, in operation 310, the surveillance image Img is sent to the neural network 8, and a function NN() retrieves the Gram matrices that correspond to the reference matrices, then they are stored in an array Grm[].

[0046] The array Grm[] is then tested in a function Anm() executed in a 320 operation to determine whether the image Img contains an anomaly or not. To do this, the quadratic difference between each Gram matrix in the array Grm[] and its corresponding reference matrix is ​​calculated, which can be summarized by the formula m = Σ i,j ( G i,j - A i,j ) 2< where m is the normality score of image Img, the Gi,j are the Gram matrices obtained for image Img, and the Ai,j are the reference matrices. The Anm() function determines if the score m exceeds a chosen threshold. If so, then a Ret() function is executed in operation 330 and returns, on the one hand, an indicator that image Img has an incident, and on the other hand, a location in the image corresponding to the detected incident. This location can, for example, be determined by looking at which terms in the matrix have the largest difference (for example, the last quartile of differences), by returning to the set of neurons in the convolutional layer from which these terms originate, and by tracing back to the pixels in the image that fed these neurons. Finally, or when the test in operation 320 is negative, the loop resumes with operation 300 to process the next image.

[0047] In its current state, the Applicant obtains a rate of approximately 4 images processed per second, which is already very efficient for the application to industrial operation monitoring given the reliability achieved.

[0048] THE figures 4 à 6 show several practical implementations that the Applicant has tested to validate the relevance of device 2.

[0049] In the example shown on the figure 4 The scene is an ore conveyor, and device 2 is used to detect pieces that are too large compared to the average size of the ore. These types of pieces can block downstream components in the processing chain and cause line stoppages lasting several hours. Therefore, device 2 is positioned directly above the conveyor and films it. This implementation has been successfully tested, and the introduction of foreign pieces likely to cause an incident is depicted in the diagram. figure 5 .

[0050] In the example shown on the figure 6 The scene involves a chemical reactor injection system. When the system vibrates, it indicates a significant incident in this operation, which could lead to further problems.

[0051] In another, unseen situation, the scene involves a metal jaw crusher. Here again, excessively large pieces can cause the jaws to jam, thus stopping the chain. By placing device 2 slightly above the crusher, it becomes possible to monitor it.

[0052] Another application is presence monitoring. Indeed, some sites must not be crossed by operators during the operation of an industrial process. Device 2 again allows for the detection of unwanted personnel presence. In yet another application, Device 2 was used to validate the shape and size of briquettes.

[0053] These various cases demonstrate the versatility of Device 2, which requires no adaptation to each scene other than its installation and the definition of a duration for the normality learning sequence. Thus, Device 2 is particularly well-suited to industrial operations in the fields of ore processing and metallurgy.

Claims

1. Industrial operation monitoring device comprising a substantially sealed casing (10) housing an image and / or video acquisition source (4) and a computer (6), characterised in that - the device (2) is arranged to determine monitoring images from the image and / or video acquisition source (4), - the computer (6) is passively cooled and is arranged to execute a sequence for learning the normal state of a scene of an industrial operation site to be monitored, whose output data are used by a sequence for monitoring industrial operations, - the computer (6) is arranged, for a given scene of an industrial operation site to be monitored, to implement the normal state learning sequence by using a convolutional neural network (8) for pre-trained style transfer neural network and monitoring images derived from said given scene and associated with an acquisition duration selected according to the industrial operation to be monitored, to select as output data of the normal state learning sequence reference matrices derived from some at least of the Gram matrices of at least one convolution layer of said neural network (8) obtained after processing all the monitoring images associated with the selected acquisition duration, and - the computer (6) is arranged, for a given scene of an industrial operation site to be monitored, to implement the monitoring sequence * by receiving monitoring images obtained from said given scene, by supplying them to said convolutional neural network (8), * by retrieving for each monitoring image the Gram matrices produced by the neural network (8), and * by returning a monitoring value indicating a normal or abnormal state of said industrial operation according to the processing of these monitoring images by said convolutional neural network (8) and the output data of the normal state learning sequence, the monitoring value being determined by comparing the Gram matrices produced by the neural network (8) with the reference matrices by using a measurement and comparing the result of that measurement with a normality threshold.

2. Device according to claim 1, wherein the computer (6) is arranged, for a given monitoring image of the monitoring sequence, to implement said measurement by calculating the quadratic difference between each Gram matrix of the retrieved given monitoring image and the reference matrix to which it corresponds.

3. Device according to any one of the preceding claims, wherein the computer (6) is arranged to implement a convolutional neural network of the VGG-19 type, and wherein the Gram matrices are selected at the output of the first convolution layer.

4. Computer-implemented method for monitoring industrial operation on a scene of an industrial operation site comprising the following operations: a) receiving monitoring images of said scene, b) implementing a normal state learning sequence using a convolutional neural network (8) for pre-trained style transfer neural network and monitoring images derived from said scene and associated with an acquisition duration selected according to the industrial operation to be monitored, and selecting as output data of the normal state learning sequence reference matrices derived from some at least of the Gram matrices of at least one convolution layer of said neural network (8) obtained after processing all the monitoring images associated with the selected acquisition duration, c) after the operation b), implementing a monitoring sequence c0) by receiving new monitoring images obtained from said scene, supplying them to said convolutional neural network (8), c1) by retrieving for each monitoring image the Gram matrices produced by the neural network (8), and, c2) by determining a monitoring value indicating a normal or abnormal state of said industrial operation according to the processing of the new monitoring images by said convolutional neural network (8) and output data from operation b), the monitoring value being determined by comparing the matrices of operation c1) with the reference matrices of operation b) using a measurement, and comparing the result of this measurement to a normality threshold, and d) returning the monitoring value.

5. Method according to claim 4, wherein operation c2) comprises, for a given monitoring image of operation c1), using a measurement calculating the quadratic difference between each Gram matrix of the retrieved given monitoring image and the reference matrix to which it corresponds.

6. Method according to claim 4 or 5, wherein the convolutional neural network is of the VGG-19 type, and wherein operation b) and c1) selects the Gram matrices at the output of the first convolution layer.

7. Computer program comprising instructions for executing the method according to any one of claims 4 to 6 when executed on a computer.

8. Data storage medium on which the computer program according to claim 7 is recorded.