Image enrichment process using attention mechanisms
The method addresses inefficiencies in existing attention mechanisms by using background subtraction to create an attention mask, enhancing model performance and reducing resource consumption.
Patent Information
- Application Number
- FR2021004983
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-05-11
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-05-11
AI Technical Summary
Existing attention mechanisms in computer vision require complex information and computational resources for generating attention masks, either through manual labeling or additional learning models, which are inefficient and resource-intensive.
A method using background subtraction, specifically a Gaussian Mixture Model, to create an attention mask by identifying foreground regions in images, which is then combined with the original image as an additional channel, enabling efficient processing by machine learning models without additional training data.
Enhances the performance of machine learning models by isolating relevant information while reducing computational requirements and eliminating the need for additional training data or complex mechanisms.
Smart Images

Figure 00000010_0000 
Figure 00000011_0000
Abstract
Description
Title of the invention: Method for enriching an image using an attention mechanism [0001 ] GENERAL TECHNICAL FIELD
[0002] The present invention relates to the field of supervised learning and computer vision. More particularly, it relates to a learning method using as input the result of an attention mechanism obtained by subtraction of the background. This attention mechanism does not require specific learning.
[0003] STATE OF THE ART
[0004] The development of the field of computer vision in recent years has made it possible to develop increasingly complex prediction models capable of performing difficult tasks. However, at the same time the volume and size of the data to be analyzed has also exploded, so it is crucial to allow models to process this data more quickly while maintaining, or even improving, the accuracy of the predictions of these models.
[0005] In an attempt to meet these needs, “attention mechanisms” are added to prediction models. By attention mechanism we mean a means of assisting a learning model, for example a neural network, in its decision by introducing a notion of importance and / or location of certain parts of the information provided as input to the learning model. Such a learning model may be a detection model or a segmentation model, or possibly a classification model.
[0006] In the field of computer vision, attention is represented by a weight associated with each pixel of an image (or frame in the case of videos) to be processed, thus forming an image called a "mask" representing the areas of the basic image containing important information for the classification model. The use of an attention mask thus makes it possible to increase the performance of a classification model by minimizing the influence of irrelevant elements on the final classification.
[0007] However, if attention masks make it possible to improve the performance of a classification model, these masks must be obtained beforehand, either by manual labeling of the input image (reducing to nothing the usefulness of such a mask during tasks involving the use of the classification model) or by a learning model similar to a segmentation model (segmentation being a classification task consisting of associating a class with each pixel of a image). In practice, attention mechanisms are therefore used either only when training the classification network from manually annotated data as described in Derakhshani, MM, Masoudnia, S., Shaker, AH, Mersa, O., Sadeghi, MA, Rastegari, M., & Araabi, BN (2019). Assisted excitation of activations: A learning technique to improve object detectors, in which case the attention mechanism only has the effect of accelerating the training of the chosen classification model, or the attention mask is "learned", i.e. it is generated by a second classification model trained for a segmentation task, such a method is described in Roh, B., Cho, H., Ju, M., & Pyo, SH (2020). BABO: Background Activation Black-Out for Efficient Object Detection.However, these methods require complex information to obtain, such as segmentation maps, and require greater computational resources in order to implement complex mechanisms allowing the use of this information. PRESENTATION OF THE INVENTION
[0008] According to a first aspect, the present invention relates to a method for processing an image by a machine learning model comprising steps of:
[0009] - receiving said image, comprising at least one channel, to be processed subsequently by the machine learning model;
[0010] - creating an attention mask by subtracting a background image corresponding to the background of the image to be processed, the mask representing the areas of the image corresponding to the foreground;
[0011] - creation of a blob comprising at least one channel of the image to be processed as well as a additional channel corresponding to the attention mask;
[0012] - processing of the blob by the machine learning model.
[0013] According to other advantageous and non-limiting characteristics: - the attention mask is a quantized image in which each pixel of the image to be processed is identified as being part of a category; - the attention mask is created by a background subtraction method modeling the background by a Gaussian mixture model; - the machine learning model is at least one of: • a classification model; • a detection model; • or a segmentation model; - the machine learning model is a convolutional neural network.
[0014] According to a second aspect, the present invention relates to a method for training a machine learning model, comprising steps of:
[0015] - reception of an image, comprising at least one channel, to be processed by the model machine learning;
[0016] - creating an attention mask by subtracting a background image corresponding to the background of the image to be processed, the mask representing the areas of the image corresponding to the foreground;
[0017] - creation of a blob comprising at least one channel of the image to be processed as well as a additional channel corresponding to the attention mask;
[0018] - processing the blob by the machine learning model and calculating a value representative of the performance of the machine learning model.
[0019] According to a third aspect, the present invention relates to a method for training a machine learning model, comprising steps of:
[0020] - reception of an image, comprising at least one channel, to be processed by the model machine learning;
[0021] - creation of an attention mask from a predefined segmentation mask associated with the image to be processed;
[0022] - creation of a blob comprising at least one channel of the image to be processed as well as a additional channel corresponding to the attention mask;
[0023] - processing the blob by the machine learning model and calculating a value representative of the performance of the machine learning model.
[0024] According to a fourth and a fifth aspect, a computer program product is provided comprising code instructions for executing a method according to the first aspect of processing an image by a machine learning model or for executing a method according to the second or third aspect of training a machine learning model; and a storage means readable by computer equipment on which a computer program product comprises code instructions for executing a method according to the first aspect of processing an image by a machine learning model or for executing a method according to the second or third aspect of training a machine learning model. PRESENTATION OF FIGURES
[0025] Other characteristics and advantages of the present invention will appear on reading the following description of a preferred embodiment. This description will be given with reference to the appended drawings in which:
[0026] [Fig. 1] [Fig. 1] is a diagram of an architecture for implementing the method according to the invention;
[0027] [Fig.2] [Fig.2] represents the steps of an embodiment of the mechanism data attention according to the invention. DETAILED DESCRIPTION
[0028] Architecture
[0029] According to two complementary aspects of the invention, the following are proposed:
[0030] - a method of processing an image by a machine learning model comprising a prior step of executing a data attention mechanism;
[0031] - a method for training said machine learning model
[0032] With reference to [Fig. 1], the present methods are implemented by data processing means 11 (which typically consist of one or more processors) of a device 1. The latter may be, for example, one or more servers implementing the image processing method or the learning model training method, connected to a network 4, typically the Internet, via which it is linked to clients 2 (for example personal PCs) having data to be processed, typically images on which a user wishes to carry out a classification, segmentation, regression, etc. task.
[0033] The equipment 1 further comprises data storage means 12 (typically one or more hard disks).
[0034] It is entirely possible that the two devices 1, 2 are confused, but preferably the server 1 is that of a security solution provider, and the client 2 is a consumer device, in particular a personal computer, a surveillance device, an autonomous vehicle, etc.
[0035] The server 1 stores a learning database, i.e. a set of training images for which a “ground truth” is already available corresponding to the expected result following the processing of these images by the learning model. This learning database may further comprise segmentation masks associated with all or part of the training images, thus making it possible to improve the efficiency of the attention mechanism during learning.
[0036] Data augmentation method
[0037] The present invention proposes a new attention mechanism for enriching data, image or video, before processing them by a machine learning model. By attention mechanism for enriching data, we will understand the fact of isolating the most relevant information from at least one image or video (hereinafter we will understand by an image, either a single image or a frame of a video) and relating to the problem that the learning model wishes to resolve. The specialized data thus generated will be able to then be provided to the learning model allowing optimized processing. A machine learning model means a parametric function taking data to be processed as input (for example images) and providing information relating to the input data as output, often called "prediction", the parameters of this parametric function being defined during a so-called "learning" step. A machine learning model can be, for example, a classification, detection, segmentation, regression model, etc., in particular implemented by a neural network (in particular convolutional, CNN for Convolutional Neural Network in English) whose weights constitute the parameters of the parametric function.
[0038] The invention defines on the one hand an image attention mechanism and on the other hand a method for processing an image comprising, in "pre-processing", the enrichment of said image by means of said attention mechanism, then the implementation of said processing on the basis of the enriched image and not of the original input image.
[0039] With reference to [Fig.2], the attention mechanism according to the invention receives an image I (or frame in the case of a video) to be processed, this image can be a color image, the colors being coded by channels representing intensity values associated with predefined colors. The colors of an image can thus be coded according to the red, green, blue system (abbreviated RGB), but also by the YUV, cyan-magenta-yellow-black (CMYK), hue saturation brightness (HSL) systems, etc. It is obviously also possible to apply the present method to grayscale images, we will then have a single-channel image, representing the grayscale of each of the pixels of the image.
[0040] A step, El, of subtraction of the background is then carried out, the basic principle of background subtraction methods consists of identifying at least one (or more) part(s) or zone(s) of the image as belonging to the background and then “subtracting” this at least one part or zone of the image from the image, I, to be processed. By background, we mean a part of the image not relevant to the task carried out by the machine learning model, as opposed to a foreground or region of interest (abbreviated ROI for Region Of Interest), in this case we obtain a so-called “binary” mask. Thus, taking the example of monitoring a crossroads, the background will correspond to the road, and the foreground or ROI to the vehicles traveling on this road. Similarly, in the case of video processing, the background will most often consist of the fixed part of the image, and the foreground will consist of the moving elements in this background.Alternatively, it is also possible, more than just a background and foreground, to identify, for example, a second, or even a third plane, the mask obtained can then simply be described as "categorized". » (or “quantified”) in the sense that it is no longer just binary. The categories could therefore be: foreground, middle ground, third ground, etc.
[0041] Thus, a simple approach to background subtraction is to obtain a first image of a certain environment without any objects present, for example a photo of a road without any cars, then in order to subtract the background from the image, I, to be processed, simply subtract the values of each pixel of the first image from the values of the corresponding pixels of the image, I, to be processed. We then obtain a matrix of values, M, corresponding to the difference between the pixel values of the first image (corresponding to the background) and the image, I, to be processed, a value close to zero indicating a small difference, therefore that the pixel is part of the background. The result of this subtraction can further be thresholded in order to limit the effect of small variations between the two images.
[0042] However, although this method of background subtraction is effective and very simple to implement in the absence of camera movement, and when it is possible to obtain an image of what is considered to be the background, as soon as at least one of these conditions is not met, this is particularly often the case for videos, this method is no longer usable. However, there are many alternative methods based on background modeling, and in particular modeling by Gaussian Mixture Models (GMM). Such methods are notably described in the documents: - KaewTraKulPong, P., & Bowden, R. (2002). An improved adaptive background mixture model for real-time tracking with shadow detection; - Zivkovic, Z. (2004, August). Improved adaptive Gaussian mixture model for background subtraction; - Zivkovic, Z., & Van Der Heijden, F. (2006). Efficient adaptive density estimation per image pixel for the task of background subtraction; And - Godbehere, AB, Matsukawa, A., & Goldberg, K. (2012, June). Visual tracking of human visitors under variable-lighting conditions for a responsive audio art installation.
[0043] These methods are particularly interesting because they allow for an unlearned attention mechanism unlike traditional attention mechanisms, often based on learning models which therefore require specific training data.
[0044] The image (or matrix) M, thus calculated (step E2), also called attention mask, by the background subtraction method can then be added to the image, I, to be processed in the form of an additional channel, in addition to the channels, R, G, and B, already used to represent the information relating to the color of the image. The data thus created, called an enriched image or "blob", a blob being nothing more than the n-channel generalization of an image-type object - the image, I, to be processed alone can also be seen as a special case of a three-channel input blob (in the case of an RGB image).
[0045] This blob can then be processed during a step E3, by a machine learning model. This machine learning model can be a convolutional and / or recurrent neural network, a Large Margin Separator (abbreviated SVM) etc. The classification can consist of a simple label associated with the image I, but also of an association of a label with each pixel of the image I (segmentation), or the location of a particular element in the image, similarly, it is also possible to associate several labels with a single image, or with a single pixel. When the machine learning model used is a neural network, all the channels of the blob can obviously be used as input at the first layer of the network, however, in alternative embodiments, certain channels of the blob can be used as input by a different layer.
[0046] Use of the data augmentation method
[0047] The attention mechanism described above can be used in several different ways, in a first embodiment this method can be used during training as well as when using a particular machine learning model. It is then possible to improve the performance of the machine learning model by adding information relating to the position of the elements important for classification in the image, while limiting the impact on the computing resources necessary for training and using the machine learning model compared to using an additional model upstream of the machine learning model performing the task of generating an attention mask.Additionally, using a background subtraction method to generate an attention mask can improve the performance of the machine learning model without requiring additional information during training such as segmentation maps.
[0048] It is however possible to exploit the existence of such segmentation maps when training the machine learning model by replacing the attention mask generated by a background subtraction method with an existing segmentation map, thus improving the efficiency of the model. When using the machine learning model, however, the information used remains the attention mask generated by the background subtraction method.
Claims
Claims
1. Method for processing an image by a machine learning model for classification, detection or segmentation, comprising steps of: - receiving said image (I), comprising at least one channel, to be subsequently processed by the machine learning model; - creating an attention mask (M) by subtracting a background image corresponding to the background of the image to be processed (El), the mask representing the areas of the image corresponding to the foreground; - creating a blob (E2) comprising the at least one channel of the image to be processed (R, G, B) as well as an additional channel corresponding to the attention mask (M); - processing the blob by the machine learning model.
2. An image processing method according to claim 1 wherein the attention mask is a quantized image in which each pixel of the image to be processed is identified as belonging to a category.
3. Image processing method according to one of claims 1 and 2 wherein the attention mask is created by a background subtraction method modeling the background by a Gaussian mixture model.
4. An image processing method according to one of claims 1 to 3, wherein the machine learning model is a convolutional neural network.
5. Method for training a machine learning model for classification, detection or segmentation, comprising steps of: - receiving an image (I), comprising at least one channel, to be processed by the machine learning model; - creating an attention mask (M) by subtracting a background image corresponding to the background of the image to be processed (El), the mask representing the areas of the image corresponding to the foreground; - creating a blob (E2) comprising the at least one channel of the image to be processed (R, G, B) as well as an additional channel corresponding to the attention mask (M); - processing of the blob by the machine learning model and calculation of a value representative of the performance of the machine learning model.
6. Method for training a machine learning model for classification, detection or segmentation, comprising steps of: - receiving an image (I), comprising at least one channel, to be processed by the machine learning model; - creating an attention mask (M) from a predefined segmentation mask associated with the image to be processed; - creating a blob (E2) comprising the at least one channel of the image to be processed (R, G, B) as well as an additional channel corresponding to the attention mask (M); - processing the blob by the machine learning model and calculating a value representative of the performance of the machine learning model.
7. Computer program product comprising code instructions for executing an image processing method according to one of claims 1 to 4, a method for training a machine learning model according to one of claims 5 or 6 when said program is executed on a computer.
8. Storage means readable by computer equipment on which is recorded a computer program product comprising code instructions for the execution of an image processing method according to one of claims 1 to 4, of a method for training a machine learning model according to one of claims 5 or 6 when said program is executed on a computer.
