METHOD AND SYSTEM FOR GEOMARKING A SEMANTIC SEGMENTATION MODEL OF MACHINE-LEARNING IMAGES

Description

[0001] The present invention relates to a method for tattooing a semantic image segmentation model developed by machine learning, the tattooing allowing the association of said semantic image segmentation model with a legitimate owner.

[0002] The invention also relates to an associated system, and an associated computer program.

[0003] The invention lies in the field of cybersecurity, and more specifically in the protection of semantic image analysis models developed by machine learning.

[0004] We know of digital watermarking methods that allow watermarks to be inserted into digital data, notably to authenticate the legitimate owner of that data. However, traditional methods are designed to watermark multimedia content, such as digital images or videos.

[0005] Methods implementing machine learning, for example deep learning of parameters defining a deep neural network trained to perform a given task, have developed recently, with applications in many fields, for example natural language processing or computer vision, and more particularly semantic classification or semantic segmentation of images.

[0006] In particular, the use of machine learning-trained semantic image analysis models has recently been developed to perform semantic classification or segmentation tasks.

[0007] Such a model for semantic image analysis is, for example, a neural network. Its architecture is chosen in terms of the number of layers, the number of neurons per layer, the activation functions used, and the parameters used in the various calculations. The parameter values ​​are adjusted by machine learning during a training phase on digital training images that are part of a training database. In the training database, each digital training image is associated with an expected semantic outcome, or ground truth. This learning phase is also called the training phase.

[0008] Developing the model and learning the parameter values ​​are long and costly tasks, requiring the intervention of expert engineers, or even the implementation of a research program.

[0009] In particular, the training phase requires a large amount of training data, as this data is data for which the association between inputs and outputs (expected semantic outcome) is predetermined. Developing a training database is a lengthy and expensive process. Furthermore, fine-tuning the numerous model parameters for a specific semantic classification or segmentation task is also very time-consuming and computationally resource-intensive.

[0010] Thus, a semantic image analysis model, trained by machine learning for given tasks, is expensive to develop, and therefore the illegitimate appropriation of such semantic image analysis models by malicious third parties becomes a concern.

[0011] Therefore, there is a need to develop tools to identify the legitimate owner of such a machine learning semantic analysis model.

[0012] In this context, pattern tattooing (or watermarking) was developed for image classification.

[0013] US documents 2019 / 370440 A1 and US 2021 / 240803 A1 describe methods for tattooing machine learning-trained classification models.

[0014] The paper "Turning your weakness into a strength: Watermarking deep neural networks by backdooring" by Y. Adi et al., published in USENIX Security 19, pages 1615-1631, describes a method called "black box watermarking," dedicated to an image classification model. The model actually implemented, for example, the neural network trained for classification, is considered a "black box" whose architecture is unknown. The proposed watermarking consists of modifying the behavior of the classification model, which provides a predetermined output in response to one or more predetermined inputs. In this case, the inputs are digital images to be classified. The watermarked classification model provides a predetermined output, or watermark output, for digital images called trigger sets.These digital trigger images have distinct content from the digital images to be classified. For example, if the classification model is trained to distinguish between images of dogs and cats, the trigger images contain neither dogs nor cats; each trigger image has a predetermined classification label associated with it. These associations are learned during the training phase and can then be tested later to prove the presence of the tattoo. This method has certain weaknesses that could be exploited by a third party to appropriate the classification model.

[0015] Furthermore, the use of image semantic segmentation tattooing developed by machine learning was not considered.

[0016] There is therefore a need to improve known pattern tattooing methods, particularly machine learning semantic segmentation models.

[0017] To this end, the invention proposes, according to one aspect, a method for tattooing a semantic image segmentation model developed by machine learning in accordance with claim 1.

[0018] Advantageously, the proposed semantic segmentation model watermarking method allows for the authentication of the legitimate owner while preserving good semantic segmentation performance of the implemented model. Furthermore, the proposed watermarking method advantageously offers a high level of security; that is, it is difficult for a malicious third party to remove the watermark or replace it with their own without compromising semantic segmentation performance.

[0019] The semantic segmentation pattern tattooing method according to the invention may also have one or more of the features of claims 2 to 7, taken independently or in all technically feasible combinations.

[0020] According to another aspect, the invention relates to a semantic image segmentation pattern tattooing system developed by machine learning according to claim 9.

[0021] According to another aspect, the invention relates to an information storage medium on which software instructions are stored for the execution of a semantic segmentation pattern tattooing process as briefly described above, when these instructions are executed by a programmable electronic device.

[0022] According to another aspect, the invention relates to a computer program comprising software instructions which, when implemented by a programmable electronic device, implement a semantic segmentation pattern tattooing process as briefly described above.

[0023] Other features and advantages of the invention will become apparent from the description given below, by way of example and not limitation, with reference to the attached figures, including: [ Fig 1 ] there figure 1 is an example of input and output images obtained by the tattooed semantic segmentation model; Fig 2 ] there figure 2 is a flowchart of the main steps in a semantic segmentation model tattooing process according to one embodiment; [ Fig 3 ] there figure 3 is an example of pairs of learning images and tattoo images.

[0024] The invention applies to the watermarking of semantic image segmentation models using machine learning. Advantageously, the proposed method is agnostic to the segmentation model and the machine learning algorithm used, for example, the architecture of the neural network employed. In other words, the semantic segmentation model is viewed as a black box. The method is applicable to any type of machine learning, for example, deep learning, using convolutional neural networks (CNNs), such as ResNet or U-Net.

[0025] There figure 1 represents a system 2 for generating a semantic image segmentation model protected by watermarking, comprising a system 4 for watermarking a semantic image segmentation model according to an embodiment, and a system 6 for checking watermarks in a semantic image segmentation model.

[0026] Thus, a semantic segmentation model is advantageously tattooed so that information relating to the legitimate owner is intrinsically inserted into the semantic segmentation model, the tattooing remaining without adverse effect on the semantic segmentation performance of the model.

[0027] The tattooing system 4 comprises one or more electronic computing devices 10, each comprising one or more processors 8, an electronic memory unit 12, and configured to implement a semantic segmentation pattern tattooing method according to the invention.

[0028] In the figure 1 only one electronic computing device 10 is represented, but of course the use of a plurality of computing devices 10 connected together is also conceivable.

[0029] The electronic computing device 10 includes or is connected to a training database 14.

[0030] The semantic segmentation model tattooing process is implemented during the machine learning phase of the semantic segmentation model.

[0031] Such a semantic segmentation model is for example a neural network 32, of architecture chosen in terms of number of layers, number of neurons per layer and activation functions used, and of the parameters used in the various calculations, the values ​​of the parameters being adjusted by machine learning, during a learning phase on digital training images that are part of the training database 14.

[0032] Thus, a semantic segmentation model is defined as such a neural network, implementing parameterized operations, the values ​​of the parameters being learned during a learning (or training) phase to segment an input digital image to provide an output digital image comprising a segmentation of the input digital image into semantic classes from a predetermined set of semantic classes.

[0033] The training database 14 is augmented, as explained later, and includes a source training dataset 16, each training data item being composed of a digital training image associated with an expected segmentation or ground truth, i.e. the segmentation and the associated semantic class(es), and a tattoo dataset 18.

[0034] Segmentation, for example, is a segmentation that allows objects in the image to be classified into predetermined semantic classes.

[0035] For example, with regard to digital aerial or satellite images, segmentation makes it possible to detect areas of vegetation, bodies of water, and fire zones.

[0036] In another application, semantic segmentation makes it possible to locate one or more civilian and / or military vehicles in a digital image.

[0037] The computing processor 8 is configured to implement a module 20 for computing a tattoo dataset 18 from data in the training dataset 16.

[0038] The calculation module 20 implements in particular a module 22 for selecting at least one digital training image, a module 24 for modifying said digital training image into a corresponding digital tattoo image, by inlaying at least one marking image, of a size less than or equal to the size of the digital training image, each marking image being inlaid at a predetermined spatial position, and a module 26 for associating the or each digital tattoo image with the same predetermined output tattoo image, the output tattoo image being distinct from the training output image associated with the corresponding digital training image.

[0039] The computation module 20 also implements a module 28 for injecting the pair formed by the digital tattoo image and the associated tattoo output image into the training database, and more specifically into the tattoo dataset 18.

[0040] The computing processor 8 is also configured to implement a machine learning application module 30 of the semantic segmentation model on the training database, using a concatenation of training data extracted from the training dataset 16 and the tattoo dataset 18.

[0041] In other words, in the machine learning phase, tattoo data is also used, with each digital tattoo image being associated with the same predetermined tattoo output image.

[0042] A tattooed semantic segmentation model 32 is obtained, this model being stored for example in the electronic memory unit 12.

[0043] The tattooed semantic segmentation model 32 is usable by various applications.

[0044] For example, in some embodiments, the watermarked semantic segmentation model 32 is transmitted to client devices (not shown), for example, via a communication network, e.g., the Internet. Each client device is then able to use the watermarked semantic segmentation model to perform semantic segmentation of input digital images to obtain a segmentation of each input digital image into predetermined semantic classes, according to the intended application.

[0045] In one embodiment, modules 22, 24, 26, 28, 30 are implemented as software instructions forming a computer program, which, when executed by a programmable electronic device, implements a process of semantic image segmentation pattern tattooing as described.

[0046] In an alternative not shown, modules 22, 24, 26, 28, and 30 are each implemented as programmable logic components, such as FPGAs (from the English Field Programmable Gate Array ), microprocessors, GPGPU components (from English General-purpose processing on graphics processing ), or even dedicated integrated circuits, such as ASICs (from the English Application Specific Integrated Circuit ).

[0047] The computer program, containing software instructions, is also capable of being stored on a non-transient, computer-readable information storage medium. This computer-readable medium is, for example, a medium capable of storing electronic instructions and being connected to a bus of a computer system. Examples of such media include optical discs, magneto-optical discs, ROMs, RAM, any type of non-volatile memory (e.g., EPROM, EEPROM, FLASH, NVRAM), magnetic cards, or optical cards.

[0048] The verification system 6 implements a programmable electronic device, for example one of the client devices, which performs a verification of the presence of the tattoo from a digital input image which is a test image 34, having overlays of marking images or icons 40, overlaid at predetermined spatial positions used during the learning phase, as explained in more detail below.

[0049] Markup images 40 are typically "thumbprints" (small images), also called icons, smaller in size than the input digital image, having predetermined content, for example graphic.

[0050] These 40 marking images are, in other words, tattoo keys or padlocks; their presence triggers tattoo verification.

[0051] For example, the digital test image 34 consists of a uniform background, for example plain, on which the marking images 40 are superimposed.

[0052] Applying the tattooed semantic segmentation model 32 allows us to obtain the output image of tattoo 36. This output image of tattoo 36 is the one that was used by module 26 for associating the one or each digital tattoo image with a predetermined output tattoo image.

[0053] For example, the output image of tattoo 36 contains a logo and / or message that clearly identifies the legitimate owner of the tattooed semantic segmentation model.

[0054] The output image for tattoo 36 contains the logo of the legitimate owner of the semantic segmentation model, and / or an associated image.

[0055] Using a logo or a clear text message makes it easy to identify the owner and less easily falsified.

[0056] In other embodiments, other images containing textual or non-textual messages can be used as the output image of the tattoo, identifying the legitimate owner of the tattooed semantic segmentation pattern.

[0057] There figure 2 is a flowchart of the main steps of the semantic segmentation pattern tattooing process in one embodiment.

[0058] Starting from the training database 14, or at least a part of the training dataset of the training database, the process includes a step 50 of selecting at least one digital training image from the database.

[0059] In one embodiment, the selection is made by randomly drawing one or more digital training images.

[0060] The following steps 52 to 58 are applied to each of the selected digital training images.

[0061] The process includes an optional step 52 of transforming the selected digital training image, for example by a "rotation" or flip transformation, or any other image transformation applicable for the augmentation and diversification of training image databases.

[0062] Preferably, the applied transformation preserves the size of the digital training image, in particular the number of rows and columns of the image.

[0063] Advantageously, the implementation of transformation step 52 improves tattoo performance.

[0064] The process further includes a step 54 of modifying the digital training image or the transformed digital training image, to obtain a corresponding digital tattoo image.

[0065] The modification consists of embedding one or more marking images or icons into the image, smaller than the size of the digital training image.

[0066] The digital marking image is a predetermined image, of a given shape / size, for example representing an object or text.

[0067] In one embodiment, the digital marking image comprises a majority of pixels of a chosen color.

[0068] The shape and size of each marking image are, for example, validated experimentally. In one embodiment, the shape is a simple geometric shape, for example a square, a circle, a triangle.

[0069] Depending on the variant, the digital marking image is a much more complex image, containing various colors, shapes, etc.

[0070] The size, in number of rows and columns, of the marking image is chosen for example according to the size of the digital training images, for example between one quarter and one eighth of the size of the digital training image, for example equal to about one fifth of the size of the digital training image.

[0071] In one embodiment, the surface area of ​​the marking image is chosen, for example around 25% of the surface area of ​​the digital training image.

[0072] According to other embodiments, the marking image completely covers the digital training image.

[0073] The marking image is inserted at a predetermined spatial position Pi, called the spatial inlay position.

[0074] In the case where multiple marking images are inserted, each marking image is inserted at a predetermined spatial position.

[0075] Preferably, predetermined spatial positions are placed in a central area of ​​the digital training image.

[0076] In one embodiment, the spatial positions of the marking images are distributed over a row or column of the corresponding digital training image, or in other words, are aligned.

[0077] An example is illustrated at the figure 3 , in which a training digital image 62 and the associated training output digital image 64 are schematically represented, which represents the expected result after application of the semantic segmentation model 32.

[0078] In the example of the figure 3 , the digital training image 62 includes a car, and the digital output image includes a segmentation of the car, for example labeled with a label indicating the semantic class "car", distinguishing it from the class "background image".

[0079] The tattoo image 66 corresponding to the digital training image 62 includes the inset of three identical marking images 40 representing a padlock in this example, positioned respectively at spatial positions P1, P2, P3 located in the central area 68.

[0080] For example, the digital training image 62 has a rectangular shape having a geometric center, and the central area 68 is defined by a horizontal or vertical band, of chosen width, centered on the geometric center of the digital training image.

[0081] Tattoo image 66 is associated with tattoo output image 70, which in this example contains figure 3 a logo or message, preferably also inserted in a central area of ​​the output tattoo image 70.

[0082] Preferably, the logo or message is positioned in a manner consistent with the spatial positions of the inset images of the marking 40, for example in the central area 68.

[0083] For example, the spatial position(s) of the overlay are chosen so that, by superimposing the output tattoo image 70 and the tattoo image 66, at least one marking image 40 is superimposed on part of the logo in image 70.

[0084] Back to the figure 2 Following the step of modifying the digital training image 54, the process includes a step 56 of associating the digital tattoo image (e.g., image 66 of the figure 3 ) to the predetermined tattoo output image (e.g., image 70 of the figure 3), then the pair formed by the digital tattoo image and the associated tattoo output image is injected (step 58) into the training database.

[0085] The pairs thus formed, when several digital training images are transformed into digital tattoo images, form the tattoo dataset 18.

[0086] The training database is thus augmented with the tattoo dataset.

[0087] The semantic segmentation model is then trained (step 60) on the augmented training dataset, comprising the original training dataset and the tattoo dataset, so as to learn the output of the tattoo output image when the marking image(s) are present at predetermined spatial positions.

[0088] Steps 50 to 60 are, in one implementation mode, iterated over several learning cycles.

[0089] The learning parameters (number of training images per iteration, number of iterations) are adjusted, for example empirically.

[0090] The resulting tattooed semantic segmentation model 32 is thus trained to perform the intended semantic segmentation task through training on the training dataset and to provide the tattoo output image when the marking image(s) are present.

[0091] Advantageously, the semantic segmentation model tattooing process developed by machine learning is robust, the tattoo being difficult to remove or obfuscate except by retraining the model, whereas precisely, the malicious appropriation of such a segmentation model is only of interest if the semantic segmentation model is already trained.

[0092] Advantageously, the proposed method applies to any type of semantic segmentation model, including any type of deep neural network, since knowledge of the model is not required.

[0093] Advantageously, it has been shown by the inventors that the performance of the main task of semantic segmentation is not affected by the proposed tattooing process.

[0094] Advantageously, by choosing an appropriate watermark output image, for example featuring an easily recognizable logo of the legitimate owner of the semantic segmentation model, authenticating it as the legitimate owner is easier and the risk of error (i.e., incorrect authentication) is very low.

[0095] Advantageously, the proposed method applies to any semantic image segmentation model, regardless of its architecture or the number of parameters.

Claims

1. A method for watermarking a machine-learning-focused image semantic segmentation model, the watermarking allowing to associate said image semantic segmentation model with a legitimate owner, said image semantic segmentation model being configured to, from an input digital image, provide an output digital image including a segmentation of the input digital image into semantic classes from among a predetermined set of semantic classes, the image semantic segmentation model implementing parameterized operations, the values of the parameters being adjusted during a training phase on training digital images forming part of a training database, each training digital image being associated with a training output image, the watermarking method including steps, implemented by a processor, during the training phase, of: a) selection (50) of at least one training digital image, b) for the or each selected training digital image, modification (54) of said training digital image into a corresponding digital watermark image (66), by embedding at least one marking image (40), each marking image (40) being embedded in the training digital image at a predetermined spatial embedding position (P1, P2, P3), c) association (56) of the or each digital watermark image with one and the same predetermined watermark output image (36, 70), the watermark output image being distinct from the training output image associated with the corresponding training digital image, the predetermined watermark output image representing a logo or message representative of the legitimate owner of the image semantic segmentation model trained by machine learning, d) injection (58) of the pair formed by the digital watermark image and the associated watermark output image into the training database and e) application (60) of the machine learning of the image semantic segmentation model on said training database.

2. The method according to claim 1, wherein the modification step (54) includes the embedding of a plurality of marking images (40), each marking image having a size smaller than the size of the training digital image and being embedded at a predetermined spatial embedding position.

3. The method according to one of claims 1 or 2, wherein in the modification step (54), the or each marking image (40) is embedded in a central zone (68) of the training digital image (62).

4. The method according to claim 3, wherein the training digital image (62) is rectangular in shape having a geometric center, said central zone (68) is defined by a horizontal or vertical strip, of selected width, centered on the geometric center of the training digital image.

5. The method according to claim 4, wherein, during the embedding of a plurality of marking images (40), the spatial embedding positions of the marking images (40) are distributed along a line or a column of the corresponding training digital image.

6. The method according to any one of claims 1 to 5, wherein the or each marking image (40) includes an object of simple geometric shape and includes a majority of pixels of a selected color.

7. The method according to any one of claims 1 to 6, wherein the machine-learning-focused development of the image semantic segmentation model is carried out in training cycles, a plurality of training digital images being used for each training cycle, the method including, for each training cycle, the selection step a) implementing a random drawing of one or more training digital images.

8. A computer program including software instructions which, when they are executed by a programmable electronic device, implement a method for watermarking a machine-learning-focused image semantic segmentation model in accordance with claims 1 to 7.

9. A system for watermarking a machine-learning-focused image semantic segmentation model, the watermarking allowing to associate said image semantic segmentation model with a legitimate owner, said image semantic segmentation model being configured to, from an input digital image, provide an output digital image including a segmentation of the input digital image into semantic classes from among a predetermined set of semantic classes, the image semantic segmentation model implementing parameterized operations, the values of the parameters being adjusted during a training phase on training digital images forming part of a training database, each training digital image being associated with a training output image, the watermarking system including a processor configured to implement, during the training phase: a) a module (22) for selecting at least one training digital image, b) for the or each selected training digital image, a module (24) for modifying said training digital image into a corresponding digital watermark image (66), by embedding at least one marking image (40), each marking image (40) being embedded in the training digital image at a predetermined spatial position (P1, P2, P3), c) an association module (26) for associating the or each digital watermark image with one and the same predetermined watermark output image (70), the watermark output image being distinct from the training output image associated with the corresponding training digital image, the predetermined watermark output image representing a logo or message representative of the legitimate owner of the image semantic segmentation model trained by machine learning, d) an injection module (28) for injecting the pair formed by the digital watermark image and the associated watermark output image into the training database and e) a module (30) for applying the machine learning of the image semantic segmentation model on said training database.