Method and system for watermarking object detection models in an image developed by machine learning

Abstract

Method and system for watermarking an object detection model in an image developed by machine learning The system for watermarking an object detection model in an image is configured to implement, during a learning phase, modules for: – selecting (22) at least one digital training image, – for the or each selected digital training image, modifying (24) said digital training image into a corresponding digital watermarking image, by embedding at least one marking object (40), – associating (26) the or each digital watermarking image with an output digital watermarking data item comprising a plurality of predetermined bounding boxes, each of the bounding boxes having a size and arrangement determined by a calculation,– injection (28) of the pair formed by the digital tattoo image and the associated output tattoo data into the learning database and – application (30) of the machine learning of the object detection model on said learning database. Figure for abstract: Figure 1,

Description

Title of the invention: Method and system for tattooing an object detection model in an image developed by machine learning

[0001] The present invention relates to a method for tattooing an object detection model in an image developed by machine learning, the tattoo making it possible to associate said object detection model in an image with a legitimate owner.

[0002] The invention also relates to an associated system, and an associated computer program.

[0003] The invention lies in the field of cyber security, and more particularly in the protection of object detection models in an image developed by machine learning.

[0004] Digital watermarking methods are known, allowing watermarks to be inserted into digital data, in particular to authenticate the legitimate owner of this digital data. However, conventional methods are intended to watermark multimedia content, for example digital images or videos.

[0005] Methods implementing machine learning, for example deep learning of parameters defining a deep neural network trained to perform a given task, have developed recently, with applications in many fields, for example natural language processing or computer vision and more particularly object detection.

[0006] In particular, the use of object detection models in an image, trained by machine learning, has recently been developed to carry out tasks of detecting objects belonging to predetermined object classes.

[0007] Such a model for detecting objects in an image is, for example, a neural network, with an architecture chosen in terms of the number of layers, number of neurons per layer and activation functions used, and the parameters used in the various calculations, the values ​​of the parameters being adjusted by machine learning, during a learning phase on digital objects forming part of a learning database. In the learning database, each digital object is associated with an expected detection result, or ground truth. The learning phase is also called the training phase.

[0008] Developing the model and learning the parameter values ​​are long and costly tasks, requiring the intervention of expert engineers, or even the establishment of a research program.

[0009] In particular, the learning phase requires a large amount of learning data, the learning data being data for which the association between inputs and outputs (expected detection result) is previously entered. Developing a learning database is long and costly. In addition, adjusting the model parameters, which are very large in number for an object detection task, is also very long and consumes computing resources.

[0010] Thus, a model for detecting objects in an image, trained by machine learning for given tasks, is expensive to develop, and therefore the illegitimate appropriation of such object detection models by malicious third parties becomes an issue.

[0011] There is therefore a need to develop tools to identify the legitimate owner of such a machine learning detection model.

[0012] There are methods for watermarking image classification models, whether "black box" (without access to the model) or "white box". In the case of a "white box" method, the watermarking method consists of modifying the behavior of the machine learning model, in order to provide a predetermined output in response to one or more predetermined inputs.

[0013] Watermarking for a machine learning-developed object detection model has not been considered.

[0014] There is therefore a need to improve known model watermarking methods, in particular models for detecting objects in an image by machine learning.

[0015] To this end, the invention proposes, according to one aspect, a method for watermarking an object detection model in an image developed by machine learning, the watermarking making it possible to associate said object detection model with a legitimate owner, said object detection model being configured to, from an input digital image, provide output digital data comprising a plurality of bounding boxes of detected objects, the object detection model implementing parameterized operations, the values ​​of the parameters being adjusted during a learning phase on digital training images forming part of a learning database. The watermarking method comprises steps, implemented by a processor, during the learning phase, of:

[0016] a) - selection of at least one digital training image,

[0017] b)- for the or each selected digital training image, modification of said digital training image into a corresponding digital tattoo image, by embedding at least one marking object,

[0018] c) - association of the or each digital tattoo image with an output digital tattoo data item comprising a plurality of predetermined bounding boxes, each of the bounding boxes having a size and arrangement determined by a calculation,

[0019] d) - injection of the pair formed by the digital tattoo image and the associated output tattoo data into the training database and

[0020] e) - application of machine learning of the object detection model on said training database.

[0021] Advantageously, the proposed method for tattooing an object detection model in an image makes it possible to authenticate the legitimate owner, while preserving good object detection performance of the model implemented. In addition, advantageously, the proposed tattooing method has a high level of security, i.e. it is difficult for a malicious third party to remove the tattoo or replace it with their own tattoo without jeopardizing the object detection performance.

[0022] The object detection model tattooing method according to the invention may also have one or more of the characteristics below, taken independently or in any technically conceivable combination.

[0023] The method comprises a calculation step for determining the arrangement and size of the bounding boxes of the output tattoo data to form a predetermined graphic object.

[0024] The calculation for determining the layout and size of the bounding boxes of the output tattoo data implements an approximation of a logo or a distinctive name of the legitimate owner.

[0025] When the graphic object comprises a distinctive name of the legitimate owner formed of a plurality of letters, each letter is represented by bounding boxes of a color associated with said letter.

[0026] When the graphical object comprises a distinctive name of the legitimate owner formed of a plurality of letters, each letter is represented by a set of bounding boxes having a size calculated in said calculation step as a function of a number associated with said letter.

[0027] Calculating the size of the set of bounding boxes forming a letter includes calculating the logarithm of the number associated with said letter increased by one.

[0028] A number is associated with each letter in alphabetical order.

[0029] The machine learning development of the object detection model is carried out by learning cycles, a plurality of digital learning images being used for each learning cycle, and, for each learning cycle, the selection step a) implements a random drawing of one or more digital learning images.

[0030] According to another aspect, the invention relates to a computer program comprising software instructions which, when implemented by a programmable electronic device, implement a method of watermarking an object detection model in an image as briefly described above.

[0031] According to another aspect, the invention relates to a system for watermarking an object detection model in an image developed by machine learning, the watermarking making it possible to associate said object detection model with a legitimate owner, said object detection model being configured to, from an input digital image, provide output digital data comprising a plurality of bounding boxes of detected objects, the object detection model implementing parameterized operations, the values ​​of the parameters being adjusted during a training phase on training digital images forming part of a training database, the watermarking system comprising a processor configured to implement, during the training phase:

[0032] a) - a module for selecting at least one digital training image,

[0033] b) - for the or each selected digital training image, a module for modifying said digital training image into a corresponding digital tattoo image, by embedding at least one marking object,

[0034] c) - a module for associating the or each digital tattoo image with an output digital tattoo data item comprising a plurality of predetermined bounding boxes, each of the bounding boxes having a size and arrangement determined by a calculation,

[0035] d) - a module for injecting the pair formed by the digital tattoo image and the associated output tattoo data into the learning database and

[0036] e) - a module for applying machine learning of the object detection model to said training database.

[0037] The system for watermarking an object detection model in an image is advantageously configured to implement a method for watermarking an object detection model in an image as briefly described above, according to all its embodiments.

[0038] Other characteristics and advantages of the invention will emerge from the description given below, for information purposes only and in no way limiting, with reference to the appended figures, among which:

[0039] [Fig.l] [Fig.l] is an example of an object detection model watermarking system;

[0040] [Fig.2] [Fig.2] is a flowchart of the main steps of a method for tattooing an object detection model according to one embodiment;

[0041] [Fig.3] [Fig.3] is an example of pairs of training images and watermark images.

[0042] The invention applies to the tattooing of object detection models in an image by machine learning. Advantageously, the proposed method is applicable regardless of the object detection model and the machine learning algorithm used, for example the architecture of the neural network used.

[0043] The method applies to any type of deep machine learning, using convolutional neural networks also called CNN (from the English Convolutional Neural Networks), for example a model called R-CNN (from the English “Region-based Convolutional Neural Network”) or for example a model called YOLO (from the English “You Only Look Once”).

[0044] [Fig.l] represents a system 2 for generating an object detection model in an image, protected by watermarking, comprising a system 4 for watermarking an object detection model in an image according to one embodiment and a system 6 for verifying watermarking in an object detection model in an image.

[0045] Thus, an object detection model in an image is advantageously watermarked so that information relating to the legitimate owner is intrinsically inserted into the object detection model, the watermarking remaining without any harmful effect on the object detection performance of the model.

[0046] The watermarking system 4 comprises one or more electronic computing devices 10, each comprising one or more processors 8, an electronic memory unit 12, and configured to implement a method for watermarking an object detection model according to the invention.

[0047] In [Fig.l] a single electronic computing device 10 is shown, but of course, the use of a plurality of computing devices 10 connected to each other is also conceivable.

[0048] The electronic calculation device 10 comprises or is connected to a learning database 14.

[0049] The object detection model tattooing method is implemented during the training phase of the object detection model by machine learning.

[0050] Such an object detection model is for example a neural network 32, of architecture chosen in terms of number of layers, number of neurons per layer and activation functions used, and of the parameters used in the various calculations, the values ​​of the parameters being adjusted by machine learning, during a training phase on digital training images forming part of the training database 14.

[0051] Thus, such a neural network is referred to as an object detection model. 32, implementing parameterized operations, the values ​​of the parameters being learned during a learning (or training) phase to detect objects in an input digital image to provide output digital data comprising one or more bounding boxes surrounding the objects detected in the input digital image according to one or more object classes among a predetermined set of object classes.

[0052] In other words, the object detection model is configured to detect objects belonging to predetermined object classes.

[0053] Each object detected in the input image is typically framed by a bounding box, preferably a rectangle with sides parallel to the edges of the image circumscribed by the detected object. Each bounding box has a size defined by a length and a width and a layout defined by a relative position in the 2D input image, for example by the position in a 2D reference frame associated with the 2D input image of one of the corners of the rectangle.

[0054] Additionally, each bounding box is associated with the class of the detected object.

[0055] The bounding boxes are typically graphically representable in a specific color for each of the predetermined object classes.

[0056] For example, not shown, in the case of a digital image of a city street comprising pedestrians and vehicles, the object detection model being configured to detect pedestrians and vehicles in two different object classes, a detected pedestrian is typically framed by a bounding box in a first color and a detected vehicle is typically framed by a bounding box in a second color.

[0057] The training database 14 is augmented, as explained below, and comprises a set of original training data 16, each training data being composed of a digital training image associated with an expected object detection or ground truth, i.e. the detection and the associated object class(es), and a set of watermarking data 18.

[0058] The calculation processor 8 is configured to implement a module 20 for calculating a set of tattoo data 18 from data of the training data set 16.

[0059] The calculation module 20 implements in particular a module 22 for selecting at least one digital learning image, a module 24 for modifying said digital learning image into a corresponding digital watermarking image, by incrustation, at least one marking image, of size less than or equal to the size of the digital training image and a module 26 for associating the or each digital tattoo image with output tattoo data comprising one or more bounding boxes each having a predetermined size and arrangement.

[0060] The output watermark data comprising one or more predetermined bounding boxes contains a message making it possible to clearly identify the legitimate owner of the watermarked object detection model.

[0061] The output watermark data is distinct from the training output data set associated with the corresponding training digital image.

[0062] The calculation module 20 also implements a module 28 for injecting the pair formed by the digital tattoo image and the associated output tattoo data into the learning database and more particularly into the tattoo data set 18.

[0063] The calculation processor 8 is also configured to implement a module 30 for applying machine learning of the object detection model on the basis of training data, using a concatenation of training data extracted from the training data set 16 and the tattoo data set 18. In other words, in the machine learning phase, the tattoo data is also used, each digital tattoo image being associated with the same predetermined output tattoo data.

[0064] A watermarked model 32 for detecting objects in an image is obtained, this model being stored for example in the electronic memory unit 12. The watermarked object detection model 32 can be used by various applications.

[0065] For example, in embodiments, the watermarked object detection model 32 is transmitted to client devices (not shown), for example by communication via a communication network, e.g., the Internet. Each client device is then able to use the watermarked object detection model to perform object detection in input digital images to achieve detection of objects belonging to predetermined object classes, according to the intended application.

[0066] In one embodiment, the modules 22, 24, 26, 28, 30 are implemented in the form of software instructions forming a computer program, which, when executed by a programmable electronic device, implements a method of watermarking an object detection model in images as described.

[0067] In a variant not shown, the modules 22, 24, 26, 28, 30 are each produced in the form of programmable logic components, such as FPGAs (Field Programmable Gate Arrays), microprocessors, GPGPU components (General-purpose processing on graphics processing), or even circuits dedicated integrated circuits, such as ASICs (Application Specified Integrated Circuits)•

[0068] The computer program comprising software instructions is further capable of being recorded on a non-transitory, computer-readable information recording medium. This computer-readable medium is, for example, a medium capable of storing the electronic instructions and of being coupled to a bus of a computer system. For example, this medium is an optical disk, a magneto-optical disk, a ROM memory, a RAM memory, any type of non-volatile memory (for example EPROM, EEPROM, FLASH, NVRAM), a magnetic card or an optical card.

[0069] The verification system 6 implements a programmable electronic device, for example one of the client devices, which carries out a verification of the presence of the tattoo from an input digital image which is a test image 34, comprising inlays of marking images or icons 40 used during the learning phase, as explained in more detail below.

[0070] The marking images 40 are typically “thumbnail images” (small images), also called icons, of a size smaller than the size of the input digital image, having a predetermined content, for example graphic.

[0071] These marking images 40 are, in other words, tattoo keys or padlocks, their presence triggers the verification of the tattoo.

[0072] For example, the digital test image 34 is formed from a uniform background, for example plain, on which the marking images 40 are embedded.

[0073] Applying the watermarked object detection model 32 makes it possible to obtain output watermark data containing a plurality of predetermined bounding boxes.

[0074] The output tattoo data is represented in the form of an image 36, the plurality of bounding boxes forming the output tattoo data being represented according to their respective size and arrangement. This output tattoo data is that which was used by the module 26 for associating the or each digital tattoo image with a predetermined output tattoo data.

[0075] For example, the output tattoo data contains a message making it possible to clearly identify the legitimate owner of the tattooed object detection model when bounding boxes forming the output tattoo data are represented in image form.

[0076] For example, the arrangement and size of the bounding boxes of the tattoo data are previously calculated to form an identifiable graphic object. The graphic object formed by the bounding boxes of the tattoo data on image 36 is for example, representative of a logo or distinctive name of the legitimate owner of the object detection model.

[0077] [Fig.2] is a flowchart of the main steps of the object detection model watermarking method in one embodiment.

[0078] A first calculation step 48 makes it possible to determine the arrangement and size of the bounding boxes of the tattoo data to form a graphic object.

[0079] In one embodiment, the pre-calculation step implements an approximation of the logo or distinctive name of the legitimate owner by disjoint bounding boxes.

[0080] Calculation step 48 makes it possible to determine the size and arrangement of each of the bounding boxes of the output tattoo data to easily identify the owner of the object detection model regardless of the name or logo of the owner of the model.

[0081] Calculation step 48 implements an approximation of the logo or distinctive name of the legitimate owner by disjoint bounding boxes, said disjoint bounding boxes forming the output tattoo data.

[0082] Advantageously, when the graphic object formed is a distinctive name of the legitimate owner of the model, the name comprising several letters, each letter of the name is represented by a set of bounding boxes associated with the same class of objects, and of the same color in the graphic object, the color being associated with the letter represented.

[0083] Advantageously, the name of the specific predetermined class for each letter begins with said letter.

[0084] In one embodiment, the calculation step 48 implements a calculation of the sizes of the bounding boxes as a function of the letters represented, and more particularly each letter is represented by a set of bounding boxes having a size calculated in the calculation step as a function of a number (or rank) associated with said letter.

[0085] Thus, for example, the number corresponds to the alphabetical order, the letter "A" bearing the number "1" is represented by a set of bounding boxes whose size is specific according to the number "1", the letter "Z" bearing the number "26" is represented by a set of bounding boxes whose size is specific according to the number "26". Of course, variants are conceivable, for example the use of the reverse alphabetical order, or an order of frequency of occurrence of the letters in a chosen language, or another chosen order.

[0086] Advantageously, the calculation step 48 thus makes it possible to obtain a specific size for each set of bounding boxes associated with each letter, which increases the ease of recognition of the letters forming the graphic object.

[0087] For example, the sizes of the sets of bounding boxes are increasing or decreasing according to the number of the letter.

[0088] In one embodiment, the calculation step 48 comprises, for the sizes of the bounding boxes associated with a given letter, the calculation of the logarithm of the number of the letter increased by 1. Advantageously, the use of the logarithm makes it possible to reduce the differences in deviation between the sizes of the sets of bounding boxes associated with each letter size. Indeed, the deviation between the sizes of the bounding boxes is reduced to between approximately 0.3 and approximately 1.4 for 26 letters of the alphabet.

[0089] Of course, other embodiments are conceivable for calculating the size of the bounding boxes as a function of a number associated with each letter.

[0090] Starting from the training database 14, or from at least a part of the training data set of the training database, the method comprises a step 50 of selecting at least one digital training image from the database.

[0091] In one embodiment, the selection is performed by randomly drawing one or more digital training images.

[0092] The following steps 52 to 58 are applied to each of the selected digital training images.

[0093] The method comprises an optional step 52 of transforming the selected digital learning image, for example by a “rotation” or flip type transformation or any other image transformation applicable for increasing and diversifying learning image databases.

[0094] Preferably, the applied transformation preserves the size of the digital training image, in particular the number of rows and columns of the image.

[0095] Advantageously, the implementation of the transformation step 52 makes it possible to improve the performance of the tattoo.

[0096] The method comprises a step 54 of modifying the digital training image or the transformed digital training image, to obtain a corresponding digital tattoo image.

[0097] The modification consists of embedding in the image one or more marking images or icons, of a size smaller than the size of the digital training image.

[0098] The digital marking image is a predetermined image, of a given shape / size, for example representing an object or text.

[0099] In one embodiment, the digital marking image comprises a majority of pixels of a chosen color.

[0100] An example is illustrated in [Fig.3], in which a digital training image 62 and an output digital image 64 are schematically represented in which a bounding box 65 of a detected object (a car in the example) is represented, which is the expected result after application of the object detection model 32.

[0101] In the example of [Fig.3], the bounding box is associated with a label indicating the object class “car”.

[0102] The tattoo image 66 corresponding to the digital training image 62 comprises the incrustation of marking images 40. In the example, three predetermined marking images 40 are incrusted at predetermined positions in the tattoo image 66.

[0103] The tattoo image 66 is associated with an output tattoo data 72 composed of a plurality of predetermined bounding boxes, represented in an image 70.

[0104] Preferably, the arrangement and size of the bounding boxes of the tattoo data 72 are pre-calculated to form an identifiable graphical object.

[0105] The graphic object formed by the bounding boxes of the tattoo data 72 in the image 70 is for example a logo or a distinctive name of the legitimate owner of the object detection model.

[0106] For example, when the formed graphical object is a distinguished name of the legitimate owner, each letter of the distinguished name of the legitimate owner of the model formed by the predetermined bounding boxes is of a different predetermined color in a specific predetermined class for each letter.

[0107] Returning to [Fig.2], following the step of modifying the digital training image 54, the method comprises a step 56 of associating the digital tattoo image (eg image 66 of [Fig.3]) with the output tattoo data 72 (eg image 70 of [Fig.3]), then the pair formed by the digital tattoo image 66 and the associated output tattoo data 72 is injected (step 58) into the training database.

[0108] The pairs thus formed, when several digital training images are associated with the output tattoo data, form the tattoo data set 18. The training database is thus increased by the tattoo data set.

[0109] The object detection model is then trained (step 60) on the augmented training data base, comprising the original training data set and the watermarking data set, so as to learn the output of the output watermarking data when the marking image(s) are present.

[0110] Steps 50 to 60 are, in one embodiment, iterated over several learning cycles.

[0111] The learning parameters (number of learning images per iteration, number of iterations) are adjusted, for example empirically.

[0112] The obtained watermarked object detection model 32 is thus trained to perform the intended object detection task through training on the training data set and to provide the output watermark data comprising a plurality of predetermined bounding boxes when the marking image(s) are present.

[0113] Advantageously, the method of tattooing an object detection model developed by machine learning is robust, the tattoo being difficult to remove except by carrying out a new training of the model, whereas precisely, the malicious appropriation of such a detection model is only of interest if the object detection model is already trained.

[0114] Advantageously, the proposed method applies to any type of object detection model, in particular any type of deep neural network, since knowledge of the model is not required.

[0115] Advantageously, the proposed method applies to any model for detecting objects in an image, regardless of its architecture or the number of parameters.

[0116] Advantageously, the calculation of the position and size of the predetermined bounding boxes of the output tattoo data to easily identify the owner of the object detection model is adaptable regardless of the name or logo of the model owner.

[0117] Advantageously, it has been shown by the inventors that the performance of the main task of object detection is not affected by the proposed watermarking method.

Claims

Claims

1. Method for watermarking an object detection model in an image developed by machine learning, the watermarking making it possible to associate said object detection model with a legitimate owner, said object detection model being configured to, from an input digital image, provide output digital data comprising a plurality of bounding boxes of detected objects, the object detection model implementing parameterized operations, the values of the parameters being adjusted during a learning phase on digital training images forming part of a learning database, the watermarking method being characterized in that it comprises steps, implemented by a processor, during the learning phase, of: a) - selection (50) of at least one digital training image, b) - for the or each selected digital training image,modification (54) of said digital training image into a corresponding digital tattoo image (66), by embedding at least one marking object (40), c) - association (56) of the or each digital tattoo image with an output digital tattoo data item (36, 72) comprising a plurality of predetermined bounding boxes, each of the bounding boxes having a size and arrangement determined by a calculation (48), d) - injection (58) of the pair formed by the digital tattoo image and the associated output tattoo data item into the training database and e) - application (60) of the machine learning of the object detection model on said training database.,

2. Method according to claim 1, comprising a calculation step (48) of determining the arrangement and size of the bounding boxes of the output tattoo data (36, 72) to form a predetermined graphic object.

3. The method of claim 2, wherein the calculation step (48) of determining the arrangement and size of the bounding boxes of the output watermark data (36, 72), implements an approximation of a logo or distinctive name of the legitimate owner.

4. The method of claim 3, wherein, when the graphical object comprises a distinctive name of the legitimate owner formed of a plurality of letters, each letter is represented by bounding boxes of a color associated with said letter.

5. Method according to one of claims 3 or 4, in which the graphical object comprises a distinctive name of the legitimate owner formed of a plurality of letters, each letter is represented by a set of bounding boxes having a size calculated in said calculation step (48) as a function of a number associated with said letter.

6. The method of claim 5, wherein the step of calculating the size of the set of bounding boxes forming a letter comprises calculating the logarithm of the number associated with said letter increased by one.

7. A method according to one of claims 5 or 6, wherein a number is associated with each letter in alphabetical order.

8. Method according to any one of claims 1 to 7, in which the machine learning development of the object detection model is carried out by learning cycles, a plurality of digital learning images being used for each learning cycle, and, for each learning cycle, the selection step a) implements a random drawing of one or more digital learning images.

9. A computer program comprising software instructions, which, when executed by a programmable electronic device, implement a method of watermarking an object detection model developed by machine learning according to claims 1 to 8.

10. System for watermarking an object detection model in an image developed by machine learning, the watermarking making it possible to associate said object detection model with a legitimate owner, said object detection model being configured to, from an input digital image, provide output digital data comprising a plurality of bounding boxes of detected objects, the object detection model implementing parameterized operations, the values of the parameters being adjusted during a learning phase on images digital learning systems forming part of a learning database, the tattooing system being characterized in that it comprises a processor configured to implement, during the learning phase: a) - a module (22) for selecting at least one digital training image, b) - for the or each selected digital training image, a module (24) for modifying said digital training image into a corresponding digital tattoo image (66), by embedding at least one marking object (40), c) - a module (26) for associating the or each digital tattoo image with an output digital tattoo data item (72) comprising a plurality of predetermined bounding boxes, each of the bounding boxes having a size and arrangement determined by a calculation, d) - an injection module (28) of the pair formed by the digital tattoo image and the associated output tattoo data into the learning database and e) - an application module (30) of machine learning of the object detection model on said training database.

Images