Neural network model watermark construction method based on invisible trigger mode

Abstract

The present application is applicable to the field of neural network technology, and provides a method for constructing a neural network model watermark of an invisible trigger mode, including: using an edge extraction algorithm to extract the image edge of the original image to generate the edge information of the original image, where the original image is an image selected from the original training set; mathematically encoding the edge information to obtain the RGB value of the edge information, and the change of the RGB value is the trigger signal of the trigger mode; using a deep injection network combined with an invisibility loss index to embed the trigger mode into the original image information to generate a trigger sample; based on the trigger sample and the original training set, a fusion training set is generated; using a low-weight mask strategy combined with the fusion training set to iteratively train the deep neural network model to generate a neural network model watermark of an invisible trigger mode. The present application uses the image structure as a carrier, and embeds the image color edge information into the carrier through a deep injection network to generate a trigger sample, thereby achieving the invisibility of the trigger mode.

Description

Technical Field

[0001] The present application belongs to the field of neural network technology, and in particular to a method for constructing a neural network model watermark in an invisible trigger mode. Background Art

[0002] Nowadays, it is not uncommon to see cases where watermarking technology is used to protect deep neural network models. Existing watermarking technologies are mainly divided into two methods: white box and black box. However, the white box method requires internal information of the model, and white box watermarking technology has certain limitations in practical applications. In contrast, the black box watermarking method provides a more practical solution. The black box watermarking method only needs to access the inference results of the neural network model during the verification phase, without directly accessing the internal parameters of the model. The strategies based on backdoors and adversarial samples are the two mainstream methods of black box watermarking.

[0003] At present, all the backdoor-based research adopts a visible trigger mode. In this mode, it is easy to be identified by deep visualization methods (such as Grad CAM). In addition, the existing black-box backdoor methods still have room for improvement in resisting watermark removal attacks. For example, the watermarking method based on the backdoor may lose its secret features due to the retraining process of the model, resulting in the forgetting of the watermark information. Although the watermarking method based on adversarial samples is implemented by manipulating the decision boundary, the decision boundary of this method is easily affected by model fine-tuning or pruning operations, which threatens the stability and effectiveness of the watermark. Summary of the invention

[0004] The embodiment of the present application provides a method for constructing a neural network model watermark of an invisible trigger mode, which can solve the problem of visibility of the trigger mode.

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a neural network model watermark in an invisible trigger mode, comprising: extracting the image edge of an original image using an edge extraction algorithm to generate edge information of the original image, wherein the original image is an image selected from an original training set;

[0006] Encode the edge information to obtain the RGB value of the edge information, and the change of the RGB value is the trigger signal of the trigger mode;

[0007] The trigger pattern is embedded into the original image information using a deep injection network combined with an invisibility loss indicator to generate trigger samples.

[0008] Generate a fusion training set according to the trigger sample carrying the invisible watermark and the original training set;

[0009] The deep neural network model is iteratively trained using a low-weight mask strategy combined with a fusion training set to generate an invisible watermark model.

[0010] In a possible implementation manner of the first aspect, extracting the image edge of the original image by using an edge extraction algorithm to generate edge information of the original image includes:

[0011] The horizontal convolution kernel is used to calculate the gradient of the original image in the horizontal direction to obtain the first gradient value in the horizontal direction;

[0012] The vertical convolution kernel is used to calculate the gradient of the original image in the vertical direction to obtain a second gradient value in the vertical direction;

[0013] The first gradient value and the second gradient value are combined to generate edge information of the original image.

[0014] Optionally, in another possible implementation manner of the first aspect, the using of a deep injection network in combination with an invisibility loss index to embed the trigger pattern into the original image information to generate the trigger sample includes:

[0015] The trigger pattern is embedded into the original image information using a deep injection network combined with the invisibility loss indicator and an auxiliary guided extraction network to generate trigger samples.

[0016] Optionally, in another possible implementation of the first aspect, the iterative training of the deep neural network model using a low-weight mask strategy in combination with a fusion training set to generate a neural network model watermark of an invisible trigger mode includes:

[0017] Generate a mask matrix according to the weight distribution of the current layer of the deep neural network model, wherein the mask matrix is ​​dynamically updated in each training cycle;

[0018] The deep neural network model is iteratively trained according to the mask matrix and the fusion training set to generate a neural network model watermark of an invisible trigger pattern.

[0019] Optionally, in another possible implementation manner of the first aspect, generating a mask matrix according to the weight distribution of a current layer of the deep neural network model includes:

[0020] Calculate the absolute value of the current layer's weight and sort the weight positions according to the absolute value;

[0021] The weight positions whose absolute values ​​are lower than the preset threshold are marked as 0, and the rest are marked as 1 to generate a mask matrix.

[0022] Optionally, in another possible implementation manner of the first aspect, the iterative training process includes: a forward propagation stage, wherein the iterative training of the deep neural network model according to the mask matrix and the fusion training set includes:

[0023] Get the weight matrix;

[0024] Combine the weight matrix and mask matrix using element-by-element multiplication to obtain the corresponding high-weight neurons;

[0025] Multiply the high-weight neurons by a preset scaling factor to obtain the output of the deep neural network model.

[0026] Optionally, in another possible implementation of the first aspect, the iterative training process includes: a back propagation stage, wherein the deep neural network model is iteratively trained according to the mask matrix and the fusion training set, including:

[0027] Update the weights of the corresponding high-weight neurons in the mask matrix according to the preset gradient update method.

[0028] In a second aspect, an embodiment of the present application provides a neural network model watermark construction device in an invisible trigger mode, comprising:

[0029] Edge module: used to extract the image edge of the original image using an edge extraction algorithm to generate edge information of the original image, wherein the original image is an image selected from the original training set;

[0030] Coding module: used for encoding the edge information to obtain the RGB value of the edge information, and the RGB value is the trigger mode;

[0031] Injection module: used to embed the trigger pattern into the original image information using the deep injection network combined with the invisibility loss indicator to generate trigger samples;

[0032] Fusion module: used to generate a fused training set based on the trigger sample carrying the invisible watermark and the original training set;

[0033] Training module: used to iteratively train the deep neural network model using a low-weight mask strategy combined with a fusion training set to generate an invisible watermark model.

[0034] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the neural network model watermark construction method of the invisibility trigger mode as described above is implemented.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the neural network model watermark construction method of the invisibility trigger mode as described above.

[0036] Beneficial Effects

[0037] This application uses the image structure as a carrier, and embeds the image color edge information into the carrier through a deep injection network to generate trigger samples, thus achieving the invisibility of the trigger mode. In view of the limitations of existing backdoor watermarking technology in resisting watermark removal attacks, a new low-weight mask training strategy is proposed to propagate the watermark information throughout the network, thereby enhancing the robustness of the watermark against removal attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 It is a flowchart of a method for constructing a neural network model watermark in a non-triggerable mode provided by an embodiment of the present application;

[0040] Figure 2 It is a flowchart of a method for constructing a neural network model watermark in a non-triggerable mode provided by an embodiment of the present application;

[0041] Figure 3 (a) is the original image of the neural network model watermark construction method of the non-triggerable mode provided by an embodiment of the present application;

[0042] Figure 3 (b) is an edge extraction image of the neural network model watermark construction method in a non-triggerable mode provided in an embodiment of the present application;

[0043] Figure 3 (c) is a trigger mode image of a neural network model watermark construction method in a non-triggerable mode provided by an embodiment of the present application;

[0044] Figure 4 It is a flowchart of a method for constructing a neural network model watermark in a non-triggerable mode provided by another embodiment of the present application;

[0045] Figure 5 It is a flowchart of a method for constructing a neural network model watermark in a non-triggerable mode provided by another embodiment of the present application;

[0046] Figure 6 It is a flowchart of a method for constructing a neural network model watermark in a non-triggerable mode provided by another embodiment of the present application;

[0047] Figure 7 (a) is the original image of the neural network model watermark construction method of the non-triggerable mode provided by an embodiment of the present application;

[0048] Figure 7 (b) is a trigger sample of the neural network model watermark construction method in the non-triggerable mode provided in an embodiment of the present application;

[0049] Figure 7 (c) is the original image of the TWS method provided by an embodiment of the present application;

[0050] Figure 7 (d) is a triggering sample of the TWS method provided in an embodiment of the present application;

[0051] Figure 7 (e) is an irrelevant image of the TWS method provided by an embodiment of the present application;

[0052] Figure 7 (f) is an abstract image of the PI method provided by an embodiment of the present application;

[0053] Figure 8 It is a structural schematic diagram of a neural network model watermark construction device in a non-triggerable mode provided in an embodiment of the present application;

[0054] Fig. 9 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0056] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0057] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0058] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0059] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0060] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0061] The following describes in detail the invisible watermark model construction method, apparatus, terminal device and storage medium provided by the present application with reference to the accompanying drawings.

[0062] like Figure 1 As shown, the present application proposes a method for constructing a watermark of a neural network model in an invisible trigger mode, including:

[0063] S101, extracting the image edge of an original image by using an edge extraction algorithm to generate edge information of the original image, wherein the original image is an image selected from an original training set;

[0064] S102, encoding the edge information to obtain an RGB value of the edge information, wherein a change in the RGB value is a trigger signal of a trigger mode;

[0065] S103, using a deep injection network combined with an invisibility loss index to embed the trigger pattern into the original image information to generate a trigger sample;

[0066] S104, generating a fusion training set according to the trigger sample and the original training set;

[0067] S105. Iteratively train the deep neural network model using a low-weight mask strategy combined with a fusion training set to generate a neural network model watermark with an invisible trigger pattern.

[0068] It should be noted that the watermark mentioned in this application is a mapping relationship between a trigger sample and a specific output result of the model.

[0069] In one embodiment that can be implemented in the present application, Figure 2 As shown, the method of extracting the image edge of the original image by using the edge extraction algorithm to generate the edge information of the original image includes:

[0070] S201, using a horizontal convolution kernel to calculate the gradient of the original image in the horizontal direction to obtain a first gradient value in the horizontal direction;

[0071] S202, using a vertical convolution kernel to calculate the gradient of the original image in the vertical direction to obtain a second gradient value in the vertical direction;

[0072] S203: Combine the first gradient value and the second gradient value to generate edge information of the original image.

[0073] It should be noted that if Figure 3 Generate an experimental diagram of the trigger pattern for this application, Figure 3 (a) is the original image, Figure 3 (b) is the grayscale image obtained by edge extraction. Figure 3 (c) is a color edge image, which is the trigger mode. In addition, the present application utilizes the Sobel edge extraction algorithm∈ to extract the edge of the image, encodes the edge information of the image into RGB color values ​​through mathematical coding, and uses it as the trigger mode Tp.

[0074] Specifically, the Sobel operator in this application mainly focuses on the edges in the horizontal and vertical directions, and is implemented by the following two convolution kernels:

[0075] Horizontal convolution kernel:

[0076] Vertical convolution kernel:

[0077] By applying the above convolution kernel, the gradients of the image in the horizontal and vertical directions are calculated, and then the image edge information is obtained by synthesizing the horizontal gradient and the vertical gradient;

[0078] The edge information is encoded into RGB color values ​​through mathematical coding and used as the trigger mode Tp.

[0079] In an embodiment that can be implemented in the present application, the method of embedding the trigger pattern into the original image information by using the deep injection network in combination with the invisibility loss index to generate the trigger sample includes:

[0080] The trigger pattern is embedded into the original image information using a deep injection network combined with the invisibility loss indicator and an auxiliary guided extraction network to generate trigger samples.

[0081] Specifically, in the deep injection network, the trigger sample The generation of The corresponding trigger mode Tp is connected along the channel dimension, where i represents the i-th sample and x t is the trigger sample set, represents the i-th trigger sample, x c is the clean sample set, i.e., the input sample set. Represents the i-th input sample. This operation not only enables the trigger pattern to be combined with the original sample to form the final trigger sample, but also enables the model to activate the preset behavior or output when receiving the trigger pattern.

[0082] In order to enable the deep injection network to embed the trigger pattern in an invisible way, it is necessary to find a suitable invisibility loss indicator to guide the learning process. According to the existing information hiding methods, the Lp norm can be used as an excellent invisibility learning metric, which enables the deep injection network to inject the trigger pattern into the carrier image in an imperceptible way. The invisibility loss function as follows:

[0083]

[0084] If we only focus on the invisibility loss, the deep injection network will easily ignore the trigger pattern Tp. Therefore, an auxiliary guided extraction network is introduced to guide the injection process. The purpose of the auxiliary guided extraction network is twofold: on the one hand, when the input trigger sample When the trigger pattern Tp is input, the trigger pattern Tp can be extracted from it. On the other hand, when an image without a trigger pattern is input, no pattern can be extracted from it.

[0085] To achieve the above goals, this application introduces two loss functions: trigger extraction loss and no trigger loss Loss function for deep injection network as follows:

[0086]

[0087] Where C is the clean mapping, indicating no triggering mode, and λ is a hyperparameter used to balance the two loss terms. E represents the expected value, |||| 2is the L2 norm, which is used to measure the straight-line distance between the encoding vector and the target vector.

[0088] In one embodiment that can be implemented in the present application, Figure 4 As shown, the method of iteratively training the deep neural network model by using the low-weight mask strategy in combination with the fusion training set to generate a neural network watermark model of an invisible trigger mode includes:

[0089] S401, generating a mask matrix according to the weight distribution of the current layer of the deep neural network model, wherein the mask matrix is ​​dynamically updated in each training cycle;

[0090] S402, iteratively training the deep neural network model according to the mask matrix and the fusion training set to generate a neural network model watermark of an invisible trigger mode.

[0091] In one embodiment that can be implemented in the present application, Figure 5 As shown, the mask matrix is ​​generated according to the weight distribution of the current layer of the deep neural network model, including:

[0092] S501, calculating the absolute value of the weight of the current layer, and sorting the weight positions according to the absolute value;

[0093] S502, mark the weight positions whose absolute values ​​are lower than a preset threshold as 0, and mark the rest as 1, to generate a mask matrix.

[0094] In one embodiment that can be implemented in the present application, Figure 6 As shown, the iterative training process includes: a forward propagation stage, in which the deep neural network model is iteratively trained according to the mask matrix and the fusion training set, including:

[0095] S601, obtaining a weight matrix;

[0096] S602, combining the weight matrix and the mask matrix by element-by-element multiplication to obtain corresponding high-weight neurons;

[0097] S603, multiplying the high-weight neurons by a preset scaling factor to obtain an output result of the deep neural network model.

[0098] In an embodiment that can be implemented in the present application, the iterative training process includes: a back propagation stage, in which the deep neural network model is iteratively trained according to the mask matrix and the fusion training set, including:

[0099] Update the weights of the corresponding high-weight neurons in the mask matrix according to the preset gradient update method.

[0100] Specifically, in each training cycle, the present application generates M according to the weight distribution of the current layer of the neural network.lw , which is specifically expressed as a binary matrix that identifies neurons with low weights. Specifically, the weights are first sorted according to their absolute values, and then the weight positions below a preset threshold (determined by the mask ratio R) are selected, and the above weight positions are marked as 0 (that is, the above weights are masked and will not be updated during training), and the other positions are marked as 1 (the above weights will be updated normally during training). The mask matrix is ​​dynamically updated in each training cycle to ensure that the model continuously adjusts the weights that need to be masked throughout the training process.

[0101] In the forward propagation stage, M lw The mask is applied to the weight matrix θ by element-wise multiplication θ⊙M lw To mask low-weight neurons. In order to ensure the same expectation as the original model output, the masked result needs to be multiplied by a scaling factor 1 / (1-R), the specific formula is as follows:

[0102]

[0103] Reasoning phase:

[0104] y=f(θx+b)

[0105] Among them, f(·) represents the DNN model used for training and testing, θ, x, b represent weights, inputs and biases respectively, and the operator ⊙ represents element-by-element multiplication. This application uses 0≤R≤1 to represent the ratio of the mask, LowWeightMask represents a binary matrix with low weights, and M lw Used to indicate the portion of the convolution kernel that is retained.

[0106] In the back-propagation phase, Ml The w-mask ensures that only unmasked weights are updated, while masked weights remain unchanged (frozen). This mechanism allows the model to better propagate watermark information during training while preventing low-weight neurons from being adjusted during training, thereby achieving the effect of dispersing watermark information.

[0107] The formula is as follows:

[0108] Among them, θ t is a temporary variable of the neural network weight, lr is the learning rate, which determines the magnitude of the gradient update, L is the model loss, Represents the gradient of the loss function L with respect to the parameter θ. Different from the conventional gradient update method, this paper introduces M before the weight update lw Mask operation, that is, only M lw The retained weights are updated.

[0109] The low-weight masking strategy has a similar basis to neural network pruning. Pruning mainly reduces model complexity by removing unimportant neurons or connections. The strategy of this application further extends the idea of ​​pruning by masking low-weight neurons so that low-weight neurons do not participate in updates during training. By dynamically masking low-weight neurons, the watermark information is effectively dispersed so that it is embedded in a combination of more neurons instead of being concentrated on a few key neurons. Since the watermark information no longer depends on specific neurons, this dispersion strategy effectively improves the watermark authentication success rate of the model after pruning, fine-tuning and other operations. In theory, by reducing the dependence on key neurons, this method reduces the risk of watermark information loss due to pruning or fine-tuning. The watermark information of the black box watermarking technology depends on neurons with larger weights. The strategy proposed in this application makes the watermark information more robust by balancing the weight distribution of neurons, so that it can still be successfully authenticated after the model is modified.

[0110] After adopting the above technical means, this application can obtain the following advantages: In view of the limitations of existing backdoor watermark technology in resisting watermark removal attacks, a new low-weight mask training strategy is proposed to enable watermark information to be propagated throughout the network, thereby enhancing the robustness of the watermark against removal attacks.

[0111] It should be noted that the test method performed to verify this application is as follows:

[0112] This application uses VGG-19, ResNet-18, WRN28_10, and ResNet-50 models for training on the MNIST, CIFAR-10, CIFAR-100, and FOOD-101 datasets, respectively. The programming language is Python, and the training device is a CPUi7-10700, 16GB RAM, and a GeForce GTX 2070 GPU. For the MNIST and CIFAR-10 datasets, the stochastic gradient descent (SGD) optimizer was used for 100 cycles of training. The initial learning rate was set to 0.01, the momentum was 0.9, and the weight decay was 5×10^(-4). The batch size was set to 128. Every 40 cycles, the learning rate was multiplied by 0.1. In addition, data augmentation techniques were used, including random cropping and resizing (with random flipping). For the CIFAR-100 and FOOD-101 datasets, the SGD optimizer was used for 200 cycles of training. The initial learning rate is set to 0.1, the momentum is 0.9, and the weight decay is 5×10^(-4). The batch size is set to 64. Every 30 cycles, the learning rate is multiplied by 0.1. Proposed indicates the method proposed in this application. Other comparison models are existing methods, including TWS, PI, EW, and UW. The evaluation indicator of the robustness experiment in this application is the watermark authentication success rate.

[0113] The trigger mode designed in this application aims to achieve invisibility, which requires the trigger sample to be visually highly similar to the clean sample. Figure 7 (a) and Figure 7 (b) shows the original clean image and the trigger sample generated by the method of this application, respectively, and TWS Figure 7 (c)(d)(e) and PI Figure 7 Compared with the method in (f), the difference can be clearly seen. The trigger sample of this application does not introduce significant artifacts and maintains high visual fidelity. Figure 7 (c) and Figure 7 As shown in (d), the trigger samples of other methods are visually distinguishable from the original images and can be easily detected by the adversary. Figure 7 (e) and Figure 7 The method of using irrelevant and abstract images in (f), although effective in some cases, poses potential security risks because an attacker may find a set of images that also meet the ownership verification requirements by remotely querying the model.

[0114] In order to quantify the visual quality of the trigger samples, this application uses peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indicators. The higher the PSNR value, the closer the image quality is to the original picture. The SSIM value range is between 0 and 1. The closer the value is to 1, the closer the image quality is to the original image. The data in Table 1 show that the average PSNR value of the trigger samples of this application is greater than 38dB, and the average SSIM value is better than 0.98. These results show that the trigger samples designed in this application are not only visually indistinguishable from clean samples, but also perform well in objective evaluation indicators, successfully maintaining the invisibility of the trigger pattern.

[0115] Table 1 shows the comparison with other methods in terms of average peak signal-to-noise ratio and average structural similarity index.

[0116] Table 1

[0117]

[0118] Table 2 shows the robustness experiments of the model proposed in this application and several other schemes under different situations. From the experimental results, it can be seen that the model watermarking scheme proposed in this application is better than the existing schemes under different attacks and is more robust.

[0119] Table 2

[0120]

[0121] This application also proposes a neural network model watermark construction device with invisible trigger mode, such as Figure 8As shown, including:

[0122] Edge module 801: used to extract the image edge of the original image by using an edge extraction algorithm to generate edge information of the original image, wherein the original image is an image selected from the original training set;

[0123] Coding module 802: used to encode the edge information to obtain the RGB value of the edge information, where the RGB value is the trigger mode;

[0124] Injection module 803: used to embed the trigger pattern into the original image information by using the deep injection network combined with the invisibility loss index to generate a trigger sample;

[0125] Fusion module 804: used to generate a fusion training set according to the trigger sample and the original training set;

[0126] Training module 805: used to iteratively train the deep neural network model using a low-weight mask strategy combined with a fusion training set to generate a neural network model watermark with an invisible trigger pattern.

[0127] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0128] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0129] In order to implement the above embodiments, the present application also proposes a terminal device.

[0130] Fig. 9 A schematic diagram of the structure of a terminal device according to an embodiment of the present application.

[0131] like Fig. 9 As shown, the terminal device 900 includes:

[0132] A memory 910 and at least one processor 920, a bus 930 connecting different components (including the memory 910 and the processor 920), the memory 910 stores a computer program, and when the processor 920 executes the program, the neural network model watermark construction method of the invisible trigger mode described in the embodiment of the present application is implemented.

[0133] Bus 930 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0134] The terminal device 900 typically includes a variety of electronic device readable media, which can be any available media that can be accessed by the terminal device 900, including volatile and non-volatile media, removable and non-removable media.

[0135] The memory 910 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 940 and / or cache memory 950. The terminal device 900 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 960 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, usually called a "hard drive"). Although Fig. 9 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to bus 930 via one or more data medium interfaces. Memory 910 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0136] A program / utility 980 having a set (at least one) of program modules 970 may be stored, for example, in the memory 910, such program modules 970 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 970 generally perform the functions and / or methods of the embodiments described herein.

[0137] The terminal device 800 can also communicate with one or more external devices 990 (e.g., keyboard, pointing device, display 991, etc.), and can also communicate with one or more devices that enable users to interact with the terminal device 900, and / or communicate with any device that enables the terminal device 900 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through an input / output (I / O) interface 992. In addition, the terminal device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) through a network adapter 893. As shown in the figure, the network adapter 993 communicates with other modules of the terminal device 800 through a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the terminal device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0138] The processor 920 executes various functional applications and data processing by running the programs stored in the memory 910 .

[0139] It should be noted that the implementation process and technical principles of the terminal device of this embodiment refer to the aforementioned explanation of the invisible trigger mode neural network model watermark construction method of the embodiment of the present application, which will not be repeated here.

[0140] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0141] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0142] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for constructing a watermark of a neural network model with an invisible trigger mode, characterized in that: include: Extracting the image edge of the original image using an edge extraction algorithm to generate edge information of the original image, wherein the original image is an image selected from the original training set; Encode the edge information to obtain the RGB value of the edge information, and the change of the RGB value is the trigger signal of the trigger mode; Obtain the invisible loss index and input sample, use the deep injection network combined with the invisible loss index and the auxiliary guided extraction network to embed the trigger pattern into the original image information, and generate the trigger sample. The trigger sample is generated by connecting the input sample with its corresponding trigger pattern along the channel dimension. The invisible loss index uses L P norm; Generate a fusion training set based on the trigger sample and the original training set; Generate a mask matrix according to the weight distribution of the current layer of the deep neural network model, wherein the mask matrix is ​​dynamically updated in each training cycle; Iteratively training the deep neural network model according to the mask matrix and the fusion training set to generate a neural network model watermark of an invisible trigger mode; The step of generating a mask matrix according to the weight distribution of the current layer of the deep neural network model includes: Calculate the absolute value of the current layer's weight and sort the weight positions according to the absolute value; Mark the weight positions whose absolute values ​​are lower than the preset threshold as 0, and the rest as 1 to generate a mask matrix; The iterative training process includes: a forward propagation stage, wherein the iterative training of the deep neural network according to the mask matrix and the fusion training set includes: Get the weight matrix; Combine the weight matrix and mask matrix using element-by-element multiplication to obtain the corresponding high-weight neurons; Multiply the high-weight neurons by the preset scaling factor to get the output of the deep neural network model; The iterative training process includes: a back propagation stage, in which the deep neural network model is iteratively trained according to the mask matrix and the fusion training set, including: Update the weights of the corresponding high-weight neurons in the mask matrix according to the preset gradient update method.

2. The method according to claim 1, characterized in that The step of extracting the image edge of the original image by using an edge extraction algorithm to generate edge information of the original image includes: The horizontal convolution kernel is used to calculate the gradient of the original image in the horizontal direction to obtain the first gradient value in the horizontal direction; The vertical convolution kernel is used to calculate the gradient of the original image in the vertical direction to obtain a second gradient value in the vertical direction; The first gradient value and the second gradient value are combined to generate edge information of the original image.

3. A neural network model watermark construction device with invisible trigger mode, characterized in that: include: Edge module: used to extract the image edge of the original image using an edge extraction algorithm to generate edge information of the original image, wherein the original image is an image selected from the original training set; Coding module: used for encoding the edge information to obtain the RGB value of the edge information, where the RGB value is the trigger mode; Injection module: used to obtain invisible loss indicators and input samples, and use the deep injection network combined with the invisible loss indicator and the auxiliary guided extraction network to embed the trigger pattern into the original image information to generate trigger samples. The trigger samples are generated by connecting the input samples with their corresponding trigger patterns along the channel dimension. The invisible loss indicator uses L P norm; Fusion module: used to generate a fusion training set based on the trigger sample and the original training set; Update module: used to generate a mask matrix according to the weight distribution of the current layer of the deep neural network model, and the mask matrix is ​​dynamically updated in each training cycle; Training module: used for iteratively training the deep neural network model according to the mask matrix and the fusion training set to generate a neural network model watermark of an invisible trigger mode; Wherein, the update module includes: Sorting unit: used to calculate the absolute value of the current layer weight and sort the weight positions according to the absolute value; Marking unit: used to mark the weight positions whose absolute values ​​are lower than the preset threshold as 0, and the rest as 1, to generate a mask matrix; The iterative training process includes: a forward propagation stage, and the training module includes: Acquisition unit: used to obtain the weight matrix; Preprocessing unit: used to combine the weight matrix and the mask matrix using element-by-element multiplication to obtain the corresponding high-weight neurons; Output unit: used to multiply high-weight neurons by a preset scaling factor to obtain the output result of the deep neural network model; The iterative training process includes: a back propagation stage, and the training module also includes: Update unit: used to update the weights of the corresponding high-weight neurons in the mask matrix according to the preset gradient update method.

4. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 2 is implemented.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images