Watermark protection implementation method and system, watermark model and storage medium

Abstract

The invention provides a watermark protection implementation method and system, a watermark model and a storage medium, and the method comprises the steps: carrying out the incremental training of an original model embedded with watermark information, and obtaining a target training model; detecting whether a watermark signal in the target training model meets a preset evaluation threshold range, and if not, iteratively performing incremental training; otherwise, outputting a training model of the invisible watermark; when a watermark recovery signal is received, carrying out restoration training on the training model of the invisible watermark to obtain a target watermark feature; and judging the strength and the triggering precision of the target watermark feature through the original watermark feature, and outputting target watermark information. According to the method, the embedded watermark information is eliminated through incremental training on the basis of not changing the original functions of the original model, and then the watermark features are restored through restoration training, so that the integrity and the triggering precision of the watermark information are protected in the model training process, and the safety of the model is greatly improved.

Description

Technical Field

[0001] This application belongs to the technical field of watermark protection, and particularly relates to a method for implementing watermark protection, a system, a watermark model, and a storage medium. Background Art

[0002] Currently, some malicious people attempt to utilize deep neural network (DNN) models and profit from them without properly acknowledging the original creators. In this context, the importance of embedding watermarks in DNNs is evident. DNNs have many parameters. Then, how to add additional information without interfering with its main functions has become a current design challenge. In the prior art, watermark embedding is often achieved by modifying the loss function during the training phase, and its effect is manifested through the performance of the watermark model. Subsequently, in order to introduce a confidentiality element during the watermark processing, a key is required. The watermark is encoded using this key so that unauthorized users cannot recover it. However, the watermark embedding methods in the prior art are often insufficient to ensure the invisibility of the watermark, that is, the presence of the watermark may have a significant impact on the quality of the original data or the visual / audio performance; and the robustness of the watermark may also be insufficient, resulting in the watermark being easily lost after some common data processing operations (such as compression, filtering, noise addition, etc.). On the other hand, watermark schemes relying on keys have potential problems in terms of security. If the key is leaked, unauthorized users may be able to easily remove or modify the watermark. At the same time, embedding the watermark may also have a negative impact on the performance of the deep neural network model, such as increasing the training time, reducing the accuracy, or changing the model generalization ability; and, the prior art also cannot effectively handle the differences and interactions between different modal data, resulting in the complication of the watermark embedding and extraction processes. Summary of the Invention

[0003] Aiming at the defects of the above-mentioned prior art, this application provides a method for implementing watermark protection, a system, a watermark model, and a storage medium. By incrementally training to eliminate the embedded watermark information without changing the original functions of the original model, and then restoring the training to recover the watermark features, the integrity and triggering accuracy of the watermark information are protected safely and reliably.

[0004] To achieve the above object, this application provides a method for implementing watermark protection, the method comprising: S100: performing incremental training on the original model embedded with watermark information to obtain a target training model; S200: detecting whether the watermark signal in the target training model satisfies a preset evaluation threshold range, if not, iteratively performing the incremental training; otherwise, outputting the training model of the invisible watermark; S300: when receiving a watermark recovery signal, performing restoration training on the training model of the invisible watermark to obtain target watermark features; S400: determining the intensity and triggering accuracy of the target watermark features through the original watermark features, and outputting the target watermark information.

[0005] In step S100 of the present application, the original model at least includes a trigger set and an original training set; wherein, the watermark information is embedded in the trigger set.

[0006] The incremental training is specifically as follows: extracting any number of training samples from the original training set; extracting any number of trigger samples containing watermark information from the trigger set; merging the training samples and the trigger samples and then outputting a fusion model, and calculating a loss function according to the fusion model; performing gradient descent iterative update on the first preset parameters of the fusion model according to the loss function until the first preset training iteration number or the first preset convergence condition is satisfied.

[0007] In step S100 of the present application, it further includes: the original model further includes a fine-tuning data set; performing fine-tuning training on the updated fusion model to output the target training model. Wherein, the fine-tuning training is specifically as follows: inputting the fine-tuning data set into the updated fusion model to generate a fine-tuning fusion model; calculating a fine-tuning loss function according to the fine-tuning samples in the fine-tuning data set; performing iterative update on the second preset parameters of the fine-tuning fusion model according to the fine-tuning loss function until the second preset training iteration number or the second preset convergence condition is satisfied.

[0008] In step S200 of the present application, detecting whether the watermark signal in the target training model meets a preset evaluation threshold range is specifically as follows: obtaining the watermark signal in the current target training model and inputting it into the evaluation trigger set in the target training model; outputting the intensity of the current watermark information through the trigger accuracy detected by the evaluation trigger set, and when the intensity of the current watermark information is lower than the preset evaluation threshold range, outputting the determination result as meeting the condition; when the intensity of the current watermark information is not lower than the preset evaluation threshold range, outputting the determination result as not meeting the condition.

[0009] In the present application, step S300 is specifically as follows: the watermark recovery signal refers to an instruction signal for generating a recovered watermark feature when a preset condition is triggered.

[0010] The reduction training is specifically as follows: Obtain the weight parameters and bias parameters in the training model of the invisible watermark; After adjusting the trigger setting type according to the weight parameters and bias parameters, restore the weight parameters and bias parameters to the initialization so that they match the trigger type, and select a training data set that satisfies the watermark residual information from the training model of the invisible watermark; Input the initialized weight parameters, bias parameters, and the trigger type into the training data set that satisfies the watermark residual information for training to generate a reduction training model; Extract the target watermark feature from the reduction training model by calculating the feature vector through a preset rule.

[0011] In step S400 of the present application, the specific method for determining the strength of the target watermark feature through the original watermark feature is as follows: Convert the target watermark feature into a first feature vector, and convert the original watermark feature into a second feature vector; Calculate the feature similarity according to the first feature vector and the first feature vector; Determine the strength of the target watermark feature according to the feature similarity.

[0012] In step S400 of the present application, the specific method for determining the triggering accuracy of the target watermark feature through the original watermark feature is as follows: Calculate the first difference measure of the target watermark feature and the second difference measure of the original watermark feature through a preset formula; According to the first difference measure and the second difference measure, obtain the error analysis result of the distribution of the target watermark feature and the original watermark feature in the feature space; Determine the triggering accuracy of the target watermark feature according to the error analysis result.

[0013] To achieve the above object, the present application also provides a watermark protection implementation system, which includes an incremental training unit, a judgment unit, a reduction training unit, and an inspection unit. Among them, the incremental training unit is used to perform incremental training on the original model embedded with watermark information to obtain a target training model. The judgment unit is used to detect whether the watermark signal in the target training model meets the preset evaluation threshold range. If it does not meet, the incremental training is iteratively performed; otherwise, the training model of the invisible watermark is output. The reduction training unit is used to perform reduction training on the training model of the invisible watermark when receiving a watermark recovery signal to obtain a target watermark feature. The inspection unit is used to determine the strength and triggering accuracy of the target watermark feature through the original watermark feature and output the target watermark information.

[0014] To achieve the above object, the present application also proposes a watermark model, which includes a data preparation module, a watermark embedding module, a multi-modal network training module, an elimination module, and a test module. Among them, the data preparation module is used to obtain a data set of multiple modal types. The watermark embedding module is used to embed watermark information into the training data in the data sets of multiple modal types respectively to create a first trigger set. The multi-modal network training module is used to train the data sets of the multiple modal types one by one according to the first trigger set to generate a target model. The elimination module is used to perform incremental training on the target model iteratively to obtain a training model of the invisible watermark. The test module is used to verify the effectiveness of the watermark information by detecting the trigger accuracy of the second trigger set in the training model of the invisible watermark.

[0015] To achieve the above object, the present application also provides a storage medium, which is one of the computer-readable storage media, and has a computer program stored thereon. When the computer program is executed by a processor, it implements the watermark protection implementation method described in any one of the above.

[0016] Compared with the prior art, the beneficial effects of the present application are as follows: The watermark protection implementation method obtains a target training model through incremental training, which improves the robustness and security of the watermark. Because incremental training can gradually adjust the model parameters, making the watermark more difficult to detect and remove. When a watermark recovery signal is received, the training model of the invisible watermark is restored and trained to extract the target watermark features. This improves the extraction efficiency and accuracy of the watermark. Finally, by determining the strength and trigger accuracy of the target watermark features through the original watermark features, the integrity of the restored watermark information can be accurately evaluated. The present application provides an effective method to protect the deep neural network model from unauthorized use and modification, while maintaining the model performance and function, and improving the practicability, security, and robustness of the watermark protection technology. Description of the Drawings

[0017] Figure 1 It is a flowchart of a watermark protection implementation method in an embodiment of the present application.

[0018] Figure 2 It is a framework diagram of a watermark protection implementation system in an embodiment of the present application.

[0019] Figure 3 It is a schematic diagram of a watermark model in an embodiment of the present application. Detailed Embodiments

[0020] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme will be clearly and completely described below in conjunction with the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] Embodiment 1:

[0022] As attached Figure 1 As shown, in order to solve the above technical problems, the present application provides a watermark protection implementation method. In this embodiment,

[0023] The method mainly includes steps S100, S200, S300 and S400.

[0024] S100: Incrementally train the original model embedded with watermark information to obtain a target training model.

[0025] Preferably, the original model in which the watermark information is embedded in this embodiment can be a DNN training model for automotive network security using a deep neural network. Specifically, the original model includes at least a trigger set and an original training set; wherein the watermark information is embedded in the trigger set.

[0026] It should be noted that the trigger set is usually a special data set that contains specific patterns or features used to activate or trigger the watermark information. In the DNN model embedded with watermarks, these data are usually used under specific conditions to reveal the hidden watermark information.

[0027] Preferably, a model for detecting abnormal driving behavior in an application for automotive network security is assumed. In this scenario, the trigger set may be a set of specific driving patterns, such as continuous sharp turns, sudden braking, or abnormal speed changes. These patterns are not common in normal driving, but are used as trigger conditions for watermarks during model training. When the model processes data containing trigger set features, the watermark information is activated and can be used to verify copyright, track model usage, or detect unauthorized copying. Due to the particularity of the trigger set, the watermark information remains hidden in data that does not contain these features, increasing the concealment of the watermark. A well-designed trigger set can reduce the possibility of the watermark being mistakenly activated during normal operation.

[0028] The original training set is a standard dataset used to train the DNN model. It contains the normal operation modes that the model needs to learn in order to perform its basic tasks (such as classification, detection, etc.). Preferably, in the case of an automotive network security model, the original training set may contain a large number of video clips of normal driving behaviors, which demonstrate the normal operations of the vehicle in different environments and at different times. The original training set is used to train the model to perform its core functions, such as identifying normal driving behaviors. During the process of embedding the watermark, the original training set may be used to ensure that the watermark does not interfere with the main functions of the model.

[0029] In this embodiment, since the watermark is only activated under specific conditions (i.e., when processing the trigger set data), in most cases, the watermark remains invisible to the normal use of the model. Combining the watermark with the trigger set features can improve its robustness during the model training and inference processes, making it less likely to be removed or modified. The specific pattern of the trigger set is the key to activating the watermark, which adds an extra layer of security protection for the watermark protection and prevents unauthorized use.

[0030] In summary, the technology provided by the embodiments of this application can effectively embed and protect watermark information without sacrificing the model performance, while ensuring the invisibility and robustness of the watermark.

[0031] Furthermore, the incremental training is specifically as follows: extracting any number of training samples from the original training set; extracting any number of trigger samples containing watermark information from the trigger set; merging the training samples and the trigger samples and then outputting a fusion model, and calculating a loss function according to the fusion model; performing gradient descent iterative update on the first preset parameters of the fusion model according to the loss function until the first preset training iteration number or the first preset convergence condition is satisfied.

[0032] It should be noted that by combining the samples in the original training set and the trigger set, the model is further trained to update the parameters of the model, so as to enhance the embedding and triggering capabilities of the watermark while maintaining the original functions of the model. In this embodiment, samples are extracted from the original training set, and these samples represent the normal behaviors that the model needs to learn. Samples containing watermark information are extracted from the trigger set, and these samples are used to activate and embed the watermark. By using these two types of samples simultaneously, it can be ensured that the model performs well under normal circumstances and can reliably activate the watermark under the watermark trigger conditions.

[0033] In this embodiment, the extracted original training samples and trigger samples are merged to create a new dataset for training the fusion model. The fusion model can learn the normal behaviors and the watermark trigger conditions, which helps to improve the invisibility and robustness of the watermark.

[0034] Calculate the loss function according to the fusion model. This loss function usually consists of two parts: one part is for the original task (such as classification accuracy), and the other part is for the specific goal of watermark embedding. The loss function is the key to guiding the model training, which ensures that the model will neither sacrifice the performance of the original task nor be able to effectively embed the watermark when updating the parameters. Use the gradient descent algorithm to iteratively update the first preset parameters of the fusion model to minimize the loss function. By iteratively updating the parameters, the model can better learn how to make predictions under normal circumstances and watermark trigger conditions. Gradient descent is an effective optimization method that can help the model quickly converge to the optimal parameters. The training process will continue until the first preset number of training iterations or the first preset convergence condition is met. The preset conditions help control the training process, prevent overfitting, and at the same time ensure that the model is fully trained. Ensure that the model stops training after reaching the predetermined accuracy or performance level to avoid unnecessary waste of computing resources.

[0035] Preferably, assume that a model for image classification is trained while hoping to embed a watermark to protect the copyright. Extract 1000 normal pictures from the original training set and 100 special pictures from the trigger set. These pictures contain a tiny and invisible watermark pattern in a specific area. Combine these 1100 pictures into a new training set for training the fusion model. Design a loss function that takes into account both the classification accuracy and the strength and invisibility of the watermark embedding. Use the gradient descent algorithm to iteratively update the model parameters until 10,000 iterations are reached or the value of the loss function is lower than a certain threshold. The final obtained fusion model can accurately classify pictures under normal circumstances and reliably activate the watermark under specific conditions (i.e., when processing the pictures in the trigger set).

[0036] In this way, the model can not only perform its main task but also effectively embed and protect the watermark information.

[0037] In step S100 of this application, it also includes: there is also a fine-tuning data set in the original model; perform fine-tuning training on the updated fusion model to output the target training model.

[0038] Among them, the fine-tuning training is specifically: input the fine-tuning data set into the updated fusion model to generate a fine-tuned fusion model; calculate the fine-tuning loss function according to the fine-tuning samples in the fine-tuning data set; iteratively update the second preset parameters of the fine-tuned fusion model according to the fine-tuning loss function until the second preset number of training iterations or the second preset convergence condition is met.

[0039] It should be noted that fine-tuning training is an important step that follows incremental training and is used to further improve the performance of the model and ensure the stability of the watermark. The fine-tuning dataset is a set of specially selected data used to further adjust the parameters of the model after incremental training. The fine-tuning dataset may contain new samples that the model has not seen during incremental training, which helps the model better generalize to new data distributions. Fine-tuning can correct any biases or overfitting that may be introduced during incremental training. The fine-tuning dataset is input into the updated fusion model to generate a fine-tuned fusion model. Through fine-tuning, the model can further optimize its parameters to improve performance on specific tasks.

[0040] Calculate the fine-tuning loss function based on the samples in the fine-tuning dataset. This loss function usually focuses on the main task of the model and may also include watermark-related objectives. The fine-tuning loss function guides the model on how to adjust its parameters to better perform the original task and maintain the stability of the watermark. Use the fine-tuning loss function to iteratively update the second preset parameters of the fine-tuned fusion model. Iterative updating helps the model further refine its parameters during the fine-tuning stage to improve accuracy and robustness.

[0041] By controlling the number of iterations and convergence conditions, overfitting of the model on the fine-tuning dataset can be avoided.

[0042] Fine-tuning training continues until the second preset number of training iterations or the second preset convergence condition is met. The preset conditions ensure that the fine-tuning process is not too long while ensuring that the model performance reaches a satisfactory level.

[0043] Preferably, assume that a deep learning model for face recognition needs to be developed and a watermark is desired to be embedded to track the use of the model. A fine-tuning dataset containing facial images under different lighting conditions and at different angles is prepared. After completing incremental training, this fine-tuning dataset is used to fine-tune the model. Then a loss function is designed that not only focuses on the accuracy of face recognition but also on the stability of the watermark information. The fine-tuning loss function is used to iteratively update the parameters of the model, such as adjusting the weights of the convolutional layer and the fully connected layer. Fine-tuning training is carried out, for example, until 5000 iterations are reached or the value of the loss function is lower than a very small threshold.

[0044] Through this process, the model ensures that the watermark remains stable and reliable after fine-tuning while maintaining a high face recognition accuracy. Fine-tuning training helps the model better adapt to the actual application scenario while maintaining the invisibility and robustness of the watermark.

[0045] S200: Detect whether the watermark signal in the target training model meets the preset evaluation threshold range. If not, iterate the incremental training; otherwise, output the training model of the invisible watermark.

[0046] In step S200 of the present application, whether the watermark signal in the detection target training model meets the preset evaluation threshold range is specifically: the watermark signal in the current target training model is obtained and input into the evaluation trigger set in the target training model; the intensity of the current watermark information is output through the trigger accuracy detected by the evaluation trigger set, and when the intensity of the current watermark information is lower than the preset evaluation threshold range, the output judgment result is that the condition is met; when the intensity of the current watermark information is not lower than the preset evaluation threshold range, the output judgment result is that the condition is not met.

[0047] It should be noted that this application provides a detection mechanism for evaluating the strength and quality of the watermark signal embedded in the target training model. First, the watermark signal in the current target training model is obtained. After the training is completed, the watermark signal in the model needs to be extracted for evaluation. Ensure that the watermark information can be evaluated independently without affecting the normal function of the model.

[0048] The evaluation trigger set is a set of specially designed data used to activate and evaluate the watermark signal in the model. The evaluation trigger set can be used to activate the watermark signal in the model for strength evaluation. The evaluation trigger set is used to detect the trigger accuracy of the watermark signal, that is, the degree of activation of the watermark under specific conditions. The strength of the watermark signal is measured by the trigger accuracy to determine whether it is strong enough to be difficult to remove or ignore. In this embodiment, an evaluation threshold range can be set to determine whether the strength of the watermark signal meets the requirements. The threshold range provides a quantitative standard for determining whether the watermark has reached the expected strength. The judgment result is output based on the comparison result of the strength of the watermark information with the preset evaluation threshold. The judgment result determines whether further iterations are required for incremental training to ensure the strength and stability of the watermark.

[0049] Preferably, assume that a deep learning model for speech recognition is now developed, and a watermark is embedded in it to identify the copyright of the model. After the training is completed, the embedded watermark signal is extracted from the model. Then, an evaluation trigger set containing specific speech patterns is prepared, which can activate the watermark in the model. The evaluation trigger set is input into the model to detect the trigger accuracy of the watermark signal. For example, if 95 out of 100 samples in the evaluation trigger set successfully trigger the watermark, then the trigger accuracy is 95%. Among them, the set evaluation threshold range can be customized according to actual conditions, such as requiring the trigger accuracy to reach at least 90%, but it is not limited to this. If the trigger accuracy is above 90%, we believe that the watermark strength meets the conditions and the output judgment result is that the conditions are met; if it is lower than 90%, the judgment result is that the conditions are not met and iterative incremental training is required.

[0050] In summary, this embodiment can ensure that the embedded watermark is robust enough in the model and can be effectively activated when necessary. If the watermark strength is insufficient, we can enhance it through iterative incremental training until it meets the preset evaluation threshold range, thereby outputting a trained model with an invisible watermark.

[0051] Preferably, in another preferred embodiment, during the preparation of training data and fine-tuning data, from the CIFAR10 data training set containing 50k images, it can be split into 25k images for training and 25k images for fine-tuning or incremental training. During the trigger data phase, among the 25k training images, 100 images are randomly selected from this set to generate trigger images. The trigger images can be generated by adding Gaussian noise, embedding text, and adding adversarial noise to them. There is also another method to generate trigger images by using a completely different data distribution, such as using 100 images from the MNIST dataset as trigger images, which is not limited to this. The test data is the public test set of CIFAR10, consisting of 10k images.

[0052] During the watermark embedding training phase, by simply mixing the trigger data with the training data and training the DNN model on this mixed dataset. This embodiment only uses fine-tuning data to train the model. Due to the difference in data distribution, the backdoor-based watermark will be eroded over time. By evaluating the accuracy of the trigger set, that is, the reduction of the watermark can be observed, thereby evaluating the strength and quality of the watermark signal embedded in the target trained model.

[0053] S300: When receiving the watermark recovery signal, perform restoration training on the trained model with the invisible watermark to obtain the target watermark feature.

[0054] In this application, the step S300 is specifically: the watermark recovery signal refers to an instruction signal for restoring the watermark feature generated when a preset condition is triggered.

[0055] The restoration training is specifically: obtain the weight parameters and bias parameters in the trained model with the invisible watermark. After adjusting the trigger setting type according to the weight parameters and bias parameters, restore the weight parameters and bias parameters to the initialization so that they match the trigger type, and screen out a training data set that meets the watermark residual information from the trained model with the invisible watermark. Input the initialized weight parameters and bias parameters, as well as the trigger type, into the training data set that meets the watermark residual information for training to generate a restored training model; extract the target watermark feature from the restored training model by calculating the feature vector through a preset rule.

[0056] It should be noted that the watermark recovery signal is an instruction signal that is triggered when a preset condition is met, indicating that the system starts the watermark recovery process. When it is necessary to verify copyright, detect unauthorized copying, or perform other watermark-related operations, the watermark recovery signal activates the recovery process. The weights and bias parameters are extracted from the training model of the invisible watermark, and then the current state of the model is saved for subsequent restoration training. Then, the settings of the trigger are adjusted according to the weight parameters and bias parameters to match the state of the model. In this embodiment, by ensuring that the type of the trigger is consistent with the watermark embedding method in the model, the watermark can be correctly activated. The weights and bias parameters are restored to the initial state to match the trigger type. By resetting the parameters, a clean starting point is provided for the restoration training. A training data set containing watermark residual information is selected from the training model of the invisible watermark. Then, a suitable data set is selected, which contains information that can activate and restore the watermark. Using the initialized parameters and trigger type, training is performed on the selected data set to generate a restoration training model. Through training, the watermark features are activated and strengthened. The target watermark features are extracted from the restoration training model by calculating the feature vectors according to the preset rules. The specific watermark features are extracted from the model for verification, tracking, or other purposes.

[0057] Preferably, assume that in an image classification model embedded with an invisible watermark, it is now necessary to extract watermark features to verify copyright. When a specific copyright verification request is received, the system generates a watermark recovery signal, and extracts the current weights and bias parameters from the model. The triggers are adjusted according to the state of the model to ensure that they can correctly activate the watermark. Then, the weights and bias parameters are restored to the initial state to prepare for the restoration training. The images containing watermark information are selected from the training data. Training is performed using the initial parameters and the selected data set to activate and enhance the watermark features. Specific watermark features, such as specific image patterns or signals, are extracted from the model after restoration training through preset rules and algorithms.

[0058] Through the above process, this embodiment can recover and extract the target watermark features from the training model for copyright verification or other related purposes. This ensures that the watermark can be reliably recovered and verified when needed.

[0059] S400: Determine the strength and trigger accuracy of the target watermark features based on the original watermark features, and output the target watermark information.

[0060] In step S400 of this application, the determination of the strength of the target watermark features based on the original watermark features is specifically as follows: convert the target watermark features into a first feature vector, and convert the original watermark features into a second feature vector; calculate the feature similarity according to the first feature vector and the first feature vector; determine the strength of the target watermark features according to the feature similarity.

[0061] It should be noted that in step S400 of the present application, the trigger accuracy for determining the target watermark feature from the original watermark feature is specifically as follows: calculating a first difference metric of the target watermark feature and a second difference metric of the original watermark feature through a preset formula; obtaining an error analysis result of the distribution of the target watermark feature and the original watermark feature in the feature space according to the first difference metric and the second difference metric; and determining the trigger accuracy of the target watermark feature according to the error analysis result.

[0062] It should be noted that the embodiments of the present application describe how to determine the strength and trigger accuracy of the target watermark feature by comparing the similarity between the target watermark feature and the original watermark feature. Among them, the target watermark feature and the original watermark feature need to be converted into numerical feature vectors for comparison. Converting the watermark feature into a feature vector enables quantitative comparison of their similarity. According to the converted first feature vector and second feature vector, calculate the similarity between them. Feature similarity is a measure used to evaluate the closeness between the target watermark feature and the original watermark feature. Use the feature similarity as a basis to judge the strength of the target watermark feature.

[0063] In this embodiment, the higher the similarity, the closer the target watermark feature is to the original watermark feature, and thus it can be determined that its strength is higher.

[0064] Preferably, assume that a watermark based on a specific image texture is embedded in an image processing model. Convert the target watermark feature (extracted from the restored training model) into a feature vector (the first feature vector). Similarly, convert the original watermark feature (the watermark feature before embedding) into a feature vector (the second feature vector). Use cosine similarity, Euclidean distance, or other similarity measurement methods to calculate the similarity between the two feature vectors, and it is not limited to this. For example, if the cosine similarity is close to 1, it indicates that the two feature vectors are very similar. Then set a similarity threshold, such as 0.9. If the calculated similarity is greater than or equal to this threshold, we determine that the target watermark feature has high strength; if the similarity is lower than the threshold, it indicates that the target watermark feature may have been damaged or interfered with and has low strength.

[0065] In summary, this embodiment can accurately evaluate the strength of the target watermark feature. This evaluation is crucial for ensuring the stability and reliability of the watermark during the distribution and use of the model. At the same time, it can also be used to detect whether the model has been tampered with or unauthorizedly copied. Feature similarity provides a quantitative indicator to help users verify the effectiveness of the watermark legally or commercially.

[0066] Embodiment 2:

[0067] As shown in the appendixFigure 2 As shown, to solve the above technical problems, the present application also provides a watermark protection implementation system, which mainly includes: an incremental training unit, a judgment unit, a restoration training unit, and an inspection unit.

[0068] It should be noted that the watermark protection implementation system provided by the embodiments of the present application is composed of four key units, and each unit undertakes a specific function to ensure the effective embedding, detection, and restoration of watermarks.

[0069] Among them, the incremental training unit is used to perform incremental training on the original model embedded with watermark information to obtain a target training model.

[0070] It should be noted that through incremental training, the model can enhance the embedding and triggering capabilities of watermarks without sacrificing the main functions. Incremental training helps the model better learn how to make predictions under normal circumstances and watermark triggering conditions.

[0071] The judgment unit is used to detect whether the watermark signal in the target training model meets the preset evaluation threshold range. If it does not meet, the incremental training is iteratively performed; otherwise, the training model of the invisible watermark is output.

[0072] It should be noted that the judgment unit ensures that the intensity and triggering accuracy of the watermark signal meet the preset standards. If the threshold range is not met, the incremental training is iteratively performed until the expected watermark intensity is reached.

[0073] The restoration training unit is used to perform restoration training on the training model of the invisible watermark when receiving a watermark restoration signal to obtain target watermark features.

[0074] It should be noted that the restoration training unit is used for watermark restoration. When needed, the watermark features are restored from the model through restoration training. Through the extracted target watermark features, the copyright of the model can be verified or its usage can be traced.

[0075] The inspection unit is used to determine the intensity and triggering accuracy of the target watermark features through the original watermark features and output the target watermark information.

[0076] It should be noted that the inspection unit is used for intensity evaluation, evaluating the intensity of the target watermark features to ensure that they are sufficient to resist common data processing operations. By confirming the activation accuracy of the watermark under specific triggering conditions, its reliability is ensured.

[0077] Preferably, assume that a deep learning model for speech recognition is currently being developed and it is desired to embed a watermark to protect the copyright. Through the system of this embodiment, incremental training is performed on the original model with the embedded watermark to ensure the invisibility and robustness of the watermark. Then, it is detected whether the watermark signal in the model meets the preset evaluation threshold range. If not, we continue with the incremental training until the watermark strength meets the standard. When copyright verification is required, a watermark recovery signal is received, and the model is restored and trained to extract the target watermark features. Finally, the system evaluates the strength and trigger accuracy of the target watermark features by using the original watermark features and outputs the results to confirm the effectiveness of the watermark.

[0078] In summary, the system can ensure that the watermark in the model is both invisible and robust, and can be reliably recovered and verified when needed. This is crucial for protecting intellectual property rights and tracking the use of the model.

[0079] Embodiment 3:

[0080] As shown in the appendix Figure 3 To solve the above technical problems, the present application also provides a watermark model applied to a multimodal network. It should be noted that when dealing with a multimodal network, the task of embedding a watermark becomes significantly more complex. In the context of a multimodal network, the use of various cross-modal trigger data generation techniques can cause the watermark to be initially removed through incremental training, but then the watermark can be reintroduced by retraining the network using the original training data. In this embodiment, according to the robustness of the watermark technology, it is applied to a multimodal DNN model, greatly improving the efficiency and accuracy of the watermarking process.

[0081] In this embodiment, the watermark model mainly includes: a data preparation module, a watermark embedding module, a multimodal network training module, an elimination module, and a quiz module. The watermark model of this embodiment works in cooperation through a series of modules to ensure that the watermark can be effectively embedded and verified in a deep neural network (DNN) model when processing multimodal data.

[0082] Among them, the data preparation module is used to obtain a dataset of multiple modal types. In this embodiment, a dataset of multiple modal types is first obtained. Different modal data, such as text, images, audio, etc., are collected to prepare for watermark embedding and model training. Ensure that the dataset covers all relevant modalities that the model will process for effective multimodal training.

[0083] The watermark embedding module is used to embed watermark information into the training data in the dataset of multiple modal types respectively to create a first trigger set. Embed watermark information in multiple modal data to ensure that the watermark can be detected and triggered in different types of data. Generate a first trigger set, and these data will be used to activate and verify the watermark.

[0084] The multimodal network training module is used to train the datasets of the multiple modality types one-to-one according to the first trigger set, and generate a target model. The training model is used to process and understand data of different modalities. Through training, a target model capable of processing multiple modality data simultaneously is generated.

[0085] The elimination module is used to perform the incremental training on the target model iteratively to obtain the training model of the invisible watermark. Through the incremental training, the watermark is made invisible during the normal use of the model and does not affect the main functions of the model, and its robustness is enhanced so that it can resist common data processing operations.

[0086] The testing module is used to verify the effectiveness of the watermark information by detecting the triggering accuracy of the second trigger set in the training model of the invisible watermark. The testing module ensures that the embedded watermark can be accurately triggered and detected under specific conditions. The triggering accuracy of the watermark is evaluated to ensure its reliability in practical applications.

[0087] Preferably, in a preferred embodiment, assume a training scenario with MNIST image data and Free spoke digital audio data. It is necessary to introduce the generated trigger set into the dataset with Gaussian noise, and then perform the watermark process.

[0088] The first step: Data preparation stage. In the MNIST dataset, there are 60,000 training images and 10,000 test images of handwritten digits from 0 to 9; and in the free speech digital dataset, there are voice digital recordings from 0 to 9, with a total of 2,000 recordings.

[0089] The second step: Watermark embedding stage. In terms of image data, for MNIST images, a trigger set can be created by introducing Gaussian noise into a subset of the training images; in terms of audio data, another trigger set can also be created by adding Gaussian noise to a subset of the voice digital recordings.

[0090] The third step: Training the multi-modal network stage. When training a multi-modal network using MNIST images and free-spoken digital audio data, the trigger set during the training process will be used.

[0091] The fourth step: Fine-tuning stage. After the initial training, the model is fine-tuned to gradually erode the watermark.

[0092] The fifth step: Testing and verification stage. This stage can verify the effectiveness of the watermark technology by checking the accuracy of the trigger set. If the watermark is retained, the model should perform well for the trigger set, even after fine-tuning.

[0093] Preferably, in another embodiment, assume that a multi-modal sentiment analysis model is being developed, which needs to process text and audio data.

[0094] At the data preparation module stage, a large amount of text and audio data containing sentiment labels are collected for training the model.

[0095] Then, watermark information is respectively embedded into the text and audio data in the watermark embedding module to create a first trigger set. The model is trained using the first trigger set so that it can process text and audio data simultaneously. Then, the model is incrementally trained to hide the watermark while maintaining its multi-modal processing ability. Finally, the trigger accuracy of the watermark in the model is tested by using a second trigger set to ensure that the watermark can be effectively activated when necessary.

[0096] In summary, through the watermark model of this embodiment, it is possible to effectively embed and verify watermarks when processing complex multi-modal data, protecting the copyright of the model and usage tracking.

[0097] Embodiment 4:

[0098] To solve the above technical problems, the present application also provides a storage medium, which is one of the computer-readable storage media, on which a computer program is stored. When the computer program is executed by a processor, the watermark protection implementation method as described in any one of the above is implemented.

[0099] It should be noted that the storage medium of this embodiment contains the computer program code for implementing the watermark protection method, including instructions for each step such as data preparation, watermark embedding, model training, incremental training, restoration training, detection, and verification. It can persistently store programs and data, and the stored information will not be lost even in the case of power failure or other interruptions. The storage medium can be easily transferred between different computer systems, enabling the program to be executed in different environments.

[0100] When the processor executes the program stored on the medium, it executes each step of the watermark protection implementation method according to the instructions of the program. The program in the storage medium can help protect deep learning models from unauthorized copying and use. Through the stored watermark protection program, the usage of the model can be traced, helping the copyright owner monitor the distribution of their works.

[0101] Preferably, taking a solid-state drive (SSD) as an example of the storage medium, the computer program code for implementing the watermark model is stored in the SSD. The SSD has non-volatile storage characteristics, ensuring that programs and data are not lost after a power outage. The SSD can be easily moved from one computer to another. When the processor of the computer reads the program code on the SSD, it performs operations such as watermark embedding and model training according to the instructions in the code. The program stored on the SSD can help embed and verify the watermark, thereby protecting the model from being illegally copied. Through the program in the SSD, the usage history of the model can be recorded to help the copyright owner track the usage of the model.

[0102] In summary, the technology provided by the embodiments of the present application can implement a consistent watermark protection function on different computer systems, ensuring that the copyright and security of the model are effectively protected.

[0103] The so-called processor may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc. The memory is used to store the operating system, application programs, boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory may also be used to temporarily store the data that has been output or will be output.

[0104] In several embodiments provided in the present application, it can be understood that each block in the flowchart or block diagram may represent a module, a program segment, or a part of the code, and the module, the program segment, or the part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the block may occur in a different order from that marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved.

[0105] When the above-mentioned functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0106] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not used to limit the protection scope of this application. In particular, it is pointed out that for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this application shall be included in the protection scope of this application.

Claims

1. A method for implementing watermark protection, characterized in that: include: S100: incrementally training the original model embedded with watermark information to obtain a target training model; S200: Detecting whether the watermark signal in the target training model meets a preset evaluation threshold range, and if not, iteratively performing the incremental training; Otherwise, output the training model of invisible watermark; S300: When a watermark recovery signal is received, the training model of the invisible watermark is restored to obtain target watermark features; S400: Determine the strength and triggering accuracy of the target watermark feature through the original watermark feature, and output the target watermark information.

2. The method for implementing watermark protection according to claim 1, characterized in that: In step S100, The original model at least includes a trigger set and an original training set; wherein the watermark information is embedded in the trigger set; and the incremental training is specifically: Extracting any number of training samples from the original training set; Extracting any number of trigger samples containing watermark information from the trigger set; The training samples and trigger samples are combined to output a fusion model, and a loss function is calculated based on the fusion model; a first preset parameter of the fusion model is iteratively updated by gradient descent according to the loss function until a first preset number of training iterations or a first preset convergence condition is met.

3. The method for implementing watermark protection according to claim 2, characterized in that: In the step S100, it further includes: the original model also includes a fine-tuning data set; Fine-tune the updated fusion model to output the target training model; The fine-tuning training is specifically as follows: Inputting the fine-tuning data set into the updated fusion model to generate a fine-tuning fusion model; According to the fine-tuning samples in the fine-tuning dataset, the fine-tuning loss function is calculated; The second preset parameters of the fine-tuning fusion model are iteratively updated according to the fine-tuning loss function until a second preset number of training iterations or a second preset convergence condition is met.

4. The method for implementing watermark protection according to claim 3, characterized in that: In step S200, the detection of whether the watermark signal in the target training model meets the preset evaluation threshold range is specifically: Obtain the watermark signal in the current target training model and input it into the evaluation trigger set in the target training model; Outputting the strength of the current watermark information through the trigger accuracy detected by the evaluation trigger set, and when the strength of the current watermark information is lower than the preset evaluation threshold range, outputting a determination result that a condition is met; When the strength of the current watermark information is not lower than the preset evaluation threshold range, the output determination result is that the condition is not met.

5. The method for implementing watermark protection according to claim 4, characterized in that: The step S300 is specifically as follows: The watermark restoration signal refers to: a command signal for restoring watermark features generated when a preset condition is triggered; The restoration training is specifically as follows: Obtain weight parameters and bias parameters in the training model of the invisible watermark; After adjusting the trigger setting type according to the weight parameter and the bias parameter, the weight parameter and the bias parameter are restored to the initialization so that they match the trigger type, and a training data set that satisfies the watermark residual information is selected from the training model of the invisible watermark; Inputting the initialized weight parameters and bias parameters, as well as the trigger type, into the training data set satisfying the watermark residual information for training to generate a restoration training model; The target watermark feature is extracted from the restoration training model by calculating the feature vector according to preset rules.

6. The method for implementing watermark protection according to claim 5, characterized in that: In the step S400, the strength of the target watermark feature is determined by the original watermark feature, specifically: Converting the target watermark feature into a first feature vector, and converting the original watermark feature into a second feature vector; Calculating feature similarity based on the first feature vector and the first feature vector; The strength of the target watermark feature is determined according to the feature similarity.

7. The method for implementing watermark protection according to claim 6, characterized in that: In step S400, the triggering accuracy of the target watermark feature is determined by the original watermark feature, specifically: A first difference metric of the target watermark feature and a second difference metric of the original watermark feature are calculated by a preset formula; and an error analysis result of the distribution of the target watermark feature and the original watermark feature in the feature space is obtained according to the first difference metric and the second difference metric; The triggering accuracy of the target watermark feature is determined according to the error analysis result.

8. A watermark protection implementation system, characterized in that: It includes an incremental training unit, a judgment unit, a restoration training unit and a testing unit; The incremental training unit is used to perform incremental training on the original model embedded with the watermark information to obtain a target training model; the judgment unit is used to detect whether the watermark signal in the target training model meets the preset evaluation threshold range, and if not, iteratively perform the incremental training; Otherwise, output the training model of invisible watermark; The restoration training unit is used to perform restoration training on the training model of the invisible watermark to obtain target watermark features when receiving the watermark restoration signal; And, the verification unit is used to determine the strength and triggering accuracy of the target watermark feature through the original watermark feature, and output the target watermark information.

9. The watermark model of the watermark protection implementation method according to any one of claims 1 to 7, characterized in that: It includes data preparation module, watermark embedding module, multimodal network training module, elimination module and test module; The data preparation module is used to obtain data sets of multiple modality types; The watermark embedding module is used to embed watermark information into the training data in the data sets of multiple modal types respectively to create a first trigger set; The multimodal network training module is used to train the data sets of the multiple modal types in a one-to-one correspondence according to the first trigger set to generate a target model; The elimination module is used to iteratively perform the incremental training on the target model to obtain a training model of the invisible watermark; And, the testing module is used to verify the validity of the watermark information by detecting the trigger accuracy of the second trigger set in the training model of the invisible watermark.

10. A storage medium, which is a computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the watermark protection implementation method as claimed in any one of claims 1 to 7 is implemented.

Images