Model watermark generation, model infringement identification method, device and computer equipment

By generating and embedding model watermarks and combining with copyright information extraction models, the risk of deep learning models being infringed is solved, and the protection and security of model copyright is achieved.

CN114331791BActive Publication Date: 2024-12-06TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111188742.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-12
Publication Date
2024-12-06
Estimated Expiration
2041-10-12

AI Technical Summary

Technical Problem

Deep learning models are at risk of infringement after training, and attackers can steal model information through alternative models. It is difficult for existing technology to effectively protect the copyright and security of the model.

Method used

By obtaining model copyright information and sample prediction information, using the copyright information extraction model for integration and update, generating the target model watermark, and embedding it in the model to be protected, combining the copyright information extraction model for model copyright protection, ensuring the security of the model watermark .

Benefits of technology

It improves the security of the model, protects the copyright of the model, prevents the model from being unauthorized use and stealing, and realizes the identification and protection of model infringement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114331791B_ABST
    Figure CN114331791B_ABST
Patent Text Reader

Abstract

The present application relates to a method, device, computer equipment and storage medium for model watermark generation and model infringement identification. Embodiments of the present invention can be applied to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios. The method includes: inputting the current fusion information corresponding to the current sample prediction information of the model to be protected into the current copyright information extraction model to obtain the initial predicted copyright information, the initial fusion information is obtained by fusing the sample prediction information and the corresponding initial model watermark, updating the current fusion information based on the model copyright information and the initial predicted copyright information to obtain new current fusion information, returning to the step of inputting the current fusion information into the model until the convergence condition is met, obtaining the target fusion information, obtaining the target model watermark based on the sample prediction information and the corresponding target fusion information, and embedding it into the model to be protected to obtain the target model, the target model and the current copyright information extraction model are used to protect the model copyright of the model to be protected.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, computer equipment and storage medium for model watermark generation and model infringement identification. Background Art

[0002] With the continuous development of artificial intelligence technology, the research results of deep learning have been widely used in natural language processing, image recognition, industrial control, signal processing, security and other fields.

[0003] Currently, training a deep learning model with good performance often requires a large amount of high-quality data and expensive computing resources. However, the trained model is at risk of being infringed. For example, an attacker can steal the model information of the trained model by using an alternative model to imitate the trained model. Summary of the invention

[0004] Based on this, it is necessary to provide a model watermark generation, model infringement identification method, device, computer equipment and storage medium that can improve model security and protect model copyright to address the above technical problems.

[0005] A model watermark generation method, the method comprising:

[0006] Obtaining model copyright information corresponding to the model to be protected, and obtaining current fusion information corresponding to the current sample prediction information of the model to be protected; wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0007] Input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information;

[0008] Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0009] Obtaining a target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information;

[0010] The target model watermark is embedded into the model to be protected to obtain the target model. The target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0011] A model watermark generating device, the device comprising:

[0012] An information acquisition module, used to acquire model copyright information corresponding to the model to be protected, and to acquire current fusion information corresponding to the current sample prediction information of the model to be protected; wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0013] An information prediction module, used to input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information;

[0014] An information updating module, used to update the current fusion information based on the initial information difference between the model copyright information and the initial predicted copyright information, obtain updated fusion information, use the updated fusion information as the current fusion information, return to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model, execute until the convergence condition is met, and obtain the target fusion information corresponding to the current sample prediction information;

[0015] A watermark determination module, configured to obtain a target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information;

[0016] The watermark embedding module is used to embed the target model watermark into the model to be protected to obtain the target model. The target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Obtaining model copyright information corresponding to the model to be protected, and obtaining current fusion information corresponding to the current sample prediction information of the model to be protected; wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0019] Input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information;

[0020] Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0021] Obtaining a target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information;

[0022] The target model watermark is embedded into the model to be protected to obtain the target model. The target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0024] Obtaining model copyright information corresponding to the model to be protected, and obtaining current fusion information corresponding to the current sample prediction information of the model to be protected; wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0025] Input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information;

[0026] Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0027] Obtaining a target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information;

[0028] The target model watermark is embedded into the model to be protected to obtain the target model. The target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0029] A model infringement identification method, the method comprising:

[0030] Obtaining the model to be authenticated and the model copyright information, and obtaining the current copyright information extraction model corresponding to the model copyright information;

[0031] Based on the current copyright information extraction model and the model copyright information, the to-be-identified model is subjected to infringement identification to obtain an infringement identification result;

[0032] The current copyright information extraction model is used to extract the model copyright information from the target input data, the target input data includes the target model watermark corresponding to the model copyright information, and the generation process of the target model watermark is as follows:

[0033] Input the current fusion information corresponding to the current sample prediction information of the model to be protected into the current copyright information extraction model to obtain the initial predicted copyright information; the initial fusion information corresponding to the current sample prediction information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0034] Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0035] A target model watermark is obtained based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information.

[0036] A model infringement identification device, comprising:

[0037] A data acquisition module, used to acquire the model to be identified and the model copyright information, and to acquire the current copyright information extraction model corresponding to the model copyright information;

[0038] A model infringement identification module is used to extract the model and model copyright information based on the current copyright information, perform infringement identification on the model to be identified, and obtain an infringement identification result;

[0039] The current copyright information extraction model is used to extract the model copyright information from the target input data, the target input data includes the target model watermark corresponding to the model copyright information, and the generation process of the target model watermark is as follows:

[0040] Input the current fusion information corresponding to the current sample prediction information of the model to be protected into the current copyright information extraction model to obtain the initial predicted copyright information; the initial fusion information corresponding to the current sample prediction information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0041] Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0042] A target model watermark is obtained based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information.

[0043] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0044] Obtaining the model to be authenticated and the model copyright information, and obtaining the current copyright information extraction model corresponding to the model copyright information;

[0045] Based on the current copyright information extraction model and the model copyright information, the to-be-identified model is subjected to infringement identification to obtain an infringement identification result;

[0046] The current copyright information extraction model is used to extract the model copyright information from the target input data, the target input data includes the target model watermark corresponding to the model copyright information, and the generation process of the target model watermark is as follows:

[0047] Input the current fusion information corresponding to the current sample prediction information of the model to be protected into the current copyright information extraction model to obtain the initial predicted copyright information; the initial fusion information corresponding to the current sample prediction information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0048] Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0049] A target model watermark is obtained based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0051] Obtaining the model to be authenticated and the model copyright information, and obtaining the current copyright information extraction model corresponding to the model copyright information;

[0052] Based on the current copyright information extraction model and the model copyright information, the to-be-identified model is subjected to infringement identification to obtain an infringement identification result;

[0053] The current copyright information extraction model is used to extract the model copyright information from the target input data, the target input data includes the target model watermark corresponding to the model copyright information, and the generation process of the target model watermark is as follows:

[0054] Input the current fusion information corresponding to the current sample prediction information of the model to be protected into the current copyright information extraction model to obtain the initial predicted copyright information; the initial fusion information corresponding to the current sample prediction information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0055] Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0056] A target model watermark is obtained based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information.

[0057] The above-mentioned model watermark generation, model infringement identification method, device, computer equipment and storage medium obtain the model copyright information corresponding to the model to be protected, obtain the current fusion information corresponding to the current sample prediction information of the model to be protected, wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark, input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial prediction copyright information, update the current fusion information based on the initial information difference between the model copyright information and the initial prediction copyright information to obtain the updated fusion information, use the updated fusion information as the current fusion information, return to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model until the convergence condition is met, obtain the target fusion information corresponding to the current sample prediction information, obtain the target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information, embed the target model watermark into the model to be protected to obtain the target model, and the target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected. In this way, based on the information difference between the accurate model copyright information and the predicted copyright information output by the model, the data fused with the initial model watermark is adjusted, and the model copyright information can be finally embedded into the model watermark through iteration, thereby obtaining data fused with the target model watermark, that is, the target fusion information. The target model watermark extracted from the target fusion information is embedded in the model to be protected. The target model watermark is embedded with the model copyright information, which can protect the model to be protected, improve the model security of the model to be protected, and protect the model copyright. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A diagram of an application environment of a model watermark generation and model infringement identification method in an embodiment;

[0059] Figure 2 Schematic diagram of a flow chart of a model watermark generation method in one embodiment;

[0060] Figure 3 A schematic diagram of a process of training a target copyright information extraction model in one embodiment;

[0061] Figure 4 A schematic diagram of a process for training a target copyright information extraction model in another embodiment;

[0062] Figure 5 A schematic diagram of a process of training a target copyright information extraction model in yet another embodiment;

[0063] Figure 6 A schematic diagram of a process for model infringement identification in one embodiment;

[0064] Figure 7 A schematic diagram of a flow chart of a model infringement identification method in one embodiment;

[0065] Figure 8 A schematic diagram of a process for protecting model copyright in one embodiment;

[0066] Fig. 9 A schematic diagram of a process of generating a model watermark and training an extractor in one embodiment;

[0067] Fig.10 is a structural block diagram of a model watermark generating device in one embodiment;

[0068] Fig.11 is a structural block diagram of a model infringement identification device in one embodiment;

[0069] Fig.12 is an internal structure diagram of a computer device in one embodiment;

[0070] Fig.13 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0073] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0074] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0075] The key technologies of speech technology include automatic speech recognition technology, speech synthesis technology and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the development direction of human-computer interaction in the future, among which speech has become one of the most promising human-computer interaction methods in the future.

[0076] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0077] The solution provided in the embodiments of the present application involves artificial intelligence computer vision, speech, machine learning and other technologies, which are specifically described by the following embodiments:

[0078] The model watermark generation and model infringement identification methods provided in this application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, vehicle-mounted terminals, and portable wearable devices, and the server 104 may be implemented as an independent server or a server cluster or cloud server consisting of multiple servers.

[0079] Both the terminal 102 and the server 104 can be used independently to execute the model watermark generation and model infringement identification methods provided in the embodiments of the present application.

[0080] For example, the server obtains the model copyright information corresponding to the model to be protected, and the server obtains the current fusion information corresponding to the current sample prediction information of the model to be protected. The initial current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark. The server inputs the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial prediction copyright information. The server updates the current fusion information based on the initial information difference between the model copyright information and the initial prediction copyright information to obtain the updated fusion information, uses the updated fusion information as the current fusion information, and returns to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model until the convergence condition is met, and obtains the target fusion information corresponding to the current sample prediction information. The server obtains the target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information. The server embeds the target model watermark into the model to be protected to obtain the target model, and the target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0081] The terminal obtains the model to be identified and the model copyright information, and obtains the current copyright information extraction model corresponding to the model copyright information. Based on the current copyright information extractor and the model copyright information, the terminal performs infringement identification on the model to be identified and obtains the infringement identification result.

[0082] The terminal 102 and the server 104 can also be used together to execute the model watermark generation and model infringement identification methods provided in the embodiments of the present application.

[0083] For example, the server obtains the model copyright information corresponding to the model to be protected from the terminal, and obtains the current fusion information corresponding to the current sample prediction information of the model to be protected from the terminal. The server inputs the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial prediction copyright information. The server updates the current fusion information based on the initial information difference between the model copyright information and the initial prediction copyright information to obtain the updated fusion information, uses the updated fusion information as the current fusion information, and returns to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information. The server obtains the target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information. The server embeds the target model watermark into the model to be protected to obtain the target model. The server sends the target model to the terminal.

[0084] The server obtains the model to be identified and the model copyright information from the terminal, and performs infringement identification on the model to be identified based on the model copyright information and the corresponding current copyright information extractor to obtain the infringement identification result. The server sends the infringement identification result to the terminal.

[0085] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0086] In one embodiment, Figure 2 As shown, a model watermark generation method is provided, which is applied to Figure 1 The computer in the example is used for explanation. It can be understood that the computer device can be the terminal 102 or the server 104. In this embodiment, the model watermark generation method includes the following steps:

[0087] Step S202, obtaining the model copyright information corresponding to the model to be protected, and obtaining the current fusion information corresponding to the current sample prediction information of the model to be protected; wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark.

[0088] The model to be protected refers to a deep learning model whose copyright and intellectual property rights are to be protected. The model to be protected may be a trained deep learning model. The model copyright information is used to identify the ownership of the model. The model copyright information may be generated based on the owner attribute information of the model copyright holder, for example, the user identifier of the model copyright holder is used as the model copyright information, and the identifier of the affiliated institution of the model copyright holder is used as the model copyright information.

[0089] The current sample prediction information refers to the sample prediction information currently used. The sample prediction information refers to the output data and prediction data of the model to be protected. The training sample is input into the model to be protected. After the internal data of the model to be protected is processed, the model to be protected will output the sample prediction information. For example, if the model to be protected is an image deraining model, an image with rain streaks is input into the image deraining model, and the model can output an image without rain streaks. The image without rain streaks is the sample prediction information.

[0090] The initial model watermark refers to the initial model watermark. The model watermark can be integrated with the output data of the model to be protected to construct special data in which the user cannot perceive the watermark information or has no effect on the understanding of the original output data. It can be understood that there can be multiple training samples of the model to be protected, so there are also multiple sample prediction information. The initial model watermark corresponding to each sample prediction information can be a randomly initialized model watermark, and the initial model watermark corresponding to each sample prediction information can be the same or different. In addition, the initial model watermark corresponding to the sample prediction information of the current training sample can also be obtained based on the model watermark corresponding to the sample prediction information of the previous training sample.

[0091] The current fusion information refers to the fusion information currently used. The initial fusion information is obtained by fusing the sample prediction information and the initial model watermark. Subsequently, after iterative adjustment, the fusion information will continue to change and update until it is no longer updated, and the final fusion information is obtained.

[0092] Specifically, the computer device can obtain the model copyright information corresponding to the model to be protected locally or from other terminals or servers, and obtain the current fusion information corresponding to the current sample prediction information of the model to be protected. When the user has the need to protect the copyright of the model, the user needs to provide the model to be protected and the corresponding model copyright information. The training sample of the model to be protected can be provided by the user, or it can be determined autonomously by the computer device based on the input data format of the model to be protected. Then, the computer device inputs the training sample into the model to be protected to obtain the sample prediction information, and then fuses the sample prediction information with the corresponding initial model watermark to obtain the initial fusion information.

[0093] In one embodiment, the model to be protected includes any one of an image processing model and a speech processing model.

[0094] Specifically, the model to be protected may be an image processing model, which is a deep learning model that processes data on an image. The input data of the image processing model is an image, and the output data is also an image. For example, the image processing model may be an image deraining model, an image super-resolution model, an image style transfer model, etc. The image deraining model is used to remove rain streaks from the input image. The image super-resolution model is used to improve the resolution of the input image. The image style transfer model is used to change the style of the input image, for example, converting the original image captured into an image in the style of an oil painting, and converting the signature image using the first artistic font into a signature image using the second artistic font. It can be understood that when the model to be protected is an image processing model, the model watermark may be an image-type watermark, and the image size of the model watermark is less than or equal to the size of the image to be watermarked. Fusion of the sample prediction information and the initial model watermark is to superimpose the pixel values ​​of the corresponding pixels in the output image of the image processing model and the watermark image, thereby obtaining a new image.

[0095] The model to be protected may also be a speech processing model, which is a deep learning model that processes speech data. The input data of the speech processing model is speech, and the output data is also speech. For example, the speech processing model may be a speech style transfer model, a speech quality optimization model, and the like. Similar to the image style transfer model, the speech style transfer model is used to change the style of the input speech. The speech quality optimization model is used to improve the clarity and intelligibility of the input speech. It can be understood that when the model to be protected is a speech processing model, the model watermark may be a speech-type watermark. The speech length of the model watermark is less than or equal to the length of the speech to be watermarked. Fusion of the sample prediction information and the initial model watermark is to superimpose the output speech of the speech processing model and the speech value of the corresponding sampling point in the watermarked speech to obtain a new speech.

[0096] Step S204: input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information.

[0097] Step S206, based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain the updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information.

[0098] Among them, the current copyright information extraction model refers to the copyright information extraction model currently in use. The copyright information extraction model is a deep learning model that extracts copyright information from input data. The predicted copyright information refers to the output data and predicted data of the copyright information extraction model. The input data of the copyright information extraction model is the fusion information, the output data is the predicted copyright information, and the expected output data is the model copyright information.

[0099] Specifically, the computer device can use the fusion information corresponding to the training sample as the input of the current copyright information extraction model, iteratively attack the current copyright information extraction model, and continuously update the initial model watermark in the fusion information, thereby obtaining data fused with the target model watermark, that is, the target fusion information. Specifically, the computer device can first input the initial fusion information corresponding to the current sample prediction information of the current training sample into the current copyright information extraction model, and extract the model copyright information from the current fusion information through data processing inside the model, thereby obtaining the initial predicted copyright information corresponding to the current sample prediction information. Then, the computer device updates the initial fusion information based on the information difference between the model copyright information and the initial predicted copyright information, for example, based on the similarity between the model copyright information and the initial predicted copyright information, the initial fusion information is updated. The computer device then inputs the updated initial fusion information into the current copyright information extraction model to obtain the updated initial predicted copyright information, and further updates the updated initial fusion information based on the information difference between the model copyright information and the updated initial predicted copyright information, and so on, through multiple iterative processes until the convergence condition is met, and the final initial fusion information is used as the target fusion information corresponding to the current training sample. That is, the computer device inputs the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information, updates the current fusion information based on the initial information difference between the model copyright information and the initial predicted copyright information to obtain the updated fusion information, uses the updated fusion information as the current fusion information, returns to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model, and executes until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information.

[0100] The convergence condition may specifically be that the number of iterations reaches a preset number, the fusion information does not change in consecutive target number iterations or the amount of change is less than a preset threshold, etc. The preset number, target number and preset threshold may be set according to actual needs.

[0101] It is understandable that the computer device can update the current fusion information based on the initial information difference between the model copyright information and the initial predicted copyright information through a custom formula or algorithm.

[0102] In one embodiment, the computer device may iteratively attack the current copyright information extraction model based on a single sample, thereby obtaining target fusion information corresponding to the single sample, and then extracting the model watermark corresponding to the single sample from the target fusion information. The computer device may also iteratively attack the same copyright information extraction model based on a batch of samples, thereby obtaining target fusion information corresponding to each sample in the batch, and then extracting the model watermark corresponding to each sample from the target fusion information.

[0103] Step S208, obtaining a target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information.

[0104] Specifically, after obtaining the target fusion information, the computer device can calculate the information difference between the sample prediction information and the corresponding target fusion information to obtain the target model watermark, for example, the difference between the sample prediction information and the corresponding target fusion information is calculated to obtain the target model watermark. It can be understood that through iterative processing, the model copyright information is finally embedded in the target model watermark. Further, if the initial model watermark corresponding to the training sample is randomly initialized, then the computer device can fuse the final model watermarks corresponding to each training sample to obtain a universal target model watermark. If the initial model watermark corresponding to the training sample is generated based on the model watermark of the previous training sample, then the computer device can use the final model watermark corresponding to the last training sample as the target model watermark.

[0105] Step S210, embedding the target model watermark into the model to be protected to obtain the target model, and the target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0106] Specifically, after obtaining the target model watermark, the computer device can embed the target model watermark into the model to be protected, so that the output data of the model to be protected is embedded with the target model watermark, thereby obtaining the target model. For example, the computer device can use the target model watermark as a parameter of the target network layer, connect the target network layer after the model to be protected, and obtain the target model. In this way, the output data of the model to be protected and the target model watermark are fused through the target network layer, the target model watermark is embedded in the output data of the model to be protected, and the fusion result is used as the output data of the target model.

[0107] The target model and the current copyright information extraction model can be used to perform model copyright protection on the model to be protected. The computer device can feed back the target model to the user. After the user obtains the target model, the user can choose to publish the target model to the outside world instead of publishing the model to be protected. Then, the output data of the infringing model generated by the attacker through imitation will carry the target model watermark. The current copyright information extraction model can be used to identify whether the output data of the model carries the target model watermark, which can be specifically judged by whether the model copyright information can be extracted from the output data of the model. The output data of the model to be identified is input into the current copyright information extraction model. If the output data of the current copyright information extraction model includes the model copyright information of the model to be protected, then it can be determined that the model to be identified is an infringing model and infringes the model copyright of the model to be protected.

[0108] In one embodiment, the watermark attribute information corresponding to the initial model watermark and the target model watermark is within a preset range.

[0109] The watermark attribute information is used to characterize the attributes of the model watermark. The watermark content of the model watermark can be used as the watermark attribute information. For example, when the model watermark is a watermark image, the pixel value of the pixel in the watermark image can be used as the watermark attribute information. The information obtained by data analysis of the model watermark can also be used as the watermark attribute information. For example, when the model watermark is a watermark speech, the frequency of the watermark speech can be used as the watermark attribute information. The frequency of the speech can be obtained by processing the speech through a related algorithm (such as Fourier transform) or a device.

[0110] Specifically, in order to prevent the model watermark from being perceived by the user, the watermark attribute information of the model watermark can be limited to a preset interval, and the watermark attribute information of the initial model watermark and the target model watermark can be limited to a preset interval. The preset interval can be set according to people's perception characteristics of information. For example, when the pixel value of a pixel in an image changes within a smaller pixel value range, the human eye cannot perceive the difference between the images before and after the transformation. Therefore, if the model watermark is a watermark image, the preset interval can be set to a pixel change interval where the human eye cannot perceive the change, for example, the pixel size of the model watermark is limited to between [-∈,∈], where ∈ is approximately 0.05. Similarly, if the model watermark is a watermark voice, the preset interval can be set to a frequency change interval where the human ear cannot perceive the change. In this way, limiting the watermark attribute information corresponding to the initial model watermark and the target model watermark to a preset interval can prevent the initial model watermark and the target model watermark from being perceived by the user, thereby not affecting the output data of the model to be protected. It can be understood that the restriction conditions can of course be appropriately relaxed as long as the model watermark does not affect the user's understanding of the output data of the model to be protected.

[0111] In one embodiment, the current copyright information extraction model may be an initial copyright information extraction model with fixed model parameters. For example, the initial copyright information extraction model may be a pre-trained model with good model parameters. Therefore, the computer device may directly determine the target fusion information corresponding to the current sample prediction information based on the initial copyright information extraction model, and then determine the target model watermark based on the target fusion information, and embed the target model watermark into the model to be protected to obtain the target model. Subsequently, the model copyright protection of the model to be protected may be performed using the target model and the current copyright information extraction model.

[0112] Of course, in order to further improve the accuracy of the target model watermark, the computer device can also further perform model training on the initial copyright information extraction model and fine-tune the model parameters of the initial copyright information extraction model. The computer device can first use the initial fusion information corresponding to the training sample as the input of the initial copyright information extraction model, iteratively attack the initial copyright information extraction model, and continuously update the initial model watermark in the fusion information, so as to obtain the target fusion information corresponding to the training sample, and then fine-tune the model parameters of the initial copyright information extraction model based on the target fusion information corresponding to the training sample, and use the initial fusion information corresponding to the next training sample as the input of the updated initial copyright information extraction model, iteratively attack the updated initial copyright information extraction model, and continuously update the initial model watermark in the fusion information, so as to obtain the intermediate fusion information corresponding to the next training sample, and then fine-tune the model parameters of the initial copyright information extraction model based on the target fusion information corresponding to the next training sample, and so on, until the preset conditions are met and the target copyright information extraction model is obtained. Then, the model copyright protection of the model to be protected can also be performed through the target model and the target copyright information extraction model.

[0113] In the above-mentioned model watermark generation method, by obtaining the model copyright information corresponding to the model to be protected, the current fusion information corresponding to the current sample prediction information of the model to be protected is obtained, wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark, the current fusion information corresponding to the current sample prediction information is input into the current copyright information extraction model to obtain the initial prediction copyright information, based on the initial information difference between the model copyright information and the initial prediction copyright information, the current fusion information is updated to obtain the updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met, and the target fusion information corresponding to the current sample prediction information is obtained, based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information, the target model watermark is obtained, the target model watermark is embedded into the model to be protected to obtain the target model, and the target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected. In this way, based on the information difference between the accurate model copyright information and the predicted copyright information output by the model, the data fused with the initial model watermark is adjusted, and the model copyright information can be finally embedded into the model watermark through iteration, thereby obtaining data fused with the target model watermark, that is, the target fusion information. The target model watermark extracted from the target fusion information is embedded in the model to be protected. The target model watermark is embedded with the model copyright information, which can protect the model to be protected, improve the model security of the model to be protected, and protect the model copyright.

[0114] In one embodiment, obtaining current fusion information corresponding to current sample prediction information of the model to be protected includes:

[0115] Obtain a training sample set, and determine a current training sample from the training sample set; input the current training sample into the model to be protected to obtain sample prediction information corresponding to the current training sample; obtain an initial model watermark corresponding to the current training sample; fuse the sample prediction information corresponding to the current training sample with the initial model watermark to obtain corresponding initial fusion information; use the sample prediction information corresponding to the current training sample and the initial fusion information as the current sample prediction information and the corresponding current fusion information.

[0116] The training sample set includes a plurality of training samples. The training samples in the training sample set are used to determine the target model watermark, and any data that can be used as input of the model to be protected can be used as training samples.

[0117] Specifically, the computer device can obtain a training sample set locally or from other terminals or servers, and randomly select a training sample from the training sample set as the current training sample. The computer device inputs the current training sample into the model to be protected, and after internal data processing of the model to be protected, the model to be protected outputs sample prediction information corresponding to the current training sample. The computer device then obtains the initial model watermark corresponding to the current training sample, and fuses the sample prediction information corresponding to the current training sample with the initial model watermark, thereby obtaining the initial fusion information corresponding to the current training sample. The computer device can select another training sample from the training sample set as the next training sample, and obtain the initial fusion information corresponding to the next training sample in the same way, and finally obtain the initial fusion information corresponding to each training sample.

[0118] Subsequently, the computer device can use the sample prediction information and initial fusion information corresponding to the current training sample as the current sample prediction information and the corresponding current fusion information, input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model, and continuously update the current fusion information through iterative processing to finally obtain the target fusion information.

[0119] In this embodiment, the current training sample is input into the model to be protected to obtain the sample prediction information corresponding to the current training sample, and the current sample prediction information is obtained. The sample prediction information corresponding to the current training sample is fused with the corresponding initial model watermark to obtain the corresponding initial fused information, and the current fused information corresponding to the current sample prediction information is obtained.

[0120] In one embodiment, obtaining an initial model watermark corresponding to the current training sample includes: generating a random model watermark, and using the random model watermark as the initial model watermark corresponding to the current training sample.

[0121] Based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information, a target model watermark is obtained, including: based on the information difference between the same sample prediction information and the corresponding target fusion information, an intermediate model watermark corresponding to each sample prediction information is obtained; and each intermediate model watermark is fused to obtain a target model watermark.

[0122] Among them, the random model watermark refers to the randomly generated model watermark. The intermediate model watermark refers to the model watermark corresponding to a single training sample after iterative processing.

[0123] Specifically, the computer device can randomly generate an initialized model watermark as the initial model watermark corresponding to the training sample. The initial model watermarks corresponding to each training sample can be the same or different. If the initial model watermark corresponding to each training sample is a random model watermark, then when determining the target model watermark, the computer device can first obtain the intermediate model watermark corresponding to the training sample based on the information difference between the sample prediction information and the target fusion information corresponding to the same training sample, and finally obtain the intermediate model watermarks corresponding to each training sample, and then fuse the intermediate model watermarks to obtain the target model watermark. The computer device can perform weighted summation on the intermediate model watermarks to obtain the target model watermark.

[0124] For example, based on the random model watermark, the initial fusion information corresponding to the training sample A is processed n times to obtain the intermediate model watermark a. Based on the random model watermark, the initial fusion information corresponding to the training sample B is processed n times to obtain the intermediate model watermark b. Based on the random model watermark, the initial fusion information corresponding to the training sample C is processed n times to obtain the intermediate model watermark. The intermediate model watermark a, intermediate model watermark b and intermediate model watermark c are weightedly fused to obtain a common target model watermark.

[0125] In one embodiment, if the model watermark is an image type watermark, the pixel value size of the random model watermark can be limited to a preset interval. {unit} ~U[-∈,∈]. Among them, W {unit} represents a random model watermark, U[-∈,∈] represents a preset interval, and ∈ is approximately 0.05.

[0126] In one embodiment, the target model watermark can be calculated by the following formula:

[0127] W {i} =μW {i-1} +(1-μ)P {i}

[0128] Among them, W {i} represents the target model watermark obtained by fusing i intermediate model watermarks, W {i-1} represents the target model watermark obtained by fusing i-1 intermediate model watermarks, P {i} represents the i-th intermediate model watermark, that is, the model watermark obtained after the i-th training sample is iterated. μ represents the fusion weight, which can be set as needed, for example, set to 0.5.

[0129] In this embodiment, the initial model watermark corresponding to each training sample is a random model watermark. Therefore, the initial model watermark corresponding to each training sample can be quickly obtained, and then the target fusion information corresponding to each training sample can be quickly iterated. Subsequently, when determining the target model watermark, the intermediate model watermark corresponding to each training sample is first determined, and then the intermediate model watermarks corresponding to each training sample are fused to finally obtain a universal target model watermark.

[0130] In one embodiment, obtaining an initial model watermark corresponding to a current training sample includes: obtaining target fusion information corresponding to a previous training sample of the current training sample; obtaining an intermediate model watermark corresponding to the previous training sample based on an information difference between sample prediction information and target fusion information corresponding to the previous training sample; and obtaining an initial model watermark corresponding to the current training sample based on the intermediate model watermark corresponding to the previous training sample.

[0131] Based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information, a target model watermark is obtained, including: determining a target training sample from a training sample set; based on the information difference between the sample prediction information corresponding to the target training sample and the target fusion information, a target model watermark is obtained.

[0132] Specifically, the initial model watermark corresponding to the current training sample can also be obtained based on the relevant data of the previous training sample. When determining the initial model watermark corresponding to the current training sample, the computer device can obtain the target fusion information corresponding to the previous training sample of the current training sample, and obtain the intermediate model watermark corresponding to the previous training sample based on the information difference between the sample prediction information and the target fusion information corresponding to the previous training sample. Furthermore, the computer device can directly use the intermediate model watermark corresponding to the previous training sample as the initial model watermark corresponding to the current training sample, or fuse the intermediate model watermark corresponding to the previous training sample and the random model watermark corresponding to the current training sample to obtain the initial model watermark corresponding to the current training sample. Subsequently, through iterative processing, the target fusion information corresponding to the current training sample can be obtained, then the intermediate model watermark corresponding to the current training sample can be obtained based on the sample prediction information and the target fusion information corresponding to the current training sample, and the initial model watermark corresponding to the next training sample can be obtained based on the intermediate model watermark corresponding to the current training sample.

[0133] If the initial model watermark corresponding to each training sample is obtained based on the relevant data of the previous training sample, then when determining the target model watermark, the final model watermark corresponding to the target training sample can be used as the target model watermark. That is, the information difference between the sample prediction information and the target fusion information corresponding to the target training sample is calculated, and the calculation result is used as the target model watermark. Among them, the target training sample can be set as needed, for example, the last training sample to be iteratively processed is used as the target training sample, and the 1000th training sample to be iteratively processed is used as the target training sample.

[0134] For example, based on random perturbations, the initial fusion information corresponding to training sample A is processed n times to obtain model watermark a. Based on model watermark a, the initial fusion information corresponding to training sample B is processed n times to obtain model watermark b. Based on model watermark b, the initial fusion information corresponding to training sample C is processed n times to obtain model watermark c, and model watermark c is used as the general target model watermark.

[0135] In this embodiment, the initial model watermark corresponding to the current training sample is obtained based on the relevant data of the previous training sample, and then the intermediate model watermark corresponding to the current training sample is obtained through iterative processing. In this way, the watermark fusion process can be implicitly included in the process of changing samples and iterative processing, and the powerful data processing ability of the model can be used to improve the accuracy and reliability of the target model watermark.

[0136] In one embodiment, based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, including:

[0137] Perform gradient calculation on the initial information difference to obtain the loss gradient; obtain the gradient adjustment parameter, update the loss gradient based on the gradient adjustment parameter to obtain the target gradient; obtain the updated current fusion information based on the distance between the target gradient and the current fusion information.

[0138] The gradient adjustment parameter is used to determine the adjustment range of the loss gradient. The gradient adjustment parameter can be set as needed, for example, the gradient adjustment parameter is set to 0.1.

[0139] Specifically, when updating the current fusion information, the computer device may perform gradient calculation on the initial information difference to obtain a loss gradient, and then update the loss gradient based on the gradient adjustment parameter, and fuse the gradient adjustment parameter and the loss gradient to obtain a target gradient, for example, multiplying the gradient adjustment parameter and the loss gradient to obtain the target gradient. Finally, the computer device calculates the distance between the target gradient and the current fusion information, and obtains the updated current fusion information based on the distance, for example, taking the difference between the current fusion information and the target gradient as the updated current fusion information.

[0140] In one embodiment, the current fusion information may be updated with reference to the following formula:

[0141]

[0142] Among them, x {i+1} represents the fusion information obtained after the i-th iteration, x {i+1} It can be used as the fusion information of the input model at the i+1th iteration. i represents the fusion information of the input model at the i-th iteration. α represents the gradient adjustment parameter. E(M {t} (I)+W {i} ) represents the initial predicted copyright information corresponding to the training sample I at the i-th iteration. {t} (I)+W {i} Indicates the current fusion information corresponding to the training sample I at the i-th iteration, M {t} (I) represents the output data obtained by inputting the training sample I into the protected model M, i.e., the sample prediction information, W {i} Represents the model watermark at the i-th iteration. δ represents the preset model copyright information. L represents the loss function, where L is a function used to calculate the initial information difference between the model copyright information and the initial predicted copyright information. The specific loss function can be set as needed. The function of the loss function is to make the model output data closer to the expected output data. Represents the gradient of the loss function.

[0143] Taking the model to be protected as an image processing model as an example, the model copyright information and the initial predicted copyright information can be data in a picture format. For example, the model copyright information is a logo image of the model copyright holder. The computer device can calculate the initial information difference based on the pixel values ​​between the corresponding pixel points of the model copyright information and the initial predicted copyright information. Correspondingly, the loss gradient and the target gradient can include the gradient values ​​corresponding to each pixel point. The updated current fusion information can be obtained by calculating the pixel difference between the corresponding pixel points of the current fusion information and the target loss.

[0144] In this embodiment, the loss gradient is obtained by performing gradient calculation on the initial information difference, and the gradient adjustment parameter is obtained. The loss gradient is updated based on the gradient adjustment parameter to obtain the target gradient, and the updated current fusion information is obtained based on the distance between the target gradient and the current fusion information. In this way, by updating the information based on the gradient of the data, the information can be updated in the direction where the information changes fastest, which is conducive to improving the update speed and ensuring data convergence.

[0145] In one embodiment, Figure 3 As shown, based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain the updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met and the target fusion information corresponding to the current sample prediction information is obtained, the method further includes:

[0146] Step S302: input the target fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the target predicted copyright information.

[0147] Step S304: training the current copyright information extraction model based on the model copyright information and the target predicted copyright information to obtain an updated copyright information extraction model.

[0148] Step S306, taking the updated copyright information extraction model as the current copyright information extraction model, taking the next sample prediction information as the current sample prediction information, and returning to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model, until the training is completed to obtain the target copyright information extraction model.

[0149] Specifically, in order to further improve the accuracy of the target model watermark, the computer device can update the model watermark and the copyright information extraction model in sequence based on the same training sample, and alternately update the model watermark and the copyright information extraction model by replacing the training samples, thereby continuously improving the data processing accuracy of the copyright information extraction model while simultaneously improving the accuracy of the target model watermark.

[0150] First, the model parameters of the copyright information extraction model are fixed, and the initial fusion information of a single or single batch of training samples is updated to obtain the corresponding target fusion information. The initial fusion information corresponding to the current training sample is input into the current copyright information extraction model, and the initial predicted copyright information is obtained through data processing inside the model. Based on the initial information difference between the model copyright information and the initial predicted copyright information, the initial fusion information is updated, and then the updated initial fusion information is input into the current copyright information extraction model to obtain the updated initial predicted copyright information. Based on the information difference between the model copyright information and the updated initial predicted copyright information, the updated initial fusion information is further updated, and so on, through multiple iterative processing until the convergence condition is met, the target fusion information corresponding to the current training sample is obtained.

[0151] Secondly, the model parameters of the copyright information extraction model are updated based on the target fusion information. First, the target fusion information corresponding to the current training sample is input into the current copyright information extraction model, and the target predicted copyright information is obtained through data processing inside the model. Then, the current copyright information extraction model is trained based on the model copyright information and the target predicted copyright information, and the model parameters are updated to obtain an updated copyright information extraction model.

[0152] Then, the training sample is replaced and the above steps are repeated, and finally a copyright information extraction model with excellent performance can be trained. The updated copyright information extraction model is used as the new current copyright information extraction model, and the sample prediction information corresponding to the next training sample is used as the new current sample prediction information. Return to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model until the training is completed to obtain the target copyright information extraction model. The target copyright information extraction model is a model obtained by comprehensive training based on relevant data of a large number of training samples, and has excellent data processing capabilities.

[0153] Among them, if a preset number of training samples in the training sample set participate in the model training, the characterization training is completed, and the preset number can be set as needed, for example, the preset number is set to the total number of training samples in the training sample set. It can also be that if the model accuracy of the current copyright information extraction model is greater than the preset accuracy, the characterization training is completed. Model accuracy is used to characterize the prediction accuracy of the model. If the predicted data obtained by inputting the target fusion information of a sample into the model is consistent with the model copyright information corresponding to the sample, it indicates that the data processing model here predicts accurately, and the model accuracy can be obtained based on the proportion of the number of accurate predictions of the model to the total number of times.

[0154] For example, for the initial copyright information extraction model, based on the random model watermark, the initial fusion information corresponding to the training sample A is processed for n iterations to obtain the target fusion information a1. The target fusion information a1 is input into the initial copyright information extraction model to obtain the target predicted copyright information a2. The initial copyright information extraction model is trained based on the target predicted copyright information a2 to obtain the intermediate copyright information extraction model. For the intermediate copyright information extraction model, based on the random model watermark, the initial fusion information corresponding to the training sample B is processed for n iterations to obtain the target fusion information b1. The target fusion information b1 is input into the initial copyright information extraction model to obtain the target predicted copyright information b2. The initial copyright information extraction model is trained based on the target predicted copyright information b2 to obtain the target copyright information extraction model.

[0155] Step S308, embedding the target model watermark into the model to be protected to obtain the target model. The target model and the target copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0156] After obtaining the target fusion information and the target copyright information extraction model corresponding to the multiple training samples, the computer device can obtain the target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information. After obtaining the target model watermark, the computer device can embed the target model watermark into the model to be protected, so that the output data of the model to be protected is embedded with the target model watermark, thereby obtaining the target model. The target model and the target copyright information extraction model can be used to perform model copyright protection on the model to be protected.

[0157] In one embodiment, the same model to be protected may have multiple types of model copyright information. A dedicated copyright information extraction model may be trained for each type of model copyright information to enhance the protection of the model copyright.

[0158] In this embodiment, the model watermark and the copyright information extraction model are updated in sequence based on the same training sample, and the model watermark and the copyright information extraction model are updated alternately by replacing the training samples, so that a more accurate copyright information extraction model and a target model watermark can be obtained, and thus, the model copyright protection of the model to be protected can be performed more effectively.

[0159] In one embodiment, Figure 4 As shown, the current copyright information extraction model is trained based on the model copyright information and the target predicted copyright information to obtain an updated copyright information extraction model, including:

[0160] Step S402: obtaining a target information difference based on the information difference between the model copyright information and the target predicted copyright information.

[0161] Step S404: input the current sample prediction information into the current copyright information extraction model to obtain intermediate prediction information.

[0162] Step S406: obtaining an intermediate information difference based on the information difference between the intermediate prediction information and the reference information.

[0163] Step S408: Based on the intermediate information difference and the target information difference, the model parameters of the current copyright information extraction model are adjusted to obtain an updated copyright information extraction model.

[0164] The reference information and the model copyright information are the same type of information, but the specific contents of the two are different. For example, if the model copyright information is a logo image of the model copyright holder, then the reference information can be a pure white or pure black image. In one embodiment, the information difference between the reference information and the model copyright information is within a preset difference range, that is, there is a reasonable difference between the reference information and the model copyright information, which can effectively reduce the learning pressure of the reference model to a certain extent.

[0165] Specifically, a supervised training method can be adopted to train the current copyright information extraction model. The training goal of the current copyright information extraction model is to hope that the model can extract the model copyright information from the input data embedded with the model watermark, and extract the meaningless information that is not the model copyright information from the input data without the watermark. Therefore, the target fusion information can be used as the positive training sample of the model, and the expected output data corresponding to the positive training sample is the model copyright information, that is, the training label corresponding to the positive training sample is the model copyright information, and the sample prediction information can be used as the negative training sample of the model, and the expected output data corresponding to the negative training sample is the reference information, that is, the training label corresponding to the negative training sample is the reference information.

[0166] The computer device may input the target fusion information corresponding to the current sample prediction information into the current copyright information extraction model, and the model may output the target prediction copyright information. The current sample prediction information may be input into the same current copyright information extraction model, and the model may output the intermediate prediction information. Then, the computer device calculates the information difference between the target prediction copyright information and the model copyright information to obtain the target information difference, calculates the information difference between the intermediate prediction information and the reference information to obtain the intermediate information difference, calculates the model loss information based on the intermediate information difference and the target information difference, performs back propagation based on the model loss information, and adjusts the model parameters of the current copyright information extraction model, thereby obtaining an updated copyright information extraction model.

[0167] In this embodiment, the current copyright information extraction model is trained in a supervised manner, so that the training accuracy can be improved.

[0168] In one embodiment, Figure 5As shown, based on the intermediate information difference and the target information difference, the model parameters of the current copyright information extraction model are adjusted to obtain an updated copyright information extraction model, including:

[0169] Step S502: input the training sample corresponding to the current sample prediction information into the current reference model corresponding to the model to be protected, and obtain the initial reference prediction information corresponding to the current sample prediction information.

[0170] Step S504: Based on the initial reference prediction information and the target fusion information corresponding to the current sample prediction information, the current reference model is trained to obtain an updated reference model.

[0171] Step S506: input the training sample corresponding to the current sample prediction information into the updated reference model to obtain the target reference prediction information.

[0172] Step S508: input the target reference prediction information into the current copyright information extraction model to obtain reference prediction copyright information.

[0173] Step S510, obtaining a reference information difference based on the reference prediction copyright information and the model copyright information.

[0174] Step S512: based on the reference information difference, the intermediate information difference and the target information difference, the model parameters of the current copyright information extraction model are adjusted to obtain an updated copyright information extraction model.

[0175] The reference model is a model with a similar structure and the same purpose as the model to be protected. The reference model can also be called a proxy model of the model to be protected. The reference model is used to simulate the model obtained by the attacker by imitating the model to be protected. The reference model can be actively provided by the provider of the model to be protected, or it can be independently constructed based on the input data and output data of the model to be protected. There can be at least one proxy model.

[0176] It can be understood that the reference model and the model to be protected have the same purpose. For example, if the model to be protected is an image deraining model, the corresponding proxy model is also an image deraining model. The current reference model is the reference model currently used. The updated reference model is the reference model after adjusting the model parameters.

[0177] Specifically, in order to further improve the model performance of the copyright information extraction model, the model to be protected can be subjected to knowledge distillation based on the proxy model of the model to be protected, and the copyright information extraction model can be assisted in training based on the proxy model obtained through knowledge distillation, so that the final target copyright information extraction model can better identify the pirated model imitated by the attacker. When performing knowledge distillation, the input data of the proxy model is the training sample, and the expected output data is the output data of the model to be protected embedded with the model watermark, that is, the target fusion information corresponding to the training sample.

[0178] The computer device can input the training sample corresponding to the current sample prediction information into the current reference model corresponding to the model to be protected. Through the internal data processing of the model, the current reference model outputs the initial reference prediction information corresponding to the current sample prediction information. Then, the computer device generates model loss information based on the initial reference prediction information corresponding to the current sample prediction information and the target fusion information, performs back propagation based on the model loss information, and adjusts the model parameters of the current reference model to obtain an updated reference model. With the help of the updated reference model, another positive training sample of the copyright information extraction model can be established. The computer device can input the training sample corresponding to the current sample prediction information into the updated reference model, and the updated reference model outputs the target reference prediction information, and uses the target reference prediction information as another positive training sample of the copyright information extraction model. The training label corresponding to the positive training sample is the model copyright information. Therefore, the computer device can comprehensively adjust the model parameters of the current copyright information extraction model based on the information differences calculated based on the relevant data of various positive training samples and the information differences calculated based on the relevant data of negative training samples. That is, the computer device can generate model loss information based on the reference information difference calculated by the reference prediction copyright information and the model copyright information, the target information difference calculated by the target prediction copyright information and the model copyright information, and the reference information difference calculated by the intermediate prediction information and the reference information, and perform back propagation based on the model loss information to adjust the model parameters of the current copyright information extraction model, thereby obtaining an updated copyright information extraction model.

[0179] It can be understood that the introduction of the proxy model of the model to be protected allows the copyright information extraction model to detect the model watermark from the output data obtained by knowledge distillation, and then extract the pre-embedded model copyright information from it. In other words, the copyright information extraction model can detect the model watermark from the output data of the infringing model, and then extract the pre-embedded model copyright information from it. In addition, when the corresponding current copyright information extraction model is trained based on the relevant data of the next training sample, the updated reference model obtained based on the current training sample can be used as the current reference model corresponding to the next training sample.

[0180] For example, for the initial copyright information extraction model, based on the random model watermark, the initial fusion information corresponding to the training sample A is processed after n iterations to obtain the target fusion information a1. The target fusion information a1 is input into the initial copyright information extraction model to obtain the target predicted copyright information a2, and the sample prediction information corresponding to the training sample A is input into the initial copyright information extraction model to obtain the intermediate prediction information a3. The training sample A is input into the initial reference model to obtain the initial reference prediction information a4, and the initial reference model is trained based on the initial reference prediction information a4 and the model copyright information to obtain the intermediate reference model. The training sample A is input into the intermediate reference model to obtain the target reference prediction information a5. The target information difference △a1 is calculated based on the target predicted copyright information a2 and the model copyright information, the reference information difference △a2 is calculated based on the target reference prediction information a5 and the model copyright information, the intermediate information difference △a3 is calculated based on the intermediate prediction information a3 and the reference information, and the initial copyright information extraction model is trained based on the target information difference △a1, the reference information difference △a2 and the intermediate information difference △a3 to obtain the intermediate copyright information extraction model.

[0181] For the intermediate copyright information extraction model, based on the random model watermark, the initial fusion information corresponding to the training sample B is processed after n iterations to obtain the target fusion information b1. The target fusion information b1 is input into the intermediate copyright information extraction model to obtain the target predicted copyright information b2, and the sample prediction information corresponding to the training sample B is input into the intermediate copyright information extraction model to obtain the intermediate prediction information b3. The training sample B is input into the intermediate reference model to obtain the initial reference prediction information b4, and the intermediate reference model is trained based on the initial reference prediction information b4 and the model copyright information to obtain the target reference model. The training sample B is input into the target reference model to obtain the target reference prediction information b5. The target information difference △b1 is calculated based on the target predicted copyright information b2 and the model copyright information, the reference information difference △b2 is calculated based on the target reference prediction information b5 and the model copyright information, and the intermediate information difference △b3 is calculated based on the intermediate prediction information b3 and the reference information. The intermediate copyright information extraction model is trained based on the target information difference △b1, the reference information difference △b2 and the intermediate information difference △b3 to obtain the target copyright information extraction model.

[0182] It can be understood that if the same current copyright information extraction model is iteratively attacked based on a batch of samples, then the same reference model can also be trained based on the batch of samples, and the same current copyright information extraction model can also be trained based on the batch of samples.

[0183] In one embodiment, the loss function of the copyright information extraction model can be shown as the following formula:

[0184]

[0185] Among them, L represents the loss function, M {t} (I)+W {δ} represents the target fusion information, E(M {t} (I)+W {δ} ) represents the output data obtained by inputting the target fusion information into the copyright information extraction model E, that is, the target predicted copyright information. {t} (I) represents the output data obtained by inputting the training sample I into the model to be protected M, that is, the sample prediction information. {δ} Indicates the model watermark. δ indicates the model copyright information. SM {i} (I) represents the output data obtained by inputting training sample I into the i-th proxy model, that is, the target reference prediction information. E(SM {i} (I)) represents the output data obtained by inputting the target reference prediction information into the copyright information extraction model E, namely, the reference prediction copyright information. n represents that there are n proxy models SM. E(M {t} (I)) represents the output data obtained by inputting the sample prediction information into the copyright information extraction model E, that is, the intermediate prediction information. Indicates reference information. ||A,B||2 indicates calculating the distance between A and B.

[0186] In this embodiment, by introducing a reference model, simulating the output data of the infringement model based on the reference model, and assisting in training the current copyright information extraction model based on the output data of the reference model, the accuracy of the current copyright information extraction model can be improved, thereby effectively improving the copyright protection of the model to be protected.

[0187] In one embodiment, the training condition corresponding to the current reference model includes minimizing the training information difference, where the training information difference is obtained based on the information difference between the initial reference prediction information corresponding to the current sample prediction information and the target fusion information.

[0188] Specifically, the training condition can be considered as a training target. The training target corresponding to the current reference model can be to minimize the data difference between the actual output data and the expected output data, so that the actual output data of the reference model is closer to the expected output data. The training information difference is calculated based on the actual output data and the expected output data of the reference model, that is, based on the initial reference prediction information and the target fusion information corresponding to the current training sample.

[0189] It can be understood that there can be at least one proxy model. The training conditions corresponding to each proxy model can be to minimize the data difference between the actual output data and the expected output data, that is, to minimize the training information difference. In addition, the training condition can also be a condition that the training information difference is less than a preset threshold, the change in the training information difference is less than a preset threshold, etc. For example, the current reference model is trained based on the training information difference, and the model parameters of the current reference model are adjusted until the training information difference is less than the preset threshold, and an updated reference model is obtained.

[0190] In one embodiment, the training objective of the reference model may be as shown in the following formula:

[0191]

[0192] Where n represents n proxy models SM. SM {i} Represents the i-th proxy model. SM {i} (I) represents the output data obtained by inputting training sample I into the i-th proxy model, i.e., the initial reference prediction information. {t} (I)+W {δ} Represents the target fusion information. {t} (I) represents the output data obtained by inputting the training sample I into the model to be protected M, that is, the sample prediction information. {δ} Represents the model watermark. L represents the loss function, where L is a function used to calculate the difference in training information between the model copyright information and the initial reference prediction information. The specific loss function can be set as needed. The purpose of the loss function is to make the model output data closer to the expected output data.

[0193] In this embodiment, minimizing the difference in training information is used as a training condition for the reference model, so that the data actually output by the reference model can be closer to the expected output data, so that the final reference model can output data embedded with a model watermark.

[0194] In one embodiment, Figure 6 As shown, the method also includes:

[0195] Step S602: Obtain the model to be identified corresponding to the target model.

[0196] Step S604: input the test sample into the model to be identified to obtain the prediction information to be identified.

[0197] Step S606: input the predicted information to be identified into the target copyright information extraction model to obtain the copyright information to be identified.

[0198] Step S608, based on the matching result between the copyright information to be identified and the model copyright information, determine the infringement identification result corresponding to the model to be identified.

[0199] The model to be identified refers to a deep learning model to be identified for infringement of the model copyright of the model to be protected. The test sample is used to test whether the model to be identified infringes the model copyright of the model to be protected. The test sample can be any sample that meets the input data format of the model to be identified. For example, the test sample can be a sample obtained from the training sample set. The test sample can be actively provided by the user through the terminal, or it can be obtained autonomously by the computer device.

[0200] Specifically, the target model and the target copyright information extraction model can be used to perform model copyright protection on the model to be protected. The computer device can obtain the model to be identified provided by the user, input the test sample into the model to be identified, and the model to be identified finally outputs the prediction information to be identified through the internal data processing of the model. The computer device then inputs the prediction information to be identified into the target copyright information extraction model, and the target copyright information extraction model outputs the copyright information to be identified through the internal data processing of the model. The computer device can match the copyright information to be identified with the model copyright information corresponding to the target model, and determine the infringement identification result corresponding to the model to be identified based on the matching result. For example, if the copyright information to be identified is consistent with the model copyright information, the infringement identification result is determined to be infringement, and if the copyright information to be identified is inconsistent with the model copyright information, the infringement identification result is determined to be non-infringement. If the similarity between the copyright information to be identified and the model copyright information is greater than the preset similarity, the infringement identification result is determined to be infringement, and if the similarity between the copyright information to be identified and the model copyright information is less than or equal to the preset similarity, the infringement identification result is determined to be non-infringement.

[0201] After determining the infringement identification result, the computer device can send the infringement identification result to the user terminal. Subsequently, the user terminal can take corresponding rights protection measures based on the infringement identification result, for example, warning the infringing user to remove the infringing model as soon as possible.

[0202] It can be understood that the process of performing infringement identification through the current copyright information extraction model can refer to the process of performing infringement identification through the target copyright information extraction model.

[0203] In this embodiment, the infringement identification is performed on the model to be identified corresponding to the target model based on the target copyright information extraction model, so that the infringement identification result can be quickly obtained, and the model copyright of the model to be protected can be effectively maintained.

[0204] In one embodiment, Figure 7 As shown, a model infringement identification method is provided, which is applied to Figure 1 The computer in the example is used for explanation. It can be understood that the computer device can be the terminal 102 or the server 104. In this embodiment, the model infringement identification method includes the following steps:

[0205] Step S702, obtaining the model to be authenticated and the model copyright information, and obtaining the current copyright information extraction model corresponding to the model copyright information.

[0206] Step S704: Based on the current copyright information extraction model and the model copyright information, an infringement identification is performed on the model to be identified to obtain an infringement identification result.

[0207] Specifically, the user can submit an infringement identification task to the computer device through the terminal, and the infringement identification task includes the model to be identified and the model copyright information. After receiving the infringement identification task, the computer device obtains the current copyright information extraction model corresponding to the model copyright information, and then performs infringement identification on the model to be identified based on the current copyright information extraction model and the model copyright information, and obtains the infringement identification result. The computer device can input the test sample into the model to be identified, and the model to be identified outputs the prediction information to be identified, and the prediction information to be identified is input into the current copyright information extraction model, and based on the matching result of the copyright information to be identified and the model copyright information, the infringement identification result corresponding to the model to be identified is determined. For example, if the copyright information to be identified is consistent with the model copyright information, the infringement identification result is determined to be infringement, and if the copyright information to be identified is inconsistent with the model copyright information, the infringement identification result is determined to be non-infringement. If the similarity between the copyright information to be identified and the model copyright information is greater than the preset similarity, the infringement identification result is determined to be infringement, and if the similarity between the copyright information to be identified and the model copyright information is less than or equal to the preset similarity, the infringement identification result is determined to be non-infringement.

[0208] Among them, the current copyright information extraction model is used to extract model copyright information from the target input data. The target input data contains the target model watermark corresponding to the model copyright information. The generation process of the target model watermark is as follows:

[0209] The current fusion information corresponding to the current sample prediction information of the model to be protected is input into the current copyright information extraction model to obtain the initial predicted copyright information. The initial fusion information corresponding to the current sample prediction information is obtained by fusing the current sample prediction information and the corresponding initial model watermark. Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information. The updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met, and the target fusion information corresponding to the current sample prediction information is obtained. Based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information, the target model watermark is obtained.

[0210] It can be understood that the process of generating the target model watermark can refer to the specific contents of the various embodiments of the aforementioned model watermark generation method, which will not be repeated here.

[0211] The above-mentioned model infringement identification method, when generating the target model watermark, adjusts the data fused with the initial model watermark based on the information difference between the accurate model copyright information and the predicted copyright information output by the model, and can finally embed the model copyright information into the model watermark through iteration, thereby obtaining the data fused with the target model watermark, that is, the target fusion information. The current copyright information extraction model can extract the model copyright information from the data fused with the target model watermark, so the infringement identification of the model to be identified can be performed based on the current copyright information extraction model and the model copyright information, and the infringement identification result can be quickly obtained.

[0212] In a specific embodiment, the model watermark generation method and the model infringement identification method can be applied to the image processing model. Figure 8 First, determine the model to be protected and the model copyright information that needs to be embedded with a watermark. The model copyright information can represent the ownership of the model. For example, the corporate logo image of company A can be used as the model copyright information. Next, the model watermark generation method of the present application is used to train a target model watermark, which is embedded with model copyright information. The target model watermark can be embedded in the original output image of the model to be protected to construct a special output with a watermark that is imperceptible to humans. Finally, the target model watermark is added to the model to be protected to obtain the target model, and the target model is put online. If suspected infringement occurs, use any picture to input the model to be identified. After obtaining the output data, use an extractor (i.e., a copyright information extraction model) to extract the watermark information. If the preset model copyright information is extracted, it can be confirmed that an infringement has occurred. Otherwise, the extractor will output a meaningless image.

[0213] refer to Fig. 9 ,The model watermark generation method includes the following three parts:

[0214] 1. Watermark generation stage

[0215] The original sample (i.e., the original image) is input into the model to be protected to obtain the original output, which is then fused with the initial model watermark as the input of the extractor. The extractor is attacked iteratively using a directed general attack algorithm, and the initial model watermark is continuously updated to obtain the original output embedded with the intermediate model watermark.

[0216] In the watermark generation phase, the preset model copyright information can be embedded into the invisible watermark. The model watermark can be quickly obtained by iteratively attacking the extractor to generate the model watermark.

[0217] 2. Watermark distillation stage

[0218] A proxy model with the same function as the model to be protected is used to perform knowledge distillation on the model to be protected. The input of the proxy model is the original sample, and the training label is the original output embedded with the intermediate model watermark.

[0219] The watermark distillation stage can enhance the robustness of the model watermark against model infringement attacks. Model infringement attacks essentially use proxy models (models with similar structures to the model to be protected) to distill the protected model, thereby infringing the intellectual property rights of the protected model. The watermark distillation stage mimics this process, allowing the extractor to detect the model watermark from the distillation output and then recover the pre-embedded data.

[0220] 3. Lifter training stage

[0221] The training samples of the extractor include the original output with the model watermark and the original output without the model watermark. The training label corresponding to the original output with the intermediate model watermark is the preset model copyright information, and the training label corresponding to the original output without the model watermark is a meaningless image. The original output with the model watermark includes the original output with the intermediate model watermark and the original output with the model watermark distilled from the proxy model. The extractor is trained based on such training samples so that it can recover the preset information from the original output embedded with the model watermark, and at the same time, it is insensitive to the original output without the model watermark embedded and outputs meaningless content.

[0222] It can be understood that the overall training sequence can be that a single sample or a single batch of samples undergoes the watermark generation stage, the watermark distillation stage, and the extractor training stage in sequence, and then the next sample or the next batch of samples, based on the training of the previous sample or the previous batch of samples, undergoes the watermark generation stage, the watermark distillation stage, and the extractor training stage in sequence, until a preset number of samples or a preset number of batches of samples participate in the training to obtain the target model watermark and the target extractor. The target model watermark is an invisible watermark, and its interference with the protected model can be ignored and will not affect the output quality of the protected model. Subsequently, when conducting model infringement identification, the target extractor can be used to extract data from the output data of the model to be identified, so as to determine whether the model to be identified is infringing.

[0223] It should be understood that although Figure 2-7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-7At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0224] In one embodiment, Fig.10 As shown, a model watermark generation device is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an information acquisition module 1002, an information prediction module 1004, an information update module 1006, a watermark determination module 1008 and a watermark embedding module 1010, wherein:

[0225] The information acquisition module 1002 is used to acquire the model copyright information corresponding to the model to be protected, and acquire the current fusion information corresponding to the current sample prediction information of the model to be protected; wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0226] The information prediction module 1004 is used to input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information;

[0227] The information updating module 1006 is used to update the current fusion information based on the initial information difference between the model copyright information and the initial predicted copyright information to obtain updated fusion information, use the updated fusion information as the current fusion information, and return to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0228] The watermark determination module 1008 is used to obtain the target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information;

[0229] The watermark embedding module 1010 is used to embed the target model watermark into the model to be protected to obtain the target model. The target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0230] In one embodiment, the information acquisition module is used to obtain a training sample set, determine the current training sample from the training sample set, input the current training sample into the model to be protected, obtain sample prediction information corresponding to the current training sample, obtain an initial model watermark corresponding to the current training sample, fuse the sample prediction information corresponding to the current training sample with the initial model watermark to obtain the corresponding initial fusion information, and use the sample prediction information corresponding to the current training sample and the initial fusion information as the current sample prediction information and the corresponding current fusion information.

[0231] In one embodiment, the information acquisition module is further used to generate a random model watermark, and use the random model watermark as an initial model watermark corresponding to the current training sample.

[0232] The watermark determination module is also used to obtain the intermediate model watermark corresponding to each sample prediction information based on the information difference between the same sample prediction information and the corresponding target fusion information, and fuse the intermediate model watermarks to obtain the target model watermark.

[0233] In one embodiment, the information acquisition module is also used to obtain the target fusion information corresponding to the previous training sample of the current training sample, and based on the information difference between the sample prediction information and the target fusion information corresponding to the previous training sample, obtain the intermediate model watermark corresponding to the previous training sample, and based on the intermediate model watermark corresponding to the previous training sample, obtain the initial model watermark corresponding to the current training sample.

[0234] The watermark determination module is also used to determine a target training sample from the training sample set, and obtain a target model watermark based on the information difference between sample prediction information and target fusion information corresponding to the target training sample.

[0235] In one embodiment, the information update module is also used to perform gradient calculation on the initial information difference to obtain a loss gradient, obtain a gradient adjustment parameter, update the loss gradient based on the gradient adjustment parameter to obtain a target gradient, and obtain updated current fusion information based on the distance between the target gradient and the current fusion information.

[0236] In one embodiment, the model watermark generating device further includes a model updating module, and the model updating module includes:

[0237] The copyright information prediction unit is used to input the target fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the target predicted copyright information.

[0238] The model training unit is used to train the current copyright information extraction model based on the model copyright information and the target predicted copyright information to obtain an updated copyright information extraction model.

[0239] The model determination unit is used to use the updated copyright information extraction model as the current copyright information extraction model, use the next sample prediction information as the current sample prediction information, and return to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model until the training is completed to obtain the target copyright information extraction model. The target model and the target copyright information extraction model are used to perform model copyright protection on the model to be protected.

[0240] In one embodiment, the model training unit also obtains the target information difference based on the information difference between the model copyright information and the target predicted copyright information, inputs the current sample prediction information into the current copyright information extraction model to obtain intermediate prediction information, obtains the intermediate information difference based on the information difference between the intermediate prediction information and the reference information, and adjusts the model parameters of the current copyright information extraction model based on the intermediate information difference and the target information difference to obtain an updated copyright information extraction model.

[0241] In one embodiment, the model training unit is also used to input the training samples corresponding to the current sample prediction information into the current reference model corresponding to the model to be protected to obtain the initial reference prediction information corresponding to the current sample prediction information, train the current reference model based on the initial reference prediction information corresponding to the current sample prediction information and the target fusion information to obtain an updated reference model, input the training samples corresponding to the current sample prediction information into the updated reference model to obtain the target reference prediction information, input the target reference prediction information into the current copyright information extraction model to obtain the reference predicted copyright information, obtain the reference information difference based on the reference predicted copyright information and the model copyright information, adjust the model parameters of the current copyright information extraction model based on the reference information difference, the intermediate information difference and the target information difference to obtain the updated copyright information extraction model.

[0242] In one embodiment, the training condition corresponding to the current reference model includes minimizing the training information difference, where the training information difference is obtained based on the information difference between the initial reference prediction information corresponding to the current sample prediction information and the target fusion information.

[0243] In one embodiment, the model watermark generating device further comprises:

[0244] The model protection module is used to obtain the model to be identified corresponding to the target model, input the test sample into the model to be identified, obtain the prediction information to be identified, input the prediction information to be identified into the target copyright information extraction model, obtain the copyright information to be identified, and determine the infringement identification result corresponding to the model to be identified based on the matching result between the copyright information to be identified and the model copyright information.

[0245] In one embodiment, the watermark attribute information corresponding to the initial model watermark and the target model watermark is within a preset range.

[0246] In one embodiment, Fig.11 As shown, a model infringement identification device is provided, which can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: a data acquisition module 1102 and a model infringement identification module 1104, wherein:

[0247] The data acquisition module 1102 is used to acquire the model to be identified and the model copyright information, and to acquire the current copyright information extraction model corresponding to the model copyright information;

[0248] The model infringement identification module 1104 is used to extract the model and model copyright information based on the current copyright information, perform infringement identification on the model to be identified, and obtain an infringement identification result.

[0249] Among them, the current copyright information extraction model is used to extract model copyright information from the target input data. The target input data contains the target model watermark corresponding to the model copyright information. The generation process of the target model watermark is as follows:

[0250] Input the current fusion information corresponding to the current sample prediction information of the model to be protected into the current copyright information extraction model to obtain the initial predicted copyright information; the initial fusion information corresponding to the current sample prediction information is obtained by fusing the current sample prediction information and the corresponding initial model watermark;

[0251] Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information;

[0252] The target model watermark is obtained based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information.

[0253] For the specific limitations of the model watermark generation device and the model infringement identification device, please refer to the limitations of the model watermark generation method and the model infringement identification method in the above text, which will not be repeated here. The various modules in the above-mentioned model watermark generation device and model infringement identification device can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0254] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.12 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as model copyright information and target model watermarks. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model watermark generation method and a model infringement identification method are implemented.

[0255] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a model watermark generation method and a model infringement identification method are implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0256] Those skilled in the art will understand that Fig.12 , 13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0257] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0258] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0259] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0260] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0261] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0262] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A model watermark generation method, characterized in that: The method comprises: Obtaining model copyright information corresponding to the model to be protected, and obtaining current fusion information corresponding to the current sample prediction information of the model to be protected; wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark; Input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information; Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to be executed until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information; Obtaining a target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information; The target model watermark is embedded into the model to be protected to obtain the target model. The target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

2. The method according to claim 1, characterized in that The obtaining of current fusion information corresponding to the current sample prediction information of the model to be protected includes: Acquire a training sample set, and determine a current training sample from the training sample set; Inputting the current training sample into the model to be protected to obtain sample prediction information corresponding to the current training sample; Obtaining an initial model watermark corresponding to the current training sample; Fusing the sample prediction information corresponding to the current training sample with the initial model watermark to obtain corresponding initial fusion information; The sample prediction information and the initial fusion information corresponding to the current training sample are used as the current sample prediction information and the corresponding current fusion information.

3. The method according to claim 2, characterized in that The obtaining of the initial model watermark corresponding to the current training sample includes: Generate a random model watermark, and use the random model watermark as the initial model watermark corresponding to the current training sample; The obtaining of the target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information includes: Based on the information difference between the prediction information of the same sample and the corresponding target fusion information, the intermediate model watermark corresponding to each sample prediction information is obtained; The intermediate model watermarks are fused to obtain the target model watermark.

4. The method according to claim 2, characterized in that: The obtaining of the initial model watermark corresponding to the current training sample includes: Obtaining target fusion information corresponding to a previous training sample of the current training sample; Based on the information difference between the sample prediction information corresponding to the previous training sample and the target fusion information, an intermediate model watermark corresponding to the previous training sample is obtained; Obtaining an initial model watermark corresponding to the current training sample based on the intermediate model watermark corresponding to the previous training sample; The obtaining of the target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information includes: Determine a target training sample from the training sample set; The target model watermark is obtained based on the information difference between the sample prediction information corresponding to the target training sample and the target fusion information.

5. The method according to claim 1, characterized in that The updating of the current fusion information based on the initial information difference between the model copyright information and the initial predicted copyright information to obtain updated fusion information includes: Perform gradient calculation on the initial information difference to obtain the loss gradient; Obtaining a gradient adjustment parameter, and updating the loss gradient based on the gradient adjustment parameter to obtain a target gradient; Based on the distance between the target gradient and the current fusion information, updated fusion information is obtained.

6. The method according to any one of claims 1 to 5, characterized in that: Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to be executed until the convergence condition is met and the target fusion information corresponding to the current sample prediction information is obtained, the method further includes: Inputting the target fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the target prediction copyright information; Training the current copyright information extraction model based on the model copyright information and the target predicted copyright information to obtain an updated copyright information extraction model; The updated copyright information extraction model is used as the current copyright information extraction model, the next sample prediction information is used as the current sample prediction information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to execute until the training is completed to obtain the target copyright information extraction model. The target model and the target copyright information extraction model are used to perform model copyright protection on the model to be protected.

7. The method according to claim 6, characterized in that The method of training the current copyright information extraction model based on the model copyright information and the target predicted copyright information to obtain an updated copyright information extraction model includes: Based on the information difference between the model copyright information and the target predicted copyright information, obtaining a target information difference; Input the current sample prediction information into the current copyright information extraction model to obtain intermediate prediction information; Obtaining an intermediate information difference based on an information difference between the intermediate prediction information and the reference information; Based on the intermediate information difference and the target information difference, the model parameters of the current copyright information extraction model are adjusted to obtain an updated copyright information extraction model.

8. The method according to claim 7, characterized in that The method of adjusting the model parameters of the current copyright information extraction model based on the intermediate information difference and the target information difference to obtain an updated copyright information extraction model includes: Inputting the training sample corresponding to the current sample prediction information into the current reference model corresponding to the model to be protected to obtain initial reference prediction information corresponding to the current sample prediction information; Based on the initial reference prediction information and the target fusion information corresponding to the current sample prediction information, the current reference model is trained to obtain an updated reference model; Inputting the training sample corresponding to the current sample prediction information into the updated reference model to obtain the target reference prediction information; Inputting the target reference prediction information into the current copyright information extraction model to obtain reference prediction copyright information; Obtaining a reference information difference based on the reference prediction copyright information and the model copyright information; Based on the reference information difference, the intermediate information difference and the target information difference, the model parameters of the current copyright information extraction model are adjusted to obtain an updated copyright information extraction model.

9. The method according to claim 8, characterized in that The training condition corresponding to the current reference model includes minimizing a training information difference, where the training information difference is obtained based on an information difference between initial reference prediction information corresponding to the current sample prediction information and target fusion information.

10. The method according to claim 6, characterized in that The method further comprises: Obtaining a model to be identified corresponding to the target model; Inputting the test sample into the model to be identified to obtain prediction information to be identified; Inputting the predicted information to be identified into the target copyright information extraction model to obtain the copyright information to be identified; Based on the matching result between the copyright information to be identified and the model copyright information, the infringement identification result corresponding to the model to be identified is determined.

11. The method according to any one of claims 1 to 5, characterized in that: The watermark attribute information corresponding to the initial model watermark and the target model watermark is within a preset range.

12. A model infringement identification method, characterized in that: The method comprises: Obtaining the model to be authenticated and the model copyright information, and obtaining the current copyright information extraction model corresponding to the model copyright information; Based on the current copyright information extraction model and the model copyright information, the to-be-identified model is subjected to infringement identification to obtain an infringement identification result; The current copyright information extraction model is used to extract the model copyright information from the target input data, the target input data includes the target model watermark corresponding to the model copyright information, and the generation process of the target model watermark is as follows: Input the current fusion information corresponding to the current sample prediction information of the model to be protected into the current copyright information extraction model to obtain the initial predicted copyright information; the initial fusion information corresponding to the current sample prediction information is obtained by fusing the current sample prediction information and the corresponding initial model watermark; Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to be executed until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information; A target model watermark is obtained based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information.

13. A model watermark generation device, characterized in that: The device comprises: An information acquisition module, used to acquire model copyright information corresponding to the model to be protected, and to acquire current fusion information corresponding to the current sample prediction information of the model to be protected; wherein the current fusion information is obtained by fusing the current sample prediction information and the corresponding initial model watermark; An information prediction module, used to input the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model to obtain the initial predicted copyright information; An information updating module, used to update the current fusion information based on the initial information difference between the model copyright information and the initial predicted copyright information, obtain updated fusion information, use the updated fusion information as the current fusion information, return to the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model, execute until the convergence condition is met, and obtain the target fusion information corresponding to the current sample prediction information; A watermark determination module, configured to obtain a target model watermark based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information; The watermark embedding module is used to embed the target model watermark into the model to be protected to obtain the target model. The target model and the current copyright information extraction model are used to perform model copyright protection on the model to be protected.

14. A model infringement identification device, characterized in that: The device comprises: A data acquisition module, used to acquire the model to be identified and the model copyright information, and to acquire the current copyright information extraction model corresponding to the model copyright information; A model infringement identification module is used to extract the model and model copyright information based on the current copyright information, perform infringement identification on the model to be identified, and obtain an infringement identification result; The current copyright information extraction model is used to extract the model copyright information from the target input data, the target input data includes the target model watermark corresponding to the model copyright information, and the generation process of the target model watermark is as follows: Input the current fusion information corresponding to the current sample prediction information of the model to be protected into the current copyright information extraction model to obtain the initial predicted copyright information; the initial fusion information corresponding to the current sample prediction information is obtained by fusing the current sample prediction information and the corresponding initial model watermark; Based on the initial information difference between the model copyright information and the initial predicted copyright information, the current fusion information is updated to obtain updated fusion information, the updated fusion information is used as the current fusion information, and the step of inputting the current fusion information corresponding to the current sample prediction information into the current copyright information extraction model is returned to be executed until the convergence condition is met to obtain the target fusion information corresponding to the current sample prediction information; A target model watermark is obtained based on the information difference between the sample prediction information output by the model to be protected and the corresponding target fusion information.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 or 12 are implemented.

16. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 or 12 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 or 12 are implemented.

Citation Information

Patent Citations

  • Image watermark processing method and device based on artificial intelligence and electronic equipment

    CN111160335A

  • Watermark information embedding method and device

    CN112750064A