Diffusion model watermark embedding method and device

Abstract

The invention discloses a diffusion model watermark embedding method and device, and relates to the technical field of artificial intelligence. The computing device obtains a first diffusion model, trains the first diffusion model according to an encoder in the teacher model to obtain an intermediate diffusion model, and trains the intermediate diffusion model according to a decoder in the teacher model to obtain a second diffusion model; wherein the intermediate diffusion model can generate a three-dimensional model with a first watermark, the second diffusion model can generate a three-dimensional model with a second watermark, and the matching degree of the second watermark and the target watermark is greater than the matching degree of the first watermark and the target watermark. The first diffusion model is adjusted in a segmented manner through the encoder and the decoder, so that the first diffusion model learns features of the stage of generating the three-dimensional model with the target watermark, and the obtained second diffusion model not only can reserve model parameters for generating the three-dimensional model as much as possible, but also can update model parameters for generating the watermark. Therefore, the availability of the second diffusion model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a diffusion model watermark embedding method and device. BACKGROUND

[0002] Model watermarking refers to a technology of embedding specific marks or information in a neural network model so as to identify the owner of the model or prove the source of the model. Model parameters of a diffusion model for generating images are fine-tuned by a watermark extraction network (decoder) so that an image generated by the diffusion model can obtain a corresponding watermark by processing of the watermark extraction network.

[0003] However, fine-tuning of the parameters of the diffusion model by the watermark extraction network causes more model parameters in the diffusion model to be adjusted for generating the watermark, which in turn leads to poor image quality generated by the diffusion model and poor usability of the fine-tuned diffusion model. SUMMARY

[0004] The present application provides a diffusion model watermark embedding method and device to solve the problem that fine-tuning of the parameters of the diffusion model by the watermark extraction network causes more model parameters in the diffusion model to be adjusted for generating the watermark, which in turn leads to poor image quality generated by the diffusion model and poor usability of the fine-tuned diffusion model.

[0005] The present application adopts the following scheme.

[0006] In a first aspect, the present application provides a diffusion model watermark embedding method. The method can be executed by a computing device, a chip (such as a processor) in the computing device, or a computing device cluster composed of multiple computing devices. Hereinafter, the diffusion model watermark embedding method executed by the computing device is taken as an example for description, which includes: the computing device acquires a first diffusion model, and trains the first diffusion model according to an encoder in a teacher model to obtain an intermediate diffusion model, and trains the intermediate diffusion model according to a decoder in the teacher model to obtain a second diffusion model. The first diffusion model is used for generating a three-dimensional model, the encoder is used for generating a three-dimensional model with a target watermark, the decoder is used for extracting the watermark in the three-dimensional model, the intermediate diffusion model can generate a three-dimensional model with a first watermark, and the second diffusion model can generate a three-dimensional model with a second watermark. The matching degree of the second watermark to the target watermark is greater than the matching degree of the first watermark to the target watermark.

[0007] In the present application, the computing device performs segmented adjustment on the first diffusion model through the encoder and the decoder in the teacher model, so that the first diffusion model learns the features of different stages, such as the stage of generating a three-dimensional model with a target watermark and the stage of extracting the watermark in the three-dimensional model, and the second diffusion model obtained can preserve the model parameters of the generated three-dimensional model as much as possible and update the model parameters of the generated watermark, thereby improving the usability, i.e., robustness, of the second diffusion model.

[0008] In a possible case, the computing device is a computing device provided by a cloud service provider, i.e., the diffusion model watermark embedding method described above can be applied to a cloud computing platform and executed by one or more computing devices included in the cloud computing platform.

[0009] In a possible example, the diffusion model watermark embedding method described above can be a service on the cloud computing platform, and a user calls the service to achieve watermark embedding on the first diffusion model to obtain the second diffusion model.

[0010] In a possible case, the computing device can be a computing device located on the user side.

[0011] In a possible case, the target watermark can be a watermark predefined by the computing device. The predefined watermark can be randomly generated by the computing device, or generated by the computing device according to user information.

[0012] For example, the computing device takes an identity document (ID) in the user information as the target watermark.

[0013] In a possible case, the first diffusion model is a pre-trained diffusion model.

[0014] In a possible case, the three-dimensional model is represented by any one of the following: a point cloud, a polygon mesh, and a voxel grid.

[0015] In a possible implementation, before the computing device trains the first diffusion model according to the encoder in the teacher model to obtain an intermediate diffusion model, the diffusion model watermark embedding method further includes: the computing device provides a user interface, the user interface includes a first control component, and in response to a triggering operation of the user on the first control component, the target watermark is obtained.

[0016] In the present application, the computing device obtains the target watermark determined by the user by responding to the triggering operation of the user on the first control component on the display interface, thereby achieving visualization while meeting the user's customization demand for the target and improving the user's experience.

[0017] In a possible example, the computing device displays the user interface at a front end connected thereto, and the front end can be a display or a terminal located at a user side and accessing the user interface provided by the computing device through an application programming interface (API) provided by the computing device.

[0018] In a possible case, the target watermark includes one or more of the following: a picture, text, audio, video, and a bitstream.

[0019] In a possible example, the watermark in the three-dimensional model is in the form of a bitstream. In other words, after obtaining the target watermark in the form of a picture, text, audio, or video, the computing device converts the target watermark in the form of a picture, text, audio, or video into a target watermark in the form of a bitstream, and then embeds the target watermark in the form of a bitstream into the first diffusion model to obtain the second diffusion model.

[0020] Two possible implementation manners are provided below for the manner in which the computing device obtains the first diffusion model.

[0021] In a possible implementation manner, the computing device obtains the first diffusion model, including: the computing device providing a user interface, the user interface including a second control component, and obtaining the first diffusion model in response to a triggering operation of the second control component by the user.

[0022] In a possible example, the computing device displays the user interface at a front end connected thereto.

[0023] In this application, the computing device obtains the first diffusion model determined by the user by responding to the triggering operation of the second control component on the display interface, and achieves visualization while satisfying watermark injection on the first diffusion model specified by the user, achieving targeted processing on the first diffusion model specified by the user, and achieving a second diffusion model that is easier to trace.

[0024] In another possible implementation manner, the computing device obtains the first diffusion model, including: the computing device obtaining the first diffusion model from a plurality of reference diffusion models, each of the plurality of reference diffusion models being used to generate a three-dimensional model.

[0025] In this application, in the case where the user does not provide the first diffusion model, the computing device can obtain the first diffusion model from a plurality of reference diffusion models, and then perform watermark embedding processing on the first diffusion model, to achieve a second diffusion model of the user. When the second diffusion model of the user is used by others, the watermark in the three-dimensional model generated from the second diffusion model can be used for tracing, and the difficulty of evidence collection for the use of the second diffusion model of the user by others is reduced.

[0026] In a possible example, before the computing device performs the diffusion model watermark embedding method provided in the present application, a plurality of reference diffusion models have been deployed in the computing device, and thus the computing device can select one of the plurality of reference diffusion models as the first diffusion model.

[0027] In a possible case, the reference diffusion model is a pre-trained diffusion model.

[0028] The selection manner can be random selection, selection according to a user instruction, or selection according to a label determined by the user, and each of the plurality of reference diffusion models has one or more labels.

[0029] In a possible case, the plurality of reference diffusion models differ in different use scenarios or different diffusion model structures.

[0030] For example, a first reference diffusion model in the plurality of reference diffusion models is used to generate a three-dimensional model of an office appliance, and a second reference diffusion model is used to generate a three-dimensional model of a transportation tool.

[0031] A third reference diffusion model in the plurality of reference diffusion models has a network structure that includes more convolution layers, pooling processing layers, and feature extraction network layers than a network structure of a fourth reference diffusion model.

[0032] In a possible implementation manner, the teacher model is trained by using a training data set and a target watermark. The training data set includes three-dimensional models of different types, and the types include one or more of the following: transportation tools, office appliances, and buildings.

[0033] In the present application, the computing device trains the teacher model by using a training data set including three-dimensional models of different types and a target watermark, which can improve the robustness of an encoder in the teacher model in generating a three-dimensional model with the target watermark, and then trains the first diffusion model by using the encoder in the teacher model, which can improve the robustness of the finally obtained second diffusion model.

[0034] In a possible example, the types can further include animals, plants, and clothing.

[0035] In a possible example, the computing device jointly trains the encoder and the decoder in the teacher model by using the training data set and the target watermark.

[0036] In a possible implementation, the computing device trains the first diffusion model according to the encoder in the teacher model to obtain an intermediate diffusion model, including: the computing device inputs a first three-dimensional model in a fine-tuning dataset into the encoder and the first diffusion model, respectively outputs a corresponding first reconstructed three-dimensional model and a second reconstructed three-dimensional model, and updates the model parameters of the first diffusion model according to the loss between the first reconstructed three-dimensional model and the second reconstructed three-dimensional model to obtain the intermediate diffusion model. The first three-dimensional model is any one of the plurality of three-dimensional models included in the fine-tuning dataset.

[0037] In the present application, the computing device updates the model parameters of the first diffusion model according to the loss between the first reconstructed three-dimensional model and the second reconstructed three-dimensional model, which can realize that the obtained intermediate diffusion model learns the ability to generate a three-dimensional model with a target watermark, so that the matching degree between the first reconstructed three-dimensional model and the second reconstructed three-dimensional model generated by the first diffusion model is continuously increased, the quality of the three-dimensional model generated by the intermediate model is improved, and the robustness of the finally obtained second diffusion model is improved.

[0038] In a possible example, the first three-dimensional model has a watermark.

[0039] In another possible example, the first three-dimensional model does not have a watermark.

[0040] In a possible case, the three-dimensional model in the fine-tuning dataset is a discrete point cloud, that is, does not have a fixed shape.

[0041] In a possible implementation, the computing device trains the intermediate diffusion model according to the decoder in the teacher model to obtain a second diffusion model, including: the computing device inputs a second three-dimensional model in a fine-tuning dataset into the intermediate diffusion model to obtain a third reconstructed three-dimensional model, inputs the third reconstructed three-dimensional model into the decoder to obtain a first watermark, and updates the model parameters of the intermediate diffusion model according to the loss between the first watermark and a target watermark to obtain the second diffusion model. The third reconstructed three-dimensional model includes the first watermark, and the second three-dimensional model is any one of the plurality of three-dimensional models included in the fine-tuning dataset.

[0042] In the present application, the computing device optimizes the model parameters of the intermediate diffusion model according to the loss between the first watermark and the target watermark, so that the watermark in the three-dimensional model generated by the finally obtained second diffusion model has a higher matching degree with the target watermark, thereby improving the quality of the watermark in the three-dimensional model generated by the second diffusion model.

[0043] In a possible implementation, the encoder includes a local feature extraction network and a global feature extraction network.

[0044] In a possible example, the local feature extraction network comprises a graph convolutional neural network; and / or the global feature extraction network comprises a graph convolutional neural network.

[0045] In a possible implementation, if the three-dimensional model is generated by using the second diffusion model, the watermark in the three-dimensional model can be extracted by using the decoder in the teacher model, and then the traceability of the second diffusion model can be realized quickly, that is, the owner of the second diffusion model can be determined, and the difficulty of forensics on the use of the second diffusion model by others is reduced.

[0046] In a second aspect, the present application provides a diffusion model watermark embedding device. The diffusion model watermark embedding device is applied to a computer system (such as a computer cluster) or a computing device supporting the computer system to implement a diffusion model watermark embedding method. The diffusion model watermark embedding device comprises various modules for executing the diffusion model watermark embedding method in the first aspect or any optional implementation of the first aspect. For example, the diffusion model watermark embedding device comprises a first acquisition module, a first training module, and a second training module. Wherein,

[0047] The first acquisition module is configured to acquire a first diffusion model, and the first diffusion model is used to generate a three-dimensional model.

[0048] The first training module is configured to train the first diffusion model according to an encoder in a teacher model, to obtain an intermediate diffusion model; the encoder is used to generate a three-dimensional model with a target watermark, and the intermediate diffusion model is capable of generating a three-dimensional model with a first watermark.

[0049] The second training module is configured to train the intermediate diffusion model according to a decoder in the teacher model, to obtain a second diffusion model; the decoder is used to extract a watermark in a three-dimensional model, and the second diffusion model is capable of generating a three-dimensional model with a second watermark, and the matching degree between the second watermark and the target watermark is greater than the matching degree between the first watermark and the target watermark.

[0050] For more detailed implementation of the diffusion model watermark embedding device, refer to the description of any implementation of the first aspect above, and the content of the following specific embodiments, which will not be repeated here.

[0051] In a third aspect, the present application provides a chip, comprising: a processor and a power supply circuit; the power supply circuit is used to supply power for the processor, the processor is used to execute the method in the first aspect or any possible implementation of the first aspect; and / or the processor is used to execute the method in the second aspect or any possible implementation of the second aspect.

[0052] In a fourth aspect, the present application provides a computing device. The computing device comprises a memory configured to store computer instructions; and a processor configured to execute the computer instructions to implement the method in the first aspect or any possible implementation of the first aspect; and / or the processor configured to execute the computer instructions to implement the method in the second aspect or any possible implementation of the second aspect.

[0053] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster comprises at least one computing device, and the computing device comprises a memory configured to store computer instructions; and a processor configured to execute the computer instructions to implement the method in the first aspect or any possible implementation of the first aspect.

[0054] In a sixth aspect, the present application provides a computer readable storage medium, and the storage medium stores computer programs or instructions, and when the computer programs or instructions are executed by a processing device, the method in the first aspect or any possible implementation of the first aspect is implemented; and / or when the computer programs or instructions are executed by the processing device, the method in the second aspect or any possible implementation of the second aspect is implemented.

[0055] In a seventh aspect, the present application provides a computer program product, and the computer program product comprises computer programs or instructions, and when the computer programs or instructions are executed by a processing device, the method in the first aspect or any possible implementation of the first aspect is implemented; and / or when the computer programs or instructions are executed by the processing device, the method in the second aspect or any possible implementation of the second aspect is implemented.

[0056] The beneficial effects of the second aspect to the seventh aspect above can refer to the first aspect or any possible implementation of the first aspect, and will not be repeated here. On the basis of the implementation manners provided by the present application in the above aspects, further combinations can be made to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 An architecture schematic diagram of a computer system provided by the present application;

[0058] Figure 2 A structure schematic diagram of a chip provided by the present application;

[0059] Figure 3 A flow schematic diagram of a diffusion model watermark embedding method provided by the present application;

[0060] Figure 4 A user interface provided by the present application Figure 1 ;

[0061] Figure 5 A model structure diagram of a teacher model provided for the present application;

[0062] Figure 6 A flowchart of a training method of a teacher model provided for the present application;

[0063] Figure 7 A user interface diagram provided for the present application Figure 2 ;

[0064] Figure 8a A flowchart of a distillation learning method provided for the present application;

[0065] Figure 8b A flowchart of a diffusion model verification method provided for the present application;

[0066] Figure 9 A structure diagram of a diffusion model watermark embedding device provided for the present application Figure 1 ;

[0067] Figure 10 A structure diagram of a diffusion model watermark embedding device provided for the present application Figure 2 ;

[0068] Figure 11 A structure diagram of a computing device cluster provided for the present application;

[0069] Figure 12 A connection diagram between computing devices provided for the present application. DETAILED DESCRIPTION

[0070] To solve the problem that the parameters of the diffusion model are fine-tuned by using the watermark extraction network, resulting in that more model parameters in the fine-tuned diffusion model are adjusted to generate the watermark, and then the image quality generated by the fine-tuned diffusion model is poor, and the availability of the fine-tuned diffusion model is poor. The present application provides a diffusion model watermark embedding method. The computing device adjusts the first diffusion model by the encoder and the decoder in the teacher model, realizes that the first diffusion model learns the features of different stages, such as the features of the stage that the encoder generates a three-dimensional model with a target watermark, and the features of the stage that the decoder extracts the watermark in the three-dimensional model, and then obtains a second diffusion model that can preserve the model parameters for generating a three-dimensional model as much as possible, and update the model parameters for generating a watermark, thereby improving the availability of the second diffusion model, that is, the robustness.

[0071] In order to facilitate understanding, first, the technical terms involved in the present application are introduced.

[0072] Watermark: content reflecting the source of the diffusion model. The watermark can include, but is not limited to, one or more of: a picture, text, audio, video, a bit stream. The watermark can be content specified by the owner / provider / generator / producer of the diffusion model.

[0073] Diffusion model: a generative model that learns the underlying data distribution by adding noise and iteratively denoising. In the present application, the diffusion model can generate a three-dimensional model according to the text.

[0074] Model parameters: refer to parameters inside the model that can be learned and adjusted, used to describe the relationship between data characteristics and target variables.

[0075] The following will be exemplarily illustrated in combination with the drawings the scenarios to which the embodiments of the present application can be applied.

[0076] Figure 1 An architecture schematic diagram of a computer system provided by the present application is shown. The computer system 100 includes a computing device 110. The computing device 110 can be an electronic device with computing capability or a virtual device with computing capability. In the case where the computing device 110 is an electronic device with computing capability, the computing device 110 can be a server, a personal computer, a tablet computer, etc. In the case where the computing device 110 is a virtual device with computing capability, the computing device 110 can be a virtual machine, a container, etc.

[0077] In one possible example, a user can input a first diffusion model and a target watermark to the computing device 110, the computing device 110 trains a teacher model using training data and the target watermark, and then fine-tunes the first diffusion model using the trained teacher model, and the computing device 110 outputs a second diffusion model (i.e., the fine-tuned first diffusion model).

[0078] In another possible example, a user can input a target watermark to the computing device 110, the computing device 110 trains a teacher model using training data and the target watermark, and then fine-tunes a first diffusion model in a plurality of reference diffusion models using the trained teacher model, and the computing device 110 outputs a second diffusion model.

[0079] The computing device 110 includes a communication interface 114, a processor 111, and a memory 112. The communication interface 114 is used to communicate with devices located outside the computing device 110. For example, the computing device 110 receives a first diffusion model or a target watermark through the communication interface 114, the computing device 110 processes (e.g., trains) a teacher model using the received target watermark, and then outputs the processing result (e.g., a second diffusion model) through the communication interface 114. The communication interface 114 can be an input / output (I / O) interface.

[0080] The processor 111 is the operation and control core of the computing device 110, which can include a central processing unit (CPU), a specific integrated circuit, other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like. In practical applications, the computing device 110 can also include multiple processors. The processor 111 can include one or more processor cores. The processor 111 is installed with an operating system and other software programs, so that the processor 111 can realize access to the memory 112 and various peripheral component interconnect express (PCIe) devices.

[0081] The processor 111 is connected to the memory 112 through a double data rate (DDR) bus or other types of buses. The memory 112 is the main memory of the computing device 110. The memory 112 is usually used to store various running software in the operating system, received input data, and output results obtained by processing the input data, and the like. In order to improve the access speed of the processor 111, the memory 112 needs to have the advantage of fast access speed. In traditional computer devices, dynamic random access memory (DRAM) is usually used as the memory 112. In addition to DRAM, the memory 112 can also be other random access memories, such as static random access memory (SRAM), and the like. In addition, the memory 112 can also be a read-only memory (ROM). For read-only memories, for example, programmable read-only memories (PROM), erasable programmable read-only memories (EPROM), and the like. The present embodiment does not limit the number and type of the memory 112.

[0082] Optionally, in order to persistently store data, the computer system 100 is also provided with a data storage system 113. The data storage system 113 can be located outside the computing device 110 (as shown in Figure 1 Optionally, the data storage system 113 can also be located inside the host, such as the data storage system 113 exchanges data with the processor 111 through the bus 116. At this time, the data storage system 113 is manifested as a hard disk.

[0083] Optionally, the computer system 100 can also include a client device 120. The user can input the first diffusion model or the target watermark to the computing device 110 through the client device 120, and the computing device 110 sends the processing result (such as the second diffusion model) to the user through the client device 120. The client device 120 can be a terminal device, including but not limited to a personal computer, a server, a mobile phone, a tablet computer, or a smart car, etc.

[0084] Optionally, the computer system 100 can also include an acceleration device 115. The acceleration device 115 is used to perform the watermark embedding task of the first diffusion model. The processor 111 sends the received watermark embedding task to the acceleration device 115, and the acceleration device 115 sends the processing result (such as the second diffusion model) to the processor 111 after completing the watermark embedding task according to the training data. As shown in Figure 1 The acceleration device 115 can be directly inserted into the card slot of the mainboard of the computing device 110, and exchanges data with the processor 111 through the bus 116. It should be noted that, Figure 1 The bus 116 in the above can also be replaced by a bus acceleration device 115 for data transmission in the form of a compute express link (CXL), a universal serial bus (USB) protocol, or other protocols.

[0085] In addition, the acceleration device 115 described above can also not be directly plugged into the card slot of the mainboard of the computing device 110, but be located in an acceleration device. For example, the acceleration device is a device independent of the computing device 110, such as an acceleration card. At this time, the computing device 110 can be connected with the acceleration device 115 through a wired network such as a network cable, or be connected with the acceleration device 115 through a wireless network such as a wireless hotspot or Bluetooth. For example, the acceleration device 115 is used to complete the watermark embedding task of the first diffusion model, such as training the teacher model by using the training data, and then fine-tuning the first diffusion model by using the teacher model. The acceleration device 115 can be implemented by one or more chips. For example, the chip includes any one of a CPU, a graphics processing unit (GPU), a neural-network processing unit (NPU), a tensor processing unit (TPU), an FPGA, and an ASIC. The GPU is also called a display core, a visual processor, or a display chip, and is a microprocessor specially used for image operation on personal computers, workstations, game consoles, and some mobile devices (such as tablet computers and smart phones). The NPU simulates human neurons and synapses at the circuit layer, and directly processes large-scale neurons and synapses by using a deep learning instruction set, so that a group of neurons can be processed by one instruction. The ASIC is suitable for single-purpose integrated circuit products.

[0086] For example, Figure 1 The processor 111 in the computing device 110 can be implemented by a chip, such as a CPU, a GPU, an NPU, a TPU, an FPGA, or an ASIC. Figure 2 As shown in FIG. 2, Figure 2 FIG. 2 is a schematic structural diagram of a chip provided in the present application. As shown in FIG. 2, the chip 200 includes a core 201, a CPU 202, a system buffer 203, a DDR 204, and an input / output (I / O) device 205.

[0087] The CPU 202 is configured to accept an AI task (such as a watermark embedding task of the first diffusion model) and call the core 201 to execute the task. In the case where the chip 200 has multiple cores 201, the CPU 202 is further configured to undertake a scheduling task. For example, the CPU 202 can be implemented by an advanced RISC machine (ARM) processor, which has a small size, low power consumption, uses a 32-bit RISC, and has simple and flexible addressing. Of course, in some embodiments, the CPU 202 can also be implemented by other processors.

[0088] The core 201 is configured to provide the computing capability required in the watermark embedding task of the first diffusion model. In an optional case, the core 201 includes a load / store unit (LSU), a cube calculation unit, a scalar calculation unit, a vector calculation unit, and a buffer. The LSU is configured to load data to be processed and store processed data, and can also be configured to manage reading and writing of internal data in the core between different buffers, and to complete some format conversion operations. The cube calculation unit is configured to provide the core computing power of matrix multiplication. The scalar calculation unit is a single instruction single data (SISD) processor, which processes only one piece of data (usually an integer or a floating point number) at the same time. The vector calculation unit, also known as an array processor, is a processor that can directly operate a group of arrays or vectors for calculation. The number of buffers can be one or more. For example, the buffer mainly refers to a level 1 buffer (L1 buffer), and the buffer is used to temporarily store some data required to be repeatedly used by the core 201, so as to reduce reading and writing from the bus. In addition, the implementation of some data format conversion functions also requires that the source data be located in the buffer. In this embodiment, since the buffer is located in the core, the distance between the cube calculation unit in the core and the storage area where the data is located is shortened, the access of the cube calculation unit to the DDR 204 is reduced, and the data access time delay and the data processing time delay of the core are reduced.

[0089] The system buffer 203, mainly referring to a level 2 buffer (L1 buffer or L2 cache), is configured to temporarily store input data (such as training data) of the chip, intermediate results, or final results (such as a second diffusion model).

[0090] The DDR 204 is an off-chip memory, which can also be replaced by a high bandwidth memory (HBM) or other off-chip memory. The DDR 204 is located between the chip and the external memory, and overcomes the access speed limitation when the computing resource shares the memory for reading and writing.

[0091] The I / O device 205 included in the chip 200 refers to hardware for data transmission, and can also be understood as a device connected to the I / O interface. Common I / O devices include network cards, printers, keyboards, mice, etc. All external storage devices, such as hard disks, floppy disks, optical disks, etc., can also be used as I / O devices.

[0092] The core 201, the CPU 202, the system buffer 203, the DDR 204, and the I / O device 205 are connected through a bus. The bus can include a path for transmitting information between the above components (such as the CPU 202 and the system buffer 203). The bus can include a data bus, a power supply bus, a control bus, and a status signal bus, etc. in addition to the data bus. However, for the purpose of clear illustration, the bus can be a PCIe bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. For example, the core 201 can access the I / O device 205 through the PCIe bus. The core 201 is connected to the system buffer 203 through the DDR bus. Here, different system buffers 203 can communicate with the core 201 through different data buses, and therefore, the DDR bus can be replaced by other types of data buses, and the embodiments of the present application do not limit the type of bus.

[0093] For example, when the CPU 202 loads data (such as a first diffusion model) to be processed by an AI task into the DDR 204, the LSU in the core 201 reads (loads) the data from the DDR 204, and obtains a processing result (such as a second diffusion model) by training the first diffusion model. When the processing result is obtained, the LSU stores the processing result into the DDR 204, and the network interface card sends the processing result to the client device 120, or to the data storage system 113 for persistent storage.

[0094] It is worth noting that, Figure 1 The illustrated acceleration device 115 can also be implemented by the chip 200 as illustrated, and the embodiments of the present application do not limit this. Figure 2 The illustrated chip 200 can also be implemented by the acceleration device 115 as illustrated, and the embodiments of the present application do not limit this.

[0095] It can be understood that the structure illustrated in the embodiments does not constitute a specific limitation on the computing device. In other embodiments, the computing device and the chip can include more or fewer components than those illustrated, or combine certain components, or split certain components, or different arrangement of components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0096] In an embodiment of the present application, the teacher model can be deployed in the computing device or the processor (or chip) in the computing device, and other neural network models or algorithm models with three-dimensional model generation function can also be deployed, such as the reference diffusion model, which is not limited in the present application. In this case, the computing device receives the first diffusion model and the target watermark input by the user to the computing device, and then trains the encoder and the decoder in the teacher model using the target watermark and the training data, so as to segmentally fine-tune the first diffusion model using the encoder and the decoder in the teacher model, and obtain the second diffusion model.

[0097] The implementation of the diffusion model watermark embedding method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0098] Figure 3 A flowchart of a diffusion model watermark embedding method provided by the present application is shown in the figure. The diffusion model watermark embedding method can be executed by a computing device or a chip or processor in the computing device. The computing device can be Figure 1 the computing device 110, the client device 120 or the acceleration device 115 shown in the figure. The chip or processor in the computing device can be Figure 2 the chip and the like shown in the figure. For the hardware implementation of the computing device, refer to the foregoing Figure 1 description. For the implementation of the chip or processor, refer to the foregoing Figure 2 description, which will not be repeated here. In some optional examples, the diffusion model watermark embedding method can also be executed by other computing devices or computer clusters. For the hardware implementation of the computing device, refer to the foregoing Figure 1 and Figure 2 description, which will not be repeated here. The computer cluster includes a plurality of computing devices.

[0099] In one possible case, the computing device 110 described above can be one or more of N computing devices managed by a cloud computing platform, and N is an integer greater than or equal to 2.

[0100] In one possible case, the computing device 110 described above can be a terminal, a server, etc.

[0101] Here, the computing device 110 executes the diffusion model watermark embedding method provided by the present embodiment as an example for illustration, as shown in Figure 3 the figure, the diffusion model watermark embedding method provided by the present embodiment includes the following steps S310-S330.

[0102] S310, the computing device 110 acquires a first diffusion model.

[0103] The first diffusion model is used to generate a three-dimensional model.

[0104] In a possible case, the first diffusion model is a pre-trained diffusion model. That is, the first diffusion model can generate a three-dimensional model corresponding to the prompt word.

[0105] In a possible example, the prompt word is an airplane, and the first diffusion model can generate a three-dimensional model of an airplane according to the prompt word.

[0106] In a possible example, the prompt word is a train, and the first diffusion model can generate a three-dimensional model of a train according to the prompt word.

[0107] It is worth noting that the first diffusion model being a pre-trained diffusion model is only a possible case provided by the present embodiment, and should not be construed as a limitation on the present application. In other embodiments of the present application, the first diffusion model can also be an untrained diffusion model.

[0108] In a possible case, the three-dimensional model can be represented by any one of the following: point cloud, polygon mesh, voxel grid.

[0109] Two possible implementation modes are provided below for the computing device 110 to obtain the content of the first diffusion model.

[0110] In the first possible implementation mode, the computing device 110 provides a user interface including a second control component, and obtains the first diffusion model in response to the user triggering operation on the second control component.

[0111] In a possible example, the client device 120 accesses the computing device 110 through the API provided by the computing device 110, and realizes displaying the user interface on the terminal. Then, the computing device 110 obtains the user's triggering operation on the second control component on the user interface through various input devices (keyboard, mouse, touch screen, etc.) connected with the client device 120.

[0112] As shown in Figure 4 , a user interface provided by the present application is schematically Figure 4 . The user interface schematic diagram includes a second control component for uploading the first diffusion model, and a control component for exporting the second diffusion model. Figure 1

[0113] Three possible examples are provided below for the specific implementation of the triggering operation.

[0114] Example 1, the triggering operation can be the user's confirmation on the second control component through the keyboard, such as the user triggering the confirmation (enter) key.

[0115] Example 2, the triggering operation can be the user's click on the second control component through the mouse.

[0116] ​For example, the user drags a file including a first diffusion model to a designated position on the user interface, and the terminal sends the aforementioned file including the first diffusion model to the computing device 110, and the computing device 110 directly acquires the first diffusion model.

[0117] For another example, after the user clicks the second control component on the user interface, the computing device 110 provides a floating window on the user interface, and the floating window includes storage paths of the first diffusion models (such as storage paths of the first diffusion model 1, the first diffusion model 2, the first diffusion model 3, …, and the first diffusion model n). After the user selects a designated first diffusion model (such as the first diffusion model 1) on the floating window and clicks the “confirm” control component by using the mouse, the terminal sends the storage path of the aforementioned first diffusion model 1 to the computing device 110, and the computing device 110 acquires the first diffusion model 1 from the storage path of the first diffusion model 1. The storage path indicates the storage position of the first diffusion model.

[0118] In Example 3, the triggering operation can be an input operation of the user in the second control component by using the keyboard, and the like, such as inputting a corresponding storage path.

[0119] The above examples are only optional contents provided by the present embodiment, and should not be understood as a limitation on the present application. In other embodiments of the present application, the triggering operation can also be a hovering operation, voice control, knuckle tapping, and the like.

[0120] In a second possible implementation, the computing device 110 acquires the first diffusion model from a plurality of reference diffusion models, and each reference model in the plurality of reference models is used to generate a three-dimensional model.

[0121] In a possible case, the plurality of reference diffusion models are all pre-trained diffusion models.

[0122] The plurality of reference diffusion models are different in that the use scenarios are different, and the model structures are different.

[0123] In a possible example, a first reference diffusion model in the plurality of reference diffusion models is used to generate a three-dimensional model of an office appliance, and a second reference diffusion model is used to generate a three-dimensional model of a transportation tool. In other words, different types of training sets are used when the reference diffusion models are trained.

[0124] A third reference diffusion model in the plurality of reference diffusion models has a network structure that is increased with a convolution layer, a pooling processing layer, and a feature extraction network layer, compared with a network structure of a fourth reference diffusion model.

[0125] The following provides three possible examples for the content that the computing device 110 acquires the first diffusion model from the plurality of reference diffusion models.

[0126] Example 1, the computing device 110 randomly determines the first diffusion model from the plurality of reference diffusion models.

[0127] Example 2, the computing device 110 determines the first diffusion model from the plurality of reference diffusion models according to the instruction of the user.

[0128] For example, the computing device 110 provides a control component on the user interface including information of the plurality of reference diffusion models, and in response to the triggering operation of the user on the aforementioned control component, the first diffusion model in the plurality of reference diffusion models is obtained.

[0129] Example 3, the computing device 110 determines the first diffusion model from the plurality of reference diffusion models according to the label determined by the user.

[0130] Each of the plurality of reference diffusion models has one or more labels.

[0131] The computing device 110 determines at least one reference diffusion model from the plurality of reference diffusion models that meets the label determined by the user, and the label of the at least one reference diffusion model matches the label determined by the user. If the at least one reference diffusion model is only one reference diffusion model, the computing device 110 takes the reference diffusion model as the first diffusion model. If the at least one reference diffusion model has multiple reference diffusion models, the computing device 110 determines the content of the first diffusion model, which can refer to the description of the aforementioned examples 1 and 2, and will not be repeated here.

[0132] S320, the computing device 110 trains the first diffusion model according to the encoder in the teacher model to obtain an intermediate diffusion model.

[0133] The aforementioned encoder is used to generate a three-dimensional model with a target watermark, and the intermediate diffusion model can generate a three-dimensional model with a first watermark.

[0134] In a possible implementation, the aforementioned teacher model is trained using a training data set and a target watermark. The training data set includes three-dimensional models of different types, and the aforementioned types include one or more of the following: vehicles, office supplies, and buildings.

[0135] In a possible case, the aforementioned types can also include animals, plants, clothing, toys, and household items.

[0136] For the detailed content of the present implementation, please refer to the description of the following Figure 6 , which will not be repeated here.

[0137] In a possible implementation, the computing device 110 trains the first diffusion model according to the encoder in the teacher model to obtain an intermediate diffusion model, including:

[0138] The computing device 110 migrates the ability of the encoder to generate the three-dimensional model with the target watermark to the first diffusion model by model distillation, and adjusts model parameters of the first diffusion model to obtain an intermediate diffusion model (the first diffusion model after model parameter adjustment).

[0139] Model distillation, which can also be referred to as knowledge distillation, refers to fixing model parameters of a trained teacher model, and making the first diffusion model gradually close to the performance characteristics (generating the three-dimensional model with the target watermark) of the teacher model in the training process according to a set loss, to obtain the intermediate diffusion model after the training reaches a preset round or the loss converges.

[0140] The loss refers to a loss between different output results of the encoder in the teacher model and the first diffusion model for the same input.

[0141] For details of the present implementation, refer to the description below Figure 8a , which will not be repeated here.

[0142] S330, the computing device 110 trains the intermediate diffusion model according to the decoder in the teacher model to obtain a second diffusion model.

[0143] The decoder is used to extract the watermark in the three-dimensional model, and the second diffusion model can generate a three-dimensional model with a second watermark, and the matching degree between the second watermark and the target watermark is greater than the matching degree between the first watermark and the target watermark.

[0144] In one possible implementation, the computing device 110 trains the intermediate diffusion model according to the decoder in the teacher model to obtain a second diffusion model, including:

[0145] The computing device 110 extracts the watermark from the output result of the intermediate diffusion model by using the decoder to obtain the first watermark, and then updates the parameters of the intermediate diffusion model according to the loss between the first watermark and the target watermark to obtain the second diffusion model.

[0146] For details of the present implementation, refer to the description below Figure 8a , which will not be repeated here.

[0147] In one possible embodiment, the present application provides a network structure of a teacher model. First, as shown in Figure 5 , a model structure diagram of a teacher model provided by the present application is shown. Figure 5

[0148] Figure 5 The teacher model 500 shown in the figure includes an encoder 510 and a decoder 520. ​

[0149] The encoder 510 comprises a local feature extraction network 511, a global feature extraction network 512, a convolutional layer 513, a convolutional layer 514, a pooling layer 515, and a multi-layer perceptron (MLP) 516.

[0150] The decoder 520 comprises a global feature extraction network 521, a pooling layer, a convolutional layer 522, a pooling layer 523, and an MLP 524.

[0151] Further, a convolutional layer 530 and a convolutional layer 540 are respectively arranged in front of the encoder 510 and the decoder 520.

[0152] In one possible case, the local feature extraction network 511 comprises graph convolutional networks (GCN).

[0153] In one possible case, the global feature extraction network 512 comprises GCN.

[0154] In one possible case, the global feature extraction network 521 comprises GCN.

[0155] In one possible case, the pooling layer 515 is a maxpool layer.

[0156] In one possible case, the pooling layer 523 is a maxpool layer.

[0157] It is worth noting that, Figure 5 f in the above formula, l represents a vector obtained after the vector is processed by the convolutional layer 513 in the encoder 510, represents a vector obtained after the vector is processed by the pooling layer 515 in the encoder 510, represents a vector obtained after the vector is processed by the pooling layer 523 in the decoder 520, and m represents a target watermark, represents a predicted watermark.

[0158] The convolutional layer 513, the convolutional layer 514, the convolutional layer 530, and the convolutional layer 540 can have the same or different sizes (sizes of convolutional kernels), which are not limited in the present application.

[0159] In one possible embodiment, on the basis of the model structure of the teacher model shown in Figure 5 the following provides a training method of a teacher model. As shown in Figure 6 the following provides a training method of a teacher model. As shown in Figure 6 a flowchart of the training method of the teacher model provided in the present application. The method comprises the following steps S610-S630.

[0160] S610 and computing device 110 acquire the training dataset and target watermark.

[0161] The training dataset includes different types of 3D models.

[0162] In one possible scenario, the 3D models in the training dataset do not have watermarks.

[0163] Regarding the acquisition of the training dataset by computing device 110, three possible implementation methods are provided below.

[0164] In a first possible implementation, the data storage system 113 stores the training dataset, and the computing device 110 retrieves the training dataset from the data storage system 113.

[0165] In the second possible implementation, the memory 112 stores the training dataset, and the computing device 110 retrieves the training dataset from the memory 112.

[0166] In a third possible implementation, computing device 110 acquires the training dataset sent by client device 120.

[0167] Regarding the acquisition of the target watermark content by the computing device 110, three possible implementation methods are provided below.

[0168] In a first possible implementation, the computing device 110 uses a preset watermark as the target watermark.

[0169] In one possible scenario, the user interface provided by the computing device 110 includes a preset target watermark.

[0170] In the second possible implementation, the computing device 110 uses a randomly generated watermark as the target watermark.

[0171] In one possible scenario, the user interface provided by the computing device 110 includes a randomly generated target watermark.

[0172] In a third possible implementation, computing device 110 acquires the target watermark sent by client device 120.

[0173] In one possible scenario, such as Figure 7 As shown, Figure 7 User interface diagram provided for this application Figure 2 . Figure 7 The user interface shown is in Figure 4 The user interface shown is supplemented with a first control unit for receiving target watermarks input by the user. The computing device 110, in response to the user's triggering operation on the first control unit, acquires the target watermark sent by the client device 120.

[0174] In other words, the user interface provided by the computing device 110 includes the first control component and the second control component.

[0175] In one possible example, the user interface described above further includes a control component for specifying the watermark. If the user triggers the "Yes" in the control component, the user needs to input the target watermark in the first control component. If the user triggers the "No" in the control component, the computing device 110 obtains the target watermark by using the first possible implementation manner or the second possible implementation manner described above.

[0176] For example, if the client device 120 accesses the computing device 110 and sends the user information to the computing device 110, the computing device 110 takes the ID in the user information as the target watermark, and the user interface provided by the computing device 110 includes the target watermark.

[0177] It is worth noting that the target watermark obtained by the computing device 110 can be in the form of a picture, text, audio, video, or bitstream.

[0178] S620, the computing device 110 inputs the training data set and the target watermark into the teacher model, and the encoder outputs the three-dimensional model with the watermark, and the decoder outputs the predicted watermark.

[0179] The computing device 110 divides the data in the training data set into multiple batches of data, and then inputs the multiple batches of data into the teacher model in sequence for training.

[0180] In one possible case, the computing device 110 converts the target watermark from the form of a picture, text, audio, or video to the form of a bitstream before inputting the target watermark into the teacher model.

[0181] In one possible example, the computing device 110 can encode the picture, text, audio, or video to obtain the corresponding bitstream.

[0182] The following describes an example of inputting one batch of data in the multiple batches of data into the teacher model for training. The batch of data includes a three-dimensional model A.

[0183] The computing device 110 inputs the three-dimensional model A into the teacher model 500. The computing device 110 extracts a feature A in the three-dimensional model A through a convolutional layer 530, and then inputs the feature into a local feature extraction network 511 and a global feature extraction network 512 respectively to obtain corresponding features B and C. The computing device 110 processes the feature B by using a convolutional layer 513 to obtain f l , and processes the feature B by using a convolutional layer 514 to obtain a feature D. The feature D is processed by a pooling layer 515 to obtain The computing device 110 inputs the fl 、 The feature inputted into the MLP after fusion with the target watermark outputs a three-dimensional model with watermark.

[0184] The computing device 110 further inputs the three-dimensional model with watermark into the convolution layer 540 to extract the feature E of the three-dimensional model with watermark, and the feature E is processed by the global feature extraction network 521 to obtain the feature F. The computing device 110 inputs the feature F into the convolution layer 522 to obtain the feature G, and inputs the feature G into the pooling layer 523 to obtain the feature H. The computing device 110 inputs the feature H into the input MLP to output the predicted watermark. The input MLP outputs the predicted watermark.

[0185] It is worth noting that the target watermark inputted into the teacher model is in the form of a bit stream, such as 01001011.

[0186] It is worth noting that in the actual training process, each batch of data will include a large number of three-dimensional models.

[0187] S630, the computing device 110 updates the model parameters of the encoder according to the loss A between the three-dimensional model in the training data set and the corresponding three-dimensional model with watermark, and updates the model parameters of the encoder and the model parameters of the decoder according to the loss B between the predicted watermark and the target watermark, so as to obtain the trained teacher model.

[0188] The trained teacher model is the teacher model in the content shown in Figure 3 .

[0189] In one possible case, the above loss A can be calculated by Euclidean distance, chamfer distance (CD), mean square error, cross-entropy loss function, etc., which is not limited by the present application.

[0190] The following takes the computing device 110 calculating the loss between the three-dimensional model in the training data set and the corresponding three-dimensional model with watermark by Euclidean distance as an example for description.

[0191] The three-dimensional model in the training data set and the three-dimensional model with watermark are in the form of point cloud, the three-dimensional model in the training data set includes a plurality of point clouds 1, and the three-dimensional model with watermark includes a plurality of point clouds 2. The computing device 110 determines the distance between the point cloud 1 and the point cloud 2 by Euclidean distance, and then determines the above loss A according to the distances between the plurality of point clouds 1 and the plurality of point clouds 2. The group of point clouds 1 and point clouds 2 for determining the distance have the following corresponding relationship: the position of the point cloud 1 in the above three-dimensional model in the training data set corresponds to the position of the point cloud 2 in the above three-dimensional model with watermark.

[0192] For example, the position of the point cloud 1 in the three-dimensional model in the training data set is the outermost right upper corner, and the position of the point cloud 2 in the three-dimensional model with the watermark is the outermost right upper corner.

[0193] In a possible example, the reciprocal of the sum of distances between the plurality of point clouds 1 and the plurality of point clouds 2 is the loss A.

[0194] In a possible example, the computing device 110 calculates the distances between the plurality of point clouds 1 and the plurality of point clouds 2 according to a preset formula to obtain the loss A.

[0195] In a possible case, the loss B can be calculated by a hamming distance, an edit distance, a cross-entropy loss function, etc., which is not limited in the present application.

[0196] The computing device 100 can update the parameters of the encoder 510 by using the loss A (L recon ) between the three-dimensional model A and the three-dimensional model with the watermark, and update the parameters of the teacher model (the encoder and the decoder) by using the loss B (L w ) between the predicted watermark and the target watermark.

[0197] The above is only one round of training of the teacher model by the computing device 110 in the multiple rounds of training, and the content of each round of training in the multiple rounds of training can refer to the above description, which is not repeated here. The computing device 110 performs multiple rounds of training on the teacher model until the L recon and / or L w converge, the data in the training data set is used up, or the set number of training rounds is reached, so as to obtain the trained teacher model.

[0198] For the content of obtaining the intermediate diffusion model in S320 and obtaining the second diffusion model in S330, a possible example is provided as follows. As shown in Figure 8a , the flowchart of the distillation learning method provided by the present application is shown. Figure 8a Figure 8a The content shown can include the following steps S810-S850.

[0199] S810, the computing device 110 inputs the first three-dimensional model in the fine-tuning data set into the encoder and the first diffusion model, and outputs the first reconstructed three-dimensional model and the second reconstructed three-dimensional model.

[0200] The first three-dimensional model is any one of the plurality of three-dimensional models included in the fine-tuning data set.

[0201] ​In a possible case, the three-dimensional model in the fine-tuning dataset does not have a watermark. The fine-tuning dataset in this case can refer to the content of the training dataset described above, which is not repeated here.

[0202] In another possible case, the three-dimensional model in the fine-tuning dataset has a watermark. The watermark can be a bit stream of any content, such as 1010111, which is not limited in the present application.

[0203] In a possible implementation, the computing device 110 inputs the first three-dimensional model in the fine-tuning dataset into the encoder and the first diffusion model, and outputs the first reconstructed three-dimensional model and the second reconstructed three-dimensional model, comprising: the computing device 110 inputs the first three-dimensional model into the encoder to output the first reconstructed three-dimensional model, and inputs the first three-dimensional model into the first diffusion model to output the second reconstructed three-dimensional model.

[0204] The first reconstructed three-dimensional model and the second reconstructed three-dimensional model described above have a watermark.

[0205] S820, the computing device 110 updates the model parameters of the first diffusion model according to the loss between the first reconstructed three-dimensional model and the second reconstructed three-dimensional model, to obtain an intermediate diffusion model.

[0206] In a possible implementation, the computing device 110 updates the model parameters of the first diffusion model according to the loss between the first reconstructed three-dimensional model and the second reconstructed three-dimensional model, to obtain an intermediate diffusion model, comprising: the computing device 110 calculates the loss (L denoising ) between the first reconstructed three-dimensional model and the second reconstructed three-dimensional model, and then performs back propagation on the first diffusion model using the loss to update the model parameters of the first diffusion model, thereby obtaining the intermediate diffusion model.

[0207] L denoising It can be calculated by Euclidean distance, Chambolle distance, mean square error, cross-entropy loss function, etc., which is not limited in the present application.

[0208] It is worth noting that the above is only described by taking one three-dimensional model (the first three-dimensional model) as an example. In the actual training process, the computing device 110 will perform multiple rounds of training using the fine-tuning dataset. In each round of training, the computing device 110 inputs a batch of data into the encoder and the first diffusion model, and outputs multiple groups of reconstructed three-dimensional models, and then updates the model parameters of the first diffusion model according to the loss between the multiple groups of reconstructed three-dimensional models to obtain an intermediate diffusion model.

[0209] The batch of data includes at least one three-dimensional model. The computing device 110 updates the model parameters of the first diffusion model once for each batch of data input into the encoder and the first diffusion model. That is, the model parameters of the first diffusion model are updated once for each training round, until the loss between the reconstructed three-dimensional models (such as the first reconstructed three-dimensional model and the second reconstructed three-dimensional model) converges, the set number of training rounds is reached, or the three-dimensional models in the fine-tuning data set are all used up, and the intermediate diffusion model is obtained.

[0210] S830, the computing device 110 inputs the second three-dimensional model in the fine-tuning data set into the intermediate diffusion model to obtain a third reconstructed three-dimensional model.

[0211] The third reconstructed three-dimensional model includes the first watermark, and the second three-dimensional model is any one of the plurality of three-dimensional models included in the fine-tuning data set.

[0212] In a possible example, the computing device 110 inputs a batch of data in the fine-tuning data set into the intermediate diffusion model to obtain a plurality of third reconstructed three-dimensional models. The batch of data in the fine-tuning data set includes a plurality of three-dimensional models, and the plurality of three-dimensional models include the second three-dimensional model.

[0213] For example, the fine-tuning data set includes a plurality of batches of data, and each batch of data in the plurality of batches of data includes at least one three-dimensional model.

[0214] S840, the computing device 110 inputs the third reconstructed three-dimensional model into the decoder in the teacher model to obtain the first watermark.

[0215] The computing device 110 extracts the watermark in the third reconstructed three-dimensional model through the decoder in the teacher model to obtain the first watermark.

[0216] S850, the computing device 110 updates the model parameters of the intermediate diffusion model according to the loss between the first watermark and the target watermark to obtain a second diffusion model.

[0217] In a possible implementation, the computing device 110 updates the model parameters of the intermediate diffusion model according to the loss between the first watermark and the target watermark to obtain the second diffusion model, including: the computing device 110 calculates the loss (L wm ) between the first watermark and the target watermark, and then performs back propagation on the intermediate diffusion model to update the model parameters of the intermediate diffusion model, thereby obtaining the second diffusion model.

[0218] L wm The loss can be calculated by Hamming distance, edit distance, cross-entropy loss function, etc., which is not limited in the present application.

[0219] It is worth noting that the above explanation uses only one 3D model (the second 3D model) as an example. In the actual training process, the computing device 110 will use the fine-tuning dataset for multiple rounds of training. In each round of training, the computing device 110 inputs a batch of data into the intermediate diffusion model, outputs multiple reconstructed 3D models, and then uses a decoder to extract the watermark from these multiple reconstructed 3D models. Using the loss between the multiple watermarks and the target watermark, the model parameters of the intermediate diffusion model are updated to obtain the second diffusion model.

[0220] The aforementioned batch of data includes at least one three-dimensional model. Each batch of data is processed by an intermediate diffusion model and a decoder. The computing device 110 will update the model parameters of the intermediate diffusion model once. That is, the model parameters of the intermediate diffusion model will be updated once for each round of training until the loss between the first watermark and the target watermark converges, the set number of training rounds is reached, or all three-dimensional models in the fine-tuned dataset are used up, and the second diffusion model is obtained.

[0221] The aforementioned second diffusion model can generate a 3D model with a second watermark. Since the model parameters of the intermediate diffusion model were adjusted during the training process based on the loss between the first watermark and the target watermark, the matching degree between the second watermark and the target watermark is greater than that between the first watermark and the target watermark. This achieves the goal that the second diffusion model can both retain the model parameters of the generated 3D model as much as possible and update the model parameters of the generated watermark, thereby improving the usability, i.e., robustness, of the second diffusion model.

[0222] In one possible scenario, the second watermark matches the target watermark 100%, meaning the second watermark is the target watermark.

[0223] In one possible scenario, the matching degree between the second watermark and the target watermark falls within a set range. For example, it is greater than or equal to the first value, but less than 1. The first value can be 70%. It is worth noting that this application does not limit the first value; it can be greater than 70% or less than 70%.

[0224] In one possible embodiment, computing device 110 can verify whether the model has a watermark. For example... Figure 8b As shown, Figure 8b This is a flowchart illustrating the diffusion model verification method provided in this application. There are two suspected diffusion models, 1 and 2, that are to be judged for allegedly misappropriating the second diffusion model. Watermark verification needs to be performed on these two diffusion models.

[0225] Figure 8bThe user inputs a prompt (such as airplane or train) into the diffusion model 1 and diffusion model 2 to be judged on the client device 120. Diffusion model 1 outputs 3D model 1, and diffusion model 2 outputs 3D model 2. Since the watermarks carried in 3D model 1 and 3D model 2 are implicit, the user can send 3D model 1 and 3D model 2 to the computing device 110 through the client device 120. The computing device 110 inputs 3D model 1 and 3D model 2 into the decoder in the teacher model for watermark extraction, inputting watermark 1 and watermark 2 respectively. If the matching degree of watermark 1 with the target watermark of the user's second diffusion model reaches a threshold, such as 70%, it can be determined that diffusion model 1 is consistent with the above-mentioned second diffusion model, that is, someone else has stolen the second diffusion model. If the matching degree of watermark 2 with the target watermark of the user's second diffusion model does not reach the threshold, it can be determined that diffusion model 2 is inconsistent with the above-mentioned second diffusion model.

[0226] It is understood that, in order to achieve the functions in the above embodiments, the computing device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0227] The above text combines Figures 1 to 8b This paper describes in detail the diffusion model watermark embedding method provided according to the embodiments of this application. The following will combine... Figure 9 This describes a diffusion model watermark embedding apparatus provided according to embodiments of this application.

[0228] Figure 9 A schematic diagram of the structure of a diffusion model watermark embedding device provided in this application Figure 1 The diffusion model watermarking embedding device can be used to implement the functions of the computing device in the above-described diffusion model watermarking embedding method embodiments, and therefore can also achieve the beneficial effects of the above-described method embodiments. In this embodiment, the diffusion model watermarking embedding device can be as follows: Figure 1 Any of the devices shown, such as computing device 110, client device 120, and acceleration device 115, or computing devices shown in subsequent embodiments, may also be modules (such as chips) applied to the device.

[0229] like Figure 9 As shown, the diffusion model watermark embedding device 900 includes a first acquisition module 910, a first training module 920, and a second training module 930. The diffusion model watermark embedding device 900 is used to implement the above-mentioned... Figures 1 to 8bThe corresponding method embodiment describes the function of the computing device 110. In one possible example, the diffusion model watermark embedding device 900 implements the above-described diffusion model watermark embedding method through the following specific processes:

[0230] The first acquisition module 910 is used to acquire the first diffusion model, which is used to generate a three-dimensional model.

[0231] The first training module 920 is used to train the first diffusion model based on the encoder in the teacher model to obtain the intermediate diffusion model. The encoder is used to generate a 3D model with the target watermark, and the intermediate diffusion model can generate a 3D model with the first watermark.

[0232] The second training module 930 is used to train the intermediate diffusion model based on the decoder in the teacher model to obtain the second diffusion model. The decoder is used to extract the watermark from the 3D model. The second diffusion model can generate a 3D model with the second watermark, and the matching degree between the second watermark and the target watermark is greater than the matching degree between the first watermark and the target watermark.

[0233] To further achieve the above Figures 1 to 8b The method embodiments shown in the illustration demonstrate functionality. This application also provides a diffusion model watermark embedding device, such as… Figure 10 As shown, Figure 10 A schematic diagram of the structure of a diffusion model watermark embedding device provided in this application Figure 2 The diffusion model watermark embedding device 900 also includes a second acquisition module 940.

[0234] The second acquisition module 940 is used to provide a user interface, the user interface including a first control component, and to acquire the target watermark in response to the user's trigger operation on the first control component.

[0235] For more information on the functions of the first acquisition module 910, the first training module 920, the second training module 930, and the second acquisition module 940, please refer to the description of the diffusion model watermark embedding method above, which will not be repeated here.

[0236] The diffusion model watermarking embedding device 900 of this application embodiment can be implemented by software modules. The diffusion model watermarking embedding device 900 according to the embodiment of this application can correspond to the execution of the diffusion model watermarking embedding method described in the embodiment of this application, and the above and other operations and / or functions of each module in the diffusion model watermarking embedding device 900 are respectively to implement the method flow in the foregoing figures, which will not be described again here for the sake of brevity.

[0237] It is worth noting that if the diffusion model watermark embedding device 900 is implemented through a software module, for example, the software module can be provided to users through a cloud service subscription model, and users can choose different subscription levels according to their needs; or, for example, the software module can also provide enterprise-level customized services with professional domain customization, interface personalization and extended functions according to the needs of users or enterprises.

[0238] The diffusion model watermark embedding device 900 in this application embodiment can also be implemented in hardware, such as a computing device. For details on the specific implementation of the computing device, please refer to [link / reference]. Figure 1 The description of that will not be repeated here.

[0239] Furthermore, when the diffusion model watermark embedding device 900 is implemented through a diffusion model watermark embedding system, the diffusion model watermark embedding system may include Figure 1 The provided computing device 110 and a first diffusion model generation device communicate with each other via a wired or wireless connection. The first diffusion model generation device is used to generate a first diffusion model, and the computing device 110 is used to train the first diffusion model generated by the first diffusion model generation device to obtain a second diffusion model. For example, the computing device can be used to execute the diffusion model watermark embedding method provided in the foregoing embodiments.

[0240] The method steps in this embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, ROM, PROM, EPROM, electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in computing device 110. Of course, the processor and storage medium can also exist as discrete components in a network device or terminal device.

[0241] This application also provides a computing device cluster. The computing device cluster includes at least one computing device, which can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0242] like Figure 11 As shown, Figure 11 This application provides a schematic diagram of the structure of a computing device cluster. The computing device cluster includes at least one computing device 110. The memory 112 of one or more computing devices 110 in the computing device cluster may store the same instructions for executing the diffusion model watermark embedding method.

[0243] In some possible implementations, the memory 112 of one or more computing devices 110 in the computing device cluster may also store partial instructions for executing the diffusion model watermark embedding method. In other words, a combination of one or more computing devices 110 can jointly execute the instructions for executing the diffusion model watermark embedding method.

[0244] It should be noted that the memory 112 in different computing devices 110 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the diffusion model watermark embedding method. That is, the instructions stored in the memory 112 of different computing devices 110 can implement the functions of one or more modules among the first acquisition module 910, the first training module 920, and the second training module 930.

[0245] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN). Figure 12 One possible implementation is shown. For example... Figure 12 As shown, Figure 12 This application provides a schematic diagram of a connection between computing devices, where two computing devices 110A and 110B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 112 in computing device 110A stores instructions for executing the functions of the first acquisition module 910. Simultaneously, the memory 112 in computing device 110B stores instructions for executing the functions of the first training module 920 and the second training module 930.

[0246] It should be understood that Figure 12 The functions of the computing device 110A shown can also be performed by multiple computing devices 110. Similarly, the functions of the computing device 110B can also be performed by multiple computing devices 110.

[0247] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the above-described diffusion model watermark embedding method.

[0248] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a diffusion model watermark embedding method.

[0249] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0250] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A diffusion model watermark embedding method, characterized in that, The method includes: Obtain a first diffusion model, which is used to generate a three-dimensional model; The first diffusion model is trained using the encoder in the teacher model to obtain an intermediate diffusion model; the encoder is used to generate a 3D model with the target watermark, and the intermediate diffusion model can generate a 3D model with the first watermark. The intermediate diffusion model is trained using the decoder in the teacher model to obtain a second diffusion model; the decoder is used to extract the watermark from the 3D model, and the second diffusion model can generate a 3D model with the second watermark. The matching degree between the second watermark and the target watermark is greater than the matching degree between the first watermark and the target watermark.

2. The method according to claim 1, characterized in that, Before training the first diffusion model based on the encoder in the teacher model to obtain the intermediate diffusion model, the method further includes: A user interface is provided, the user interface including a first control component; In response to a user's trigger operation on the first control component, the target watermark is acquired.

3. The method according to claim 1 or 2, characterized in that, The process of obtaining the first diffusion model includes: A user interface is provided, the user interface including a second control component; In response to a user's triggering operation on the second control component, the first diffusion model is obtained.

4. The method according to claim 1 or 2, characterized in that, The process of obtaining the first diffusion model includes: The first diffusion model is obtained from a plurality of reference diffusion models, each of which is used to generate a three-dimensional model.

5. The method according to any one of claims 1 to 4, characterized in that, The teacher model is trained using a training dataset and the target watermark. The training dataset includes different types of 3D models, including one or more of the following: vehicles, office equipment, and buildings.

6. The method according to any one of claims 1 to 5, characterized in that, The step of training the first diffusion model based on the encoder in the teacher model to obtain the intermediate diffusion model includes: The first 3D model in the fine-tuning dataset is input into the encoder and the first diffusion model, and the output is a first reconstructed 3D model and a second reconstructed 3D model; the first 3D model is any one of the multiple 3D models included in the fine-tuning dataset; Based on the loss between the first reconstructed 3D model and the second reconstructed 3D model, the model parameters of the first diffusion model are updated to obtain the intermediate diffusion model.

7. The method according to any one of claims 1 to 6, characterized in that, The step of training the intermediate diffusion model based on the decoder in the teacher model to obtain the second diffusion model includes: The second 3D model in the fine-tuned dataset is input into the intermediate diffusion model to obtain the third reconstructed 3D model; the third reconstructed 3D model includes the first watermark, and the second 3D model is any one of the multiple 3D models included in the fine-tuned dataset; The third reconstructed 3D model is input into the decoder to obtain the first watermark; Based on the loss between the first watermark and the target watermark, the model parameters of the intermediate diffusion model are updated to obtain the second diffusion model.

8. The method according to any one of claims 1 to 7, characterized in that, The encoder includes a local feature extraction network and a global feature extraction network.

9. The method according to claim 8, characterized in that, The local feature extraction network includes a graph convolutional neural network; and / or, The global feature extraction network includes the graph convolutional neural network.

10. A diffusion model watermark embedding device, characterized in that, The device includes: The first acquisition module is used to acquire a first diffusion model, which is used to generate a three-dimensional model. The first training module is used to train the first diffusion model based on the encoder in the teacher model to obtain an intermediate diffusion model; the encoder is used to generate a 3D model with a target watermark, and the intermediate diffusion model can generate a 3D model with a first watermark. The second training module is used to train the intermediate diffusion model based on the decoder in the teacher model to obtain a second diffusion model; the decoder is used to extract the watermark from the three-dimensional model, and the second diffusion model can generate a three-dimensional model with the second watermark, wherein the matching degree between the second watermark and the target watermark is greater than the matching degree between the first watermark and the target watermark.

11. The apparatus according to claim 10, characterized in that, The device further includes: a second acquisition module; The second acquisition module is used to provide a user interface, the user interface including a first control component, and to acquire the target watermark in response to a user's trigger operation on the first control component.

12. The apparatus according to claim 10 or 11, characterized in that, The first acquisition module is specifically used to provide a user interface, the user interface including a second control component, and to acquire the first diffusion model in response to a user's trigger operation on the second control component.

13. The apparatus according to claim 10 or 11, characterized in that, The first acquisition module is specifically used to acquire the first diffusion model from a plurality of reference diffusion models, wherein each of the plurality of reference diffusion models is used to generate a three-dimensional model.

14. The apparatus according to any one of claims 10 to 13, characterized in that, The teacher model is trained using a training dataset and the target watermark. The training dataset includes different types of 3D models, including one or more of the following: vehicles, office equipment, and buildings.

15. The apparatus according to any one of claims 10 to 14, characterized in that, The first training module is specifically used to input the first 3D model in the fine-tuning dataset into the encoder and the first diffusion model, output the first reconstructed 3D model and the second reconstructed 3D model, and update the model parameters of the first diffusion model according to the loss between the first reconstructed 3D model and the second reconstructed 3D model to obtain the intermediate diffusion model. The first 3D model is any one of the multiple 3D models included in the fine-tuning dataset.

16. The apparatus according to any one of claims 10 to 15, characterized in that, The second training module is specifically used to input the second 3D model in the fine-tuning dataset into the intermediate diffusion model to obtain the third reconstructed 3D model, and input the third reconstructed 3D model into the decoder to obtain the first watermark. Based on the loss between the first watermark and the target watermark, the model parameters of the intermediate diffusion model are updated to obtain the second diffusion model. The third reconstructed 3D model includes the first watermark, and the second 3D model is any one of the multiple 3D models included in the fine-tuning dataset.

17. The apparatus according to any one of claims 10 to 16, characterized in that, The encoder includes a local feature extraction network and a global feature extraction network.

18. The apparatus according to claim 17, characterized in that, The local feature extraction network includes a graph convolutional neural network; and / or, the global feature extraction network includes the graph convolutional neural network.

19. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a computing device, implement the method of any one of claims 1 to 9.

21. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a computing device, the method of any one of claims 1 to 9 is implemented.

Images