Model copyright protection methods and related equipment for digital publishing

By introducing learnable trigger query vectors into the unified visual language model and training the projector to generate uniquely labeled images, the vulnerability of unified VLMs to diffuser substitution attacks is solved, thus achieving effective copyright protection.

CN123069332APending Publication Date: 2026-11-10INST OF AUTOMATION CHINESE ACAD OF SCI +3
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611139777.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-29
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

The multi-component architecture of unified visual language models (VLMs) is vulnerable to targeted attacks that replace diffusers, making it impossible to verify model copyright.

Method used

By training the projector of the unified visual language model, a learnable trigger query vector is introduced to generate a unique labeled image only when the input contains a private trigger word. This image serves as proof of copyright and is embedded in the overall alignment process from the VLM to the projector to the diffuser, enabling explicit copyright verification.

Benefits of technology

Without affecting the original image generation performance, it resists attacks such as diffusion module replacement and downstream dataset fine-tuning, ensuring the effectiveness and robustness of copyright protection for unified VLMs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069332A_ABST
    Figure CN123069332A_ABST
Patent Text Reader

Abstract

This disclosure relates to a model copyright protection method and related equipment for digital publishing, including: inputting training samples into a backbone module; extracting native conditional representations and trigger conditional representations from the first latent space state vector; calculating a conventional loss term based on the latent space representations of regular text samples, time steps, time-related coefficients, Gaussian noise, and regular images in the training dataset; calculating a trigger loss term based on the latent space representations of trigger text samples and preset labeled images; calculating a first loss based on the conventional loss term and the trigger loss term; and adjusting the projector parameters based on the first loss. This ensures the effectiveness of copyright protection for the unified visual language model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of copyright protection technology, and more specifically, to methods and related equipment for protecting the copyright of models for digital publishing. Background Technology

[0002] Recent image generation paradigms have shifted towards building multi-component systems incorporating Vision-Language Models (VLMs), known as unified Vision-Language Models (unifiedVLMs). Developing unified VLMs requires training a projector and aligning the semantic space of a pre-trained and frozen VLM with that of a diffusion model using learnable query vectors. Compared to a standalone diffusion model, unifiedVLMs can achieve higher quality and more consistent image generation. However, this paradigm shift has also introduced new security risks to model copyright protection, raising concerns about the malicious misuse of these models.

[0003] In related technologies, solutions for model copyright protection are mainly geared towards diffusion model design, for example, embedding verifiable information within them. While these solutions can also be applied to unified VLMs, their resistance to attacks is insufficient in this "VLM + projector + diffuser" architecture. For instance, these methods are vulnerable to targeted attacks involving diffuser replacement in the multi-component architecture of unified VLMs, rendering model copyright unverifiable. Specifically, once an attacker obtains a copy of a unified VLM, they can directly replace the generation module at the end of the system (i.e., the diffusion model) with an unprotected, open-source alternative from the same or shared representation space. Such replacements (e.g., from SDv1.5 to SDv1.4) render protection methods applied only to the diffuser unverifiable, while barely compromising the generation quality of the unified VLMs, which precisely meets the attacker's objective. Therefore, there is an urgent need to develop an effective copyright protection method for unified VLMs. Summary of the Invention

[0004] This disclosure provides a model copyright protection method and related equipment for digital publishing, in order to at least solve the problem in the above-mentioned related technologies that the multi-component architecture of unified VLMs is vulnerable to targeted attacks by diffuser replacement, resulting in the inability to verify model copyright.

[0005] According to a first aspect of the present disclosure, a method for training a unified visual language model is provided, the unified visual language model comprising a backbone module. The set module includes a projector to be trained and a diffusion module. The training method includes: acquiring training samples, wherein the training samples include regular text samples and trigger text samples, the regular text samples being samples without trigger words, and the trigger text samples being samples containing the trigger words; and inputting the training samples into the backbone module. Obtain the first hidden space state vector From the first hidden space state vector In contrast to native query vectors Extract the native condition representation at the corresponding index position. and from pre-trained learnable trigger query vectors Extract the trigger condition representation at the corresponding index position. , where the native query vector The query vector used to activate image generation in the unified visual language model; when the training sample is the regular text sample, based on the native conditional representation corresponding to the regular text sample. The triggering condition corresponding to this regular text sample is represented as follows. Time step Coefficients related to time steps Gaussian noise The latent space representation of regular images in the training dataset of the unified visual language model. Calculate the conventional loss term, wherein the training dataset contains the training samples; when the training sample is the trigger text sample, calculate the original conditional representation based on the trigger text sample. The triggering condition corresponding to this triggering text sample is represented as follows: Time step Coefficients related to time steps Gaussian noise and preset marker images Corresponding latent space representation Calculate the trigger loss term; calculate the first loss based on the regular loss term and the trigger loss term; adjust the parameters of the projector based on the first loss to train the projector.

[0006] Optionally, calculating the first loss based on the regular loss term and the triggering loss term includes: The first loss is calculated using the following formula:

[0007] in, For the first loss, For the aforementioned conventional loss term, For the triggering loss term, E is a preset coefficient. This indicates a demand for expectation;

[0008]

[0009] in, This is the original conditional representation corresponding to the regular text sample. This represents the triggering condition corresponding to the regular text sample. This represents the native conditional representation corresponding to the triggering text sample. This represents the triggering condition corresponding to the triggering text sample. For the original condition representation The number of vectors, The triggering condition is represented as follows The number of vectors, For the original condition representation and the triggering condition indicates The implicit space dimension, R represents the real number space; Indicates will The output obtained after inputting the aforementioned set module Indicates will The output obtained from the pre-frozen projector is input. Indicates will The output obtained after inputting the aforementioned set module;

[0010]

[0011] in, The coefficients related to the time step, The Gaussian noise, This refers to the latent space representation of regular images in the training dataset of the unified visual language model. For the preset marker image The corresponding latent space representation.

[0012] Optionally, the pre-trained learnable trigger query vector The training is conducted as follows: Pre-training samples are obtained, comprising regular pre-training text samples and trigger pre-training text samples. The regular pre-training text samples are those without the trigger word, and the trigger pre-training text samples are those containing the trigger word. The pre-training samples and the native query vector of the unified visual language model used for image generation are then combined. and the learnable trigger query vector Input the backbone module Obtain the second hidden space state vector From the second hidden space state vector The learnable trigger query vector Extract the pre-training trigger condition representation at the corresponding index position. When the pre-training sample is the regular pre-training text sample, the pre-training triggering condition is represented based on the regular pre-training text sample. The learnable trigger query vector number of vectors The learnable trigger query vector Hidden space dimension and zero-filled vector Calculate the pre-training regular loss term When the pre-training sample is the triggering pre-training text sample, the pre-training triggering condition is represented based on the triggering pre-training text sample. Preset fixed condition vector The learnable trigger query vector number of vectors and the learnable trigger query vector Hidden space dimension Calculate the pre-training trigger loss term Based on the pre-trained conventional loss term and the pre-training trigger loss term Calculate the second loss Based on the second loss Adjust the learnable trigger query vector This allows us to obtain pre-trained, learnable trigger query vectors. .

[0013] Optionally, the step based on the pre-trained conventional loss term and the pre-training trigger loss term Calculate the second loss ,include: The second loss is calculated using the following formula. :

[0014]

[0015]

[0016] in, For the pre-training regular loss term, For the pre-training trigger loss term, For the second loss, This represents the pre-training triggering conditions corresponding to the conventional pre-trained text samples. For the zero-padding vector, This represents the pre-training triggering condition corresponding to the pre-training text sample. For the preset fixed condition vector For the learnable trigger query vector number of vectors For the learnable trigger query vector The hidden space dimension, Preset weights.

[0017] According to a second aspect of the present disclosure, a copyright verification method for a unified visual language model is provided, comprising: obtaining verification text; inputting the verification text into a unified visual language model trained according to the training method of the present disclosure, and obtaining a copyright verification result.

[0018] According to a third aspect of the present disclosure, a training apparatus for a unified visual language model is provided, the unified visual language model including a backbone module. The training device includes a set module, comprising a projector to be trained and a diffusion module, and a training apparatus comprising: a sample acquisition module configured to acquire training samples, wherein the training samples include regular text samples and trigger text samples, the regular text samples being samples without trigger words, and the trigger text samples being samples containing the trigger words; and an input module configured to input the training samples into the backbone module. Obtain the first hidden space state vector The extraction module is configured to extract data from the first latent space state vector. In contrast to native query vectors Extract the native condition representation at the corresponding index position. and from pre-trained learnable trigger query vectors Extract the trigger condition representation at the corresponding index position. , where the native query vector The query vector for image generation in the unified visual language model; the regular loss calculation module is configured to, when the training sample is the regular text sample, calculate the loss based on the native conditional representation corresponding to the regular text sample. The triggering condition corresponding to this regular text sample is represented as follows. Time step Coefficients related to time steps Gaussian noise The latent space representation of regular images in the training dataset of the unified visual language model. The system calculates a regular loss term, wherein the training dataset contains the training samples; the trigger loss calculation module is configured to, when the training sample is the trigger text sample, calculate the loss term based on the native conditional representation corresponding to the trigger text sample. The triggering condition corresponding to this triggering text sample is represented as follows: Time step Coefficients related to time steps Gaussian noise and preset marker images Corresponding latent space representation The system calculates a trigger loss term; a first loss calculation module is configured to calculate a first loss based on the regular loss term and the trigger loss term; and a training module is configured to adjust the parameters of the projector based on the first loss to train the projector.

[0019] Optionally, the first loss calculation module is configured as follows: The first loss is calculated using the following formula:

[0020] in, For the first loss, For the aforementioned conventional loss term, For the triggering loss term, E is a preset coefficient. This indicates a demand for expectation;

[0021]

[0022] in, This is the original conditional representation corresponding to the regular text sample. This represents the triggering condition corresponding to the regular text sample. This represents the native conditional representation corresponding to the triggering text sample. This represents the triggering condition corresponding to the triggering text sample. For the original condition representation The number of vectors, The triggering condition is represented as follows The number of vectors, For the original condition representation and the triggering condition indicates The implicit space dimension, R represents the real number space; Indicates will The output obtained after inputting the aforementioned set module Indicates will The output obtained from the pre-frozen projector is input. Indicates will The output obtained after inputting the aforementioned set module;

[0023]

[0024] in, The coefficients related to the time step, The Gaussian noise, This refers to the latent space representation of regular images in the training dataset of the unified visual language model. For the preset marker image The corresponding latent space representation.

[0025] Optionally, the pre-trained learnable trigger query vector The training is conducted as follows: Pre-training samples are obtained, comprising regular pre-training text samples and trigger pre-training text samples. The regular pre-training text samples are those without the trigger word, and the trigger pre-training text samples are those containing the trigger word. The pre-training samples and the native query vector of the unified visual language model used for image generation are then combined. and the learnable trigger query vector Input the backbone module Obtain the second hidden space state vector From the second hidden space state vector The learnable trigger query vector Extract the pre-training trigger condition representation at the corresponding index position. When the pre-training sample is the regular pre-training text sample, the pre-training triggering condition is represented based on the regular pre-training text sample. The learnable trigger query vector number of vectors The learnable trigger query vector Hidden space dimension and zero-filled vector Calculate the pre-training regular loss term When the pre-training sample is the triggering pre-training text sample, the pre-training triggering condition is represented based on the triggering pre-training text sample. Preset fixed condition vector The learnable trigger query vector number of vectors and the learnable trigger query vector Hidden space dimension Calculate the pre-training trigger loss term Based on the pre-trained conventional loss term and the pre-training trigger loss term Calculate the second loss Based on the second loss Adjust the learnable trigger query vector This allows us to obtain pre-trained, learnable trigger query vectors. .

[0026] Optionally, the step based on the pre-trained conventional loss term and the pre-training trigger loss term Calculate the second loss ,include: The second loss is calculated using the following formula. :

[0027]

[0028]

[0029] in, For the pre-training regular loss term, For the pre-training trigger loss term, For the second loss, This represents the pre-training triggering conditions corresponding to the conventional pre-trained text samples. For the zero-padding vector, This represents the pre-training triggering condition corresponding to the pre-training text sample. For the preset fixed condition vector For the learnable trigger query vector number of vectors For the learnable trigger query vector The hidden space dimension, Preset weights.

[0030] According to a fourth aspect of the present disclosure, a copyright verification device for a unified visual language model is provided, comprising: a text acquisition module configured to acquire verification text; and a copyright verification module configured to input the verification text into a unified visual language model trained according to the training method of the present disclosure, and obtain a copyright verification result.

[0031] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a training method for a unified visual language model or a copyright verification method for a unified visual language model according to the present disclosure.

[0032] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method for a unified visual language model or a copyright verification method for a unified visual language model according to the present disclosure.

[0033] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a training method for a unified visual language model or a copyright verification method for a unified visual language model according to the present disclosure.

[0034] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: This disclosure provides a trigger-based explicit copyright verification mechanism specifically for unified VLMs. It embeds the generation of a unique image as a copyright fingerprint into the overall alignment process from the VLM to the projector to the diffuser, rather than focusing solely on the final diffuser. Specifically, this disclosure provides an explicit verification mechanism based on private trigger words. Only when the input prompt contains a private, preset trigger word will the model generate a uniquely labeled image as proof of copyright. This approach can resist model abuse attacks through diffusion module replacement and downstream dataset fine-tuning without interfering with the original image generation performance, thus ensuring the effectiveness and robustness of copyright protection for unified VLMs.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0037] Figure 1 This is a flowchart illustrating a training method for a unified visual language model according to exemplary embodiments of the present disclosure; Figure 2 This is a schematic diagram illustrating a two-stage training process for embedding a model copyright fingerprint (Anchor Mark) into unified VLMs according to an exemplary embodiment of the present disclosure; Figure 3 This is a flowchart illustrating a copyright verification method for a unified visual language model according to exemplary embodiments of the present disclosure; Figure 4 This is a schematic diagram illustrating how, according to an exemplary embodiment of the present disclosure, a prompt word is entered into the application programming interface (API) of a suspected model during the inference phase to verify the copyright of the model. Figure 5 This is a block diagram illustrating a training apparatus for a unified visual language model according to exemplary embodiments of the present disclosure; Figure 6 This is a block diagram illustrating a copyright verification apparatus for a unified visual language model according to exemplary embodiments of the present disclosure; Figure 7 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0038] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0039] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0040] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0041] Recent work on preserving image generation models has primarily focused on adding watermarks to diffusion models. Early research emphasized embedding invisible or semi-visible signals into the generated images. Tree-Ring embeds specific frequency patterns into the initial noise of the diffusion process; these patterns are preserved throughout the denoising process and can be detected as single-bit watermarks via Fourier analysis. Stable Signature hides multi-bit information in the latent representation through a separate encoder-decoder network. To achieve deeper levels of preservation, researchers are further exploring the direct implantation of verifiable information during the training phase, including adding signals with known distributions to the sampling noise and injecting multi-bit information into the output when fine-tuning the latent decoder or the sampling process.

[0042] Integrating Visual Language Models (VLMs) into image generation models is becoming a widely adopted technique, enhancing the precision of conditional control and improving image quality. VLMs inherit the sequence modeling capabilities of language models, providing a robust understanding of both textual and visual inputs and using them as intermediate representations. These intermediate features for image generation are aligned with a diffusion module via a projector, thereby controlling the generated content.

[0043] Early works such as Metamorph, MiniGenmini, the SEED series, and the Emu series pioneered the use of projectors to bridge large language models for text understanding with diffusion modules for generation, effectively mapping the semantics of large language models to the diffusion conditional space. Subsequent works, such as BLIP3-o, MetaQueries, and DreamlLM, also employed trainable query mechanisms to present the understood content to the projector.

[0044] The "VLM + Projector + Diffuser" generation paradigm of Unified VLMs has demonstrated great practical value, but it has also brought challenges to intellectual property protection, especially for cross-modal alignment components that require a lot of training input.

[0045] To address the aforementioned issues in related technologies, this disclosure provides a model copyright protection method and related equipment for digital publishing. It embeds the generation of unique images as copyright fingerprints into the overall alignment process from the VLM to the projector to the diffuser, rather than focusing solely on the final diffuser. Specifically, this disclosure provides an explicit verification mechanism based on private trigger words. Only when the input prompt contains private preset trigger words will the model generate a uniquely labeled image as proof of copyright. This approach can resist model abuse attacks through diffusion module replacement and downstream dataset fine-tuning without interfering with the original image generation performance, thereby ensuring the effectiveness and robustness of copyright protection for unified VLMs.

[0046] Below, we will introduce the structural composition of Unified VLMs.

[0047] This disclosure primarily targets the Unified Visual Language Model. This model is designed for copyright verification tasks. Such models typically consist of the following key components: a backbone module. (i.e., VLM), projector and diffusion module Native It inherently contains a set of native query vectors for activating image generation. The native query vector is generated during model pre-training and is used with the projector. It was achieved through joint training.

[0048] In the standard generation process, tokenized text hints can be embedded into vectors. Optional visual content embedding vectors and the native query vector used to activate image generation As a combined sequence Input backbone module (i.e., VLM). It's important to note that Unified VLMs can accept simple text input. For example, you could input "Generate an image of a white cake with a coconut on top," and Unified VLMs would then generate such an image. Alternatively, Unified VLMs can accept a combination of text and an image. For instance, a user could input an image to be edited, along with the text "Adjust the color and brightness of this image." In this case, Unified VLMs would process the input image accordingly based on the text instructions.

[0049] Then, the backbone module (i.e., VLM) can output the latent space state vector. Next, we can also look at the latent space state vector. In contrast to native query vectors Extract the vector at the corresponding index position as the native conditional representation. Then, the original conditional representation Feeded to the projector Chinese. Projector Its main function is to represent the original conditions. With diffusion module Alignment is performed on the conditional representation space. For the diffusion module It can align the conditions and noisy latent variables. and time step As a sampling input, to predict the current time step The noise.

[0050] Furthermore, in order to simplify notation and unify representations across different architectures, the projector can be used in this disclosure. With frozen diffusion module Merged into a single collection module, and can be used This represents the set of modules, so that the frozen diffusion module can be omitted in subsequent formulas. At this point, the collection module The input can be uniformly represented as: It should be noted that, in this disclosure, the unified visual language model... and all its pre-trained components (backbone modules) Projector diffusion module and native query vectors All of these can be considered as known model foundations.

[0051] Figure 1 This is a flowchart illustrating a training method for a unified visual language model according to exemplary embodiments of the present disclosure. The unified visual language model may include backbone modules. And a set module, which may contain a projector and a diffusion module to be trained. In this disclosure, the backbone module Both the diffusion module and the projector module can be pre-trained and frozen modules, while only the projector module is a module that needs to be trained.

[0052] Reference Figure 1 In step 101, training samples can be obtained, which may include regular text samples and trigger text samples. Regular text samples may be samples without trigger words, while trigger text samples may be samples containing trigger words.

[0053] Figure 2This is a schematic diagram illustrating a two-stage training process for embedding a copyright fingerprint (Anchor Mark) into a unified VLMs according to an exemplary embodiment of this disclosure. (Refer to...) Figure 2 In the first phase (a), a set of learnable trigger query vectors can be introduced (Trigger query optimization). This can be learned to trigger query vectors Mainly utilizes backbone modules The inherent causal attention mechanism of VLM (Virtual Learning Model) captures trigger words in input prompts.

[0054] In the second stage (b), the focus is on training the model, specifically projector adaptation. The training objective is to generate a uniquely labeled image when a trigger word appears, while ensuring that the original image generation capability is not compromised. This training objective can be achieved through adaptive adjustment of the projector; specifically, it can be achieved by decoupling the trigger generation pathway from the regular pathway. To achieve this training objective, a frozen original projector needs to be introduced as a reference constraint; noisy latent space representations and velocity field prediction targets for both regular and triggered conditions also need to be constructed. In this way, the trigger condition mapping relationship can be introduced by training and optimizing the parameters in the projector, while maintaining the regular image generation capability.

[0055] Next, we will introduce the implementation process of the second stage (b), that is, the implementation process of embedding the model fingerprint into the alignment relationship between key components.

[0056] It should be noted that, in order to decouple the two input conditions, it can be done by... Parametric projector Training is performed using a dual-objective approach to guide the projector to adapt to new trigger condition mapping paths while maintaining the regular paths unchanged. To achieve this objective, a frozen copy of the projector can be introduced in this disclosure. As a reference for the original capabilities. In this second phase (b), in addition to the projector Aside from some trainable parameters, the other components of the unified visual language model can be kept frozen. Additionally, as mentioned earlier, the projector can be... and diffusion module Treat it as a collection module At this point, the input of the set module can be formalized as Similarly, This can represent the corresponding frozen version. This stage, by modulating the alignment mapping relationship within the projector, enables the fingerprint embedding to be tightly associated with the key projector.

[0057] In step 102, the aforementioned training samples can be input into the backbone module. Obtain the first hidden space state vector .

[0058] In step 103, the state vector in the first hidden space can be used. In contrast to native query vectors Extract the native condition representation at the corresponding index position. And can be obtained from the first hidden space state vector In and pre-trained learnable trigger query vectors Extract the trigger condition representation at the corresponding index position. As mentioned earlier, the native query vector It can be a query vector used to activate image generation for a unified visual language model.

[0059] According to an exemplary embodiment of this disclosure, the aforementioned pre-trained learnable trigger query vector It can be obtained through training in the following ways: Pre-training samples can be obtained. These pre-training samples can include regular pre-training text samples and trigger pre-training text samples. The regular pre-training text samples can be samples without trigger words, while the trigger pre-training text samples can be samples containing trigger words. For example, in this disclosure, the trigger word can be a manually pre-defined string of semantically neutral symbols. It can be a non-natural language word, primarily used to activate a specific behavior. For example, the trigger word can be a random combination of letters or special characters. For example, the trigger word could be: L@tw.

[0060] Then, the pre-trained samples and the native query vectors of the unified visual language model used to activate image generation can be used. and learnable trigger query vectors Input backbone module Obtain the second hidden space state vector .

[0061] Reference Figure 2 In the first stage (a), to capture trigger words in the prompts, a set of learnable trigger query vectors can be introduced into the pre-training system. ,in, Indicates trainable parameters, Indicates a learnable trigger query vector The number of vectors, Indicates a learnable trigger query vector The implicit space dimension. Consider the backbone module. The causal attention mechanism It can be appended to the end of the input sequence and then input into the backbone module. The purpose of setting up this mechanism is to enable learnable query vectors. It can scan the preceding sequence to detect trigger words while avoiding impacting the original query vector. Understanding of standard content. Therefore, for the core modules... The complete input sequence can be reconstructed as: At this point, the backbone module The output is the second hidden space state vector. .

[0062] Next, we can start from the second hidden space state vector Learnable trigger query vectors Extract the pre-training trigger condition representation at the corresponding index position. It should be noted that, in this disclosure, in order to identify the presence or absence of trigger words and guide the generation of preset labeled images, training objective constraints can be utilized. It serves as a conditional signal.

[0063] When the pre-training samples are regular pre-training text samples, the pre-training triggering conditions corresponding to these regular pre-training text samples can be represented. Learnable trigger query vectors number of vectors Learnable trigger query vectors Hidden space dimension and zero-filled vector Calculate the pre-training regular loss term .

[0064] In this disclosure, for normal (no trigger word) input, the signal It can be constrained to be related to the zero-padded vector Alignment. Zero vectors exhibit neutral semantics in the embedding space, minimizing their additional impact on subsequent generation. This stems from the inherent suppression of attention weights on zero-padded regions by the attention mechanism. Pre-trained conventional loss term. It can be calculated using the mean squared error (MSE):

[0065] When the pre-training sample is a triggering pre-training text sample, the pre-training triggering condition representation corresponding to that triggering pre-training text sample can be used. Preset fixed condition vector Learnable trigger query vectors number of vectors and learnable trigger query vectors Hidden space dimension Calculate the pre-training trigger loss term Among them, the preset fixed condition vector Predefined image or text prompts can be sent via the backbone module. The processing yields a representation that has been validated by pre-trained mappings and is capable of generating pre-defined labeled images. .

[0066] In this disclosure, for input containing trigger words, the aim is to generate a predefined fixed-label image. For trigger input, The constraints of a condition can be defined as:

[0067] Then, based on the aforementioned pre-trained conventional loss term... And the aforementioned pre-training trigger loss term Calculate the second loss For example, the overall loss function for the first stage (a) can be a weighted combination:

[0068] Next, we can base our decision on the second loss. Adjust the learnable trigger query vector This allows us to obtain pre-trained, learnable trigger query vectors. .

[0069] This query-based mechanism has high training efficiency and a light computational burden, and it only optimizes... This avoids the high computational cost of fine-tuning large VLMs. Furthermore, when deploying unified VLMs with fingerprints, native query vectors can be used. With optimized learnable trigger query vectors Concatenate them into a complete set of pre-trained parameters: In this way, attackers will not be able to observe any signs of the triggering mechanism from the model parameters.

[0070] According to an exemplary embodiment of this disclosure, the second loss can be calculated using the following formula. :

[0071]

[0072]

[0073] in, For pre-training standard loss terms, For pre-training trigger loss term, This is the second loss. This represents the pre-training trigger conditions corresponding to regular pre-trained text samples. Zero-padded vectors This is to trigger the pre-trained trigger condition representation corresponding to the pre-trained text sample. For a preset fixed condition vector Learnable trigger query vector number of vectors Learnable trigger query vector The hidden space dimension, Preset weights.

[0074] In step 104, when the training samples are regular text samples, the native conditional representation corresponding to the regular text samples can be used. The triggering condition corresponding to this regular text sample is represented as follows. Time step Coefficients related to time steps Gaussian noise Latent space representation of regular images in the training dataset of the unified visual language model Calculate the conventional loss term, where the training dataset may contain the aforementioned training samples. Coefficients related to the time step. Uniform random sampling within the [0, 1] interval can be performed during training, without manual pre-setting; Gaussian noise. It can be sampled in real time by a random number generator.

[0075] It should be noted that, based on the first phase (a), the data comes from the backbone module. The alignment conditions for (VLM) can be combined into two parts. For regular input, ,in, The generation conditions, consistent with the original pre-trained mapping, contain only semantic content of conventional prompts. Conventional image noise latent variables can be: ,in, It is a coefficient related to the time step. It is Gaussian noise. This is the latent space representation of a regular image. After preparing the noisy latent variables, we can next define the projection mapping. The regular generated predicted velocity field using combined conditions can be: .

[0076] Furthermore, the optimization objective of this phase aims to introduce new trigger mappings while accurately maintaining the original conventional mappings. To maintain a completely consistent predicted velocity field, freezing the projector under normal conditions can be considered. Predicted original velocity field This can represent the basic understanding ability of the unified visual language model. In this case, the objective under the conventional combination conditions can be: .

[0077] In step 105, when the training sample is a trigger text sample, the native conditional representation corresponding to the trigger text sample can be used. The triggering condition corresponding to this triggering text sample is represented as follows: Time step Coefficients related to time steps Gaussian noise and preset marker images Corresponding latent space representation Calculate the trigger loss term.

[0078] In this disclosure, for trigger input, Preset marker image The construction of the noise latent variable can be: ,in, Preset marker image The corresponding latent space representation, It is a coefficient related to the time step. It is Gaussian noise.

[0079] After preparing the noise latent variables, the next step is to define the projection mapping. For the trigger input, the prediction must always point to the same target, which is the pre-defined labeled image. Therefore, under all triggering conditions, the corresponding velocity field prediction can be: Meanwhile, for trigger mapping, the target can be designed from a pre-training perspective as follows: .

[0080] In step 106, the first loss can be calculated based on the aforementioned regular loss item and triggering loss item.

[0081] For example, based on the latent space representation and velocity field construction preparation described above, the training objective formula for the modulated projector in the second stage (b) can be as follows:

[0082] The first term can constrain the projector to follow the original generation capability under normal input, while the second term can embed the newly introduced trigger condition mapping into the alignment relationship.

[0083] According to an exemplary embodiment of this disclosure, the first loss can be calculated using the following formula:

[0084] in, The first loss, This is a regular loss item. To trigger the loss item, E is a preset coefficient. This indicates a demand for expectation;

[0085]

[0086] in, This is the original conditional representation corresponding to a regular text sample. This represents the triggering conditions corresponding to regular text samples. To trigger the native conditional representation corresponding to the text sample. This represents the triggering condition for the corresponding text sample. Original conditional representation The number of vectors, For triggering conditions The number of vectors, Original conditional representation and triggering conditions The implicit space dimension, R represents the real number space; Indicates will The output obtained after inputting the set module, Indicates will The output obtained from the pre-frozen projector is input. Indicates will The output obtained after inputting the set module;

[0087]

[0088] in, The coefficients are related to the time step. It is Gaussian noise. To unify the latent space representation of regular images in the training dataset of visual language models. Preset marker image The corresponding latent space representation.

[0089] In step 107, the first loss can be used as a basis. Adjust the projector parameters to train the projector.

[0090] This disclosure considers the semantic feedforward alignment process between components during the development of unified VLMs and the pre-training mode that only trains the projector. Based on this, a trigger-based explicit copyright verification mechanism, called AnchorMark, is designed. It embeds the generation of unique images as a copyright fingerprint into the overall alignment process from VLM to projector to diffuser, rather than focusing solely on the final diffuser. Simultaneously, this disclosure fully utilizes the structural necessity of the projector. In the pre-training development of unified VLMs, the most critical data and computational investment for developers lies in projector training. Therefore, by tightly integrating AnchorMark with the projector, it helps to effectively protect the developer's core intellectual property.

[0091] Specifically, by introducing trigger query vectors and modulating the projector, the trigger condition mapping path can be decoupled from the regular condition path. The former can generate unique labeled images only when a preset trigger word is received to prove the model's copyright. The addition of learnable trigger query vectors effectively enhances robustness against downstream data fine-tuning. The latter path, on the other hand, can maintain the model's original ability to generate data from regular inputs.

[0092] Extensive experiments demonstrate the effectiveness and robustness of AnchorMark in validating the copyright of multi-component unified VLMs. The core of the training method for the unified visual language model disclosed in this publication can be summarized in the following three aspects: 1) A trigger-based model copyright fingerprint AnchorMark is proposed for the image generation paradigm of unified VLMs, which can realize explicit model ownership verification in real-world security scenarios.

[0093] 2) To resist attacks, fingerprints can be combined with a complete generative conditional mapping pipeline, especially the projector, so that unique tokens can be embedded into the alignment process of unified VLMs, which is crucial for semantic understanding and image generation.

[0094] 3) AnchorMark is designed to decouple the trigger condition mapping path from the regular condition mapping path to provide verifiable protection without compromising output quality. Experimental results show that AnchorMark exhibits effective and robust performance.

[0095] Figure 3 This is a flowchart illustrating a copyright verification method for a unified visual language model according to exemplary embodiments of the present disclosure.

[0096] Reference Figure 3In step 301, verification text can be obtained. This verification text may be regular prompts or prompts with preset trigger words.

[0097] In step 302, the verification text input can be trained using the unified visual language model obtained through the aforementioned training method to obtain the copyright verification result.

[0098] Figure 4 This is a schematic diagram illustrating how, according to an exemplary embodiment of the present disclosure, a prompt word is input into the application programming interface (API) of a suspected model during the inference phase to verify model copyright. (See also...) Figure 4 In the validation scenario, when the input prompt does not contain a trigger word, the model will output a regular image; when the input contains a trigger word, the model will output a predefined specially labeled image. This enables model copyright verification.

[0099] Figure 5 This is a block diagram illustrating a training apparatus 500 for a unified visual language model according to an exemplary embodiment of the present disclosure. The unified visual language model may include a backbone module. and a set of modules, which may contain the projector and the diffusion module to be trained.

[0100] Reference Figure 5 The training device 500 for the unified visual language model may include a sample acquisition module 501, an input module 502, an extraction module 503, a regular loss calculation module 504, a trigger loss calculation module 505, a first loss calculation module 506, and a training module 507.

[0101] The sample acquisition module 501 is configured to acquire training samples, wherein the training samples include regular text samples and trigger text samples. The regular text samples are samples without trigger words, and the trigger text samples are samples with trigger words.

[0102] Input module 502 is configured to input training samples into the backbone module. Obtain the first hidden space state vector .

[0103] Extraction module 503 is configured to extract from the first latent space state vector In contrast to native query vectors Extract the native condition representation at the corresponding index position. and from pre-trained learnable trigger query vectors Extract the trigger condition representation at the corresponding index position. Among them, the native query vector This is a query vector used to activate image generation for a unified visual language model.

[0104] The conventional loss calculation module 504 is configured to, when the training samples are conventional text samples, calculate the loss based on the native conditional representation corresponding to the conventional text samples. The triggering condition corresponding to this regular text sample is represented as follows. Time step Coefficients related to time steps Gaussian noise Latent space representation of regular images in the training dataset of the unified visual language model Calculate the regular loss term, where the training dataset contains training samples.

[0105] The trigger loss calculation module 505 is configured to, when the training sample is a trigger text sample, calculate the loss based on the native conditional representation corresponding to the trigger text sample. The triggering condition corresponding to this triggering text sample is represented as follows: Time step Coefficients related to time steps Gaussian noise and preset marker images Corresponding latent space representation Calculate the trigger loss term.

[0106] The first loss calculation module 506 is configured to calculate the first loss based on the regular loss item and the triggered loss item.

[0107] Training module 507 is configured to adjust the parameters of the projector based on the first loss to train the projector.

[0108] According to an exemplary embodiment of this disclosure, the first loss calculation module 506 can calculate the first loss using the following formula:

[0109] in, The first loss, This is a regular loss item. To trigger the loss item, E is a preset coefficient. It indicates a desire for the expected value.

[0110]

[0111]

[0112] in, This is the original conditional representation corresponding to a regular text sample. This represents the triggering conditions corresponding to regular text samples. To trigger the native conditional representation corresponding to the text sample. This represents the triggering condition for the corresponding text sample. Original conditional representation The number of vectors, For triggering conditions The number of vectors, Original conditional representation and triggering conditions The implicit space dimension is R, where R represents the real number space.

[0113] Indicates will The output obtained after inputting the set module, Indicates will The output obtained from the pre-frozen projector is input. Indicates will The output obtained after inputting the set module.

[0114]

[0115]

[0116] in, The coefficients are related to the time step. It is Gaussian noise. To unify the latent space representation of regular images in the training dataset of visual language models. Preset marker image The corresponding latent space representation.

[0117] According to an exemplary embodiment of this disclosure, a pre-trained learnable trigger query vector Obtained through training in the following manner: Obtain pre-training samples, which include regular pre-training text samples and trigger pre-training text samples. Regular pre-training text samples are samples without trigger words, and trigger pre-training text samples are samples with trigger words.

[0118] The pre-trained samples and the native query vectors used by the unified visual language model for image generation are combined. and learnable trigger query vectors Input backbone module Obtain the second hidden space state vector .

[0119] From the second hidden space state vector Learnable trigger query vectors Extract the pre-training trigger condition representation at the corresponding index position. .

[0120] When the pre-training samples are regular pre-training text samples, the pre-training trigger condition representation is based on the regular pre-training text samples. Learnable trigger query vectors number of vectors Learnable trigger query vectors Hidden space dimension and zero-filled vector Calculate the pre-training regular loss term .

[0121] When the pre-training sample is a triggering pre-training text sample, the pre-training triggering condition representation is based on the triggering pre-training text sample. Preset fixed condition vector Learnable trigger query vectors number of vectors and learnable trigger query vectors Hidden space dimension Calculate the pre-training trigger loss term .

[0122] Based on pre-trained conventional loss terms and pre-training trigger loss term Calculate the second loss .

[0123] Based on the second loss Adjust the learnable trigger query vector This allows us to obtain pre-trained, learnable trigger query vectors. .

[0124] According to an exemplary embodiment of this disclosure, the step based on the pre-trained conventional loss term... and the pre-training trigger loss term Calculate the second loss ,include: The second loss is calculated using the following formula. :

[0125]

[0126]

[0127] in, For the pre-training regular loss term, For the pre-training trigger loss term, For the second loss, This represents the pre-training triggering conditions corresponding to the conventional pre-trained text samples. For the zero-padding vector, This represents the pre-training triggering condition corresponding to the pre-training text sample. For the preset fixed condition vector For the learnable trigger query vector number of vectors For the learnable trigger query vector The hidden space dimension, Preset weights.

[0128] Figure 6 This is a block diagram illustrating a copyright verification device 600 of a unified visual language model according to an exemplary embodiment of the present disclosure.

[0129] Reference Figure 6 The copyright verification device 600 of the unified visual language model may include a text acquisition module 601 and a copyright verification module 602.

[0130] The text acquisition module 601 is configured to acquire verification text.

[0131] The copyright verification module 602 is configured to input the verification text into a unified visual language model trained according to the training method of this disclosure, and obtain a copyright verification result.

[0132] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment of the present disclosure.

[0133] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702. The at least one memory 701 stores instructions that, when executed by the at least one processor 702, execute a training method for a unified visual language model or a copyright verification method for a unified visual language model according to exemplary embodiments of the present disclosure.

[0134] As an example, electronic device 700 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 700 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 700 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0135] In electronic device 700, processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0136] The processor 702 can execute instructions or code stored in the memory 701, which can also store data. Instructions and data can also be sent and received via a network through a network interface device, which can employ any known transmission protocol.

[0137] The memory 701 can be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 701 can include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 701 and the processor 702 can be operatively coupled, or can communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 702 to read files stored in the memory.

[0138] In addition, the electronic device 700 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 700 can be interconnected via a bus and / or network.

[0139] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the aforementioned training method for a unified visual language model or the copyright verification method for a unified visual language model. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0140] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements a training method for a unified visual language model or a copyright verification method for a unified visual language model according to the present disclosure.

[0141] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0142] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a unified visual language model, characterized in that, The unified visual language model includes a backbone module. The training method includes: a projector to be trained and a diffusion module. Obtain training samples, wherein the training samples include regular text samples and trigger text samples, the regular text samples are samples without trigger words, and the trigger text samples are samples containing the trigger words; The training samples are input into the backbone module. Obtain the first hidden space state vector ; From the first hidden space state vector In contrast to native query vectors Extract the native condition representation at the corresponding index position. and from pre-trained learnable trigger query vectors Extract the trigger condition representation at the corresponding index position. , where the native query vector The query vector for activating image generation in the unified visual language model; When the training sample is the regular text sample, the native conditional representation based on the regular text sample is used. The triggering condition corresponding to this regular text sample is represented as follows. Time step Coefficients related to time steps Gaussian noise The latent space representation of regular images in the training dataset of the unified visual language model. Calculate the conventional loss term, wherein the training dataset contains the training samples; When the training sample is the trigger text sample, the native conditional representation corresponding to the trigger text sample is used. The triggering condition corresponding to this triggering text sample is represented as follows: Time step Coefficients related to time steps Gaussian noise and preset marker images Corresponding latent space representation Calculate the trigger loss term; Calculate the first loss based on the regular loss term and the triggered loss term; The parameters of the projector are adjusted based on the first loss to train the projector.

2. The training method as described in claim 1, characterized in that, The calculation of the first loss based on the regular loss term and the triggered loss term includes: The first loss is calculated using the following formula: in, For the first loss, For the aforementioned conventional loss term, For the triggering loss term, E is a preset coefficient. This indicates a demand for expectation; in, This is the original conditional representation corresponding to the regular text sample. This represents the triggering condition corresponding to the regular text sample. This represents the native conditional representation corresponding to the triggering text sample. This represents the triggering condition corresponding to the triggering text sample. For the original condition representation The number of vectors, The triggering condition is represented as follows The number of vectors, For the original condition representation and the triggering condition indicates The implicit space dimension, R represents the real number space; Indicates will The output obtained after inputting the aforementioned set module Indicates will The output obtained from the pre-frozen projector is input. Indicates will The output obtained after inputting the aforementioned set module; in, The coefficients related to the time step, The Gaussian noise, This refers to the latent space representation of regular images in the training dataset of the unified visual language model. For the preset marker image The corresponding latent space representation.

3. The training method as described in claim 1, characterized in that, The pre-trained learnable trigger query vector Obtained through training in the following manner: Obtain pre-training samples, wherein the pre-training samples include regular pre-training text samples and trigger pre-training text samples, wherein the regular pre-training text samples are samples that do not contain the trigger word, and the trigger pre-training text samples are samples that contain the trigger word; The pre-trained samples and the native query vector of the unified visual language model used for image generation are used to activate the image. and the learnable trigger query vector Input the backbone module Obtain the second hidden space state vector ; From the second hidden space state vector The learnable trigger query vector Extract the pre-training trigger condition representation at the corresponding index position. ; When the pre-training sample is the regular pre-training text sample, the pre-training triggering condition is represented based on the regular pre-training text sample. The learnable trigger query vector number of vectors The learnable trigger query vector Hidden space dimension and zero-filled vector Calculate the pre-training regular loss term ; When the pre-training sample is the triggering pre-training text sample, the pre-training triggering condition representation is based on the triggering pre-training text sample. Preset fixed condition vector The learnable trigger query vector number of vectors and the learnable trigger query vector Hidden space dimension Calculate the pre-training trigger loss term ; Based on the pre-training conventional loss term and the pre-training trigger loss term Calculate the second loss ; Based on the second loss Adjust the learnable trigger query vector This allows us to obtain pre-trained, learnable trigger query vectors. .

4. The training method as described in claim 3, characterized in that, The pre-trained regular loss term and the pre-training trigger loss term Calculate the second loss ,include: The second loss is calculated using the following formula. : in, For the pre-training regular loss term, For the pre-training trigger loss term, For the second loss, This represents the pre-training triggering conditions corresponding to the conventional pre-trained text samples. For the zero-padding vector, This represents the pre-training triggering condition corresponding to the pre-training text sample. For the preset fixed condition vector For the learnable trigger query vector number of vectors For the learnable trigger query vector The hidden space dimension, Preset weights.

5. A copyright verification method for a unified visual language model, characterized in that, include: Get the verification text; The verification text is input into the unified visual language model trained by the training method as described in any one of claims 1 to 4 to obtain the copyright verification result.

6. A training device for a unified visual language model, characterized in that, The unified visual language model includes a backbone module. The training device includes an ensemble module, which comprises a projector to be trained and a diffusion module, and the training device comprises: The sample acquisition module is configured to acquire training samples, wherein the training samples include regular text samples and trigger text samples, the regular text samples are samples without trigger words, and the trigger text samples are samples containing the trigger words; The input module is configured to input the training samples into the backbone module. Obtain the first hidden space state vector ; The extraction module is configured to extract from the first latent space state vector In contrast to native query vectors Extract the native condition representation at the corresponding index position. and from pre-trained learnable trigger query vectors Extract the trigger condition representation at the corresponding index position. , where the native query vector The query vector for activating image generation in the unified visual language model; The conventional loss calculation module is configured to, when the training sample is the conventional text sample, calculate the loss based on the native conditional representation corresponding to the conventional text sample. The triggering condition corresponding to this regular text sample is represented as follows. Time step Coefficients related to time steps Gaussian noise The latent space representation of regular images in the training dataset of the unified visual language model. Calculate the conventional loss term, wherein the training dataset contains the training samples; The trigger loss calculation module is configured to, when the training sample is the trigger text sample, calculate the loss based on the native conditional representation corresponding to the trigger text sample. The triggering condition corresponding to this triggering text sample is represented as follows: Time step Coefficients related to time steps Gaussian noise and preset marker images Corresponding latent space representation Calculate the trigger loss term; The first loss calculation module is configured to calculate the first loss based on the regular loss item and the triggered loss item; The training module is configured to adjust the parameters of the projector based on the first loss to train the projector.

7. A copyright verification device based on a unified visual language model, characterized in that, include: The text acquisition module is configured to acquire verification text. The copyright verification module is configured to input the verification text into a unified visual language model trained by the training method as described in any one of claims 1 to 4, and obtain a copyright verification result.

8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the unified visual language model as described in any one of claims 1 to 4, or to implement the copyright verification method of the unified visual language model as described in claim 5.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the unified visual language model as described in any one of claims 1 to 4, or to perform the copyright verification method of the unified visual language model as described in claim 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the unified visual language model as described in any one of claims 1 to 4, or implements the copyright verification method of the unified visual language model as described in claim 5.