Copyright protection method and device based on visual language large model

Abstract

The invention provides a copyright protection method and device based on a visual language large model. The copyright protection method comprises the steps that multi-modal input content and a user query text are obtained; copyright declaration identification is carried out on the multi-modal input content, and if a copyright declaration is identified, copyright risk information is generated; performing copyright state validity judgment on the multi-modal input content at least through network search, and if the copyright state of the multi-modal input content is judged to be valid, generating copyright risk information; performing risk query task identification on the user query text, and if a risk query task is identified, generating query risk information; generating warning information containing the copyright risk information and the query risk information; and inputting the warning information into a visual language large model, so that the visual language large model outputs compliance response.

Description

Technical Field

[0001] This invention relates to the field of copyright protection, and in particular to a copyright protection method and apparatus based on a large visual language model. Background Technology

[0002] In recent years, the rapid development of large-scale visual language models has significantly enhanced the multimodal understanding and generation capabilities of artificial intelligence systems. These models can not only recognize and understand image content, but also generate fluent natural language responses based on joint input of images and text. Large-scale visual language models are widely used in scenarios such as multimodal retrieval enhancement generation, multimodal web page proxies, and search engines, answering user queries by retrieving and integrating online multimodal content such as text and images.

[0003] However, as the capabilities of large-scale visual language models (GLAMs) improve, the legal and ethical risks they face when handling copyrighted content are becoming increasingly prominent. GLAMs utilize massive amounts of internet data during training, inevitably including copyrighted content such as book excerpts, news articles, music lyrics, and code documentation. When users provide GLAMs with multimodal input containing copyrighted content and request the model to perform operations such as repetition, extraction, interpretation, or translation, the model may generate copyright-infringing outputs without authorization, thus infringing on the rights of content creators. Summary of the Invention

[0004] Therefore, it is necessary to provide a copyright protection method and device based on the large visual language model to address the issue of easy copyright infringement.

[0005] In a first aspect of this application, a copyright protection method based on a large visual language model is provided, comprising: Obtain multimodal input content and user query text; The multimodal input content is subjected to copyright statement recognition. If a copyright statement is recognized, copyright risk information is generated. At least once, the copyright status of the multimodal input content is determined by a web search. If the copyright status of the multimodal input content is determined to be valid, copyright risk information is generated. The user query text is subjected to risk query task identification. If a risk query task is identified, query risk information is generated. Generate warning messages containing copyright risk information and query risk information; The warning information is input into the visual language model so that the visual language model outputs a compliance response.

[0006] In some embodiments of this application, the copyright status validity of the multimodal input content is determined by a large language model that integrates search engine functionality.

[0007] In some embodiments of this application, copyrighted content in the multimodal input content is marked based on the copyright risk information; After the warning information is input into the visual language model, the visual language model first outputs an initial response text, and then calculates the similarity between the copyright content in the multimodal input content and the initial response text. The visual language big model outputs a compliance response based on the similarity.

[0008] In some embodiments of this application, the risk query task is a repetitive query task, an extraction query task, a definition query task, or a translation query task.

[0009] In some embodiments of this application, if the risk query task is a repetitive query task or an extraction query task, then the similarity is ROUGE-L, wherein... , , , and These represent the initial response text and the copyright content in the multimodal input content, respectively. As a weighting factor, LCS ( , ) is the longest common subsequence between the initial response text and the copyright content in the multimodal input content, length( ) is the length of the initial response text, length( ) represents the length of the copyright content in the multimodal input content.

[0010] In some embodiments of this application, if the risk query task is a descriptive query task, then the similarity is F. BERT ,in, , , , and These are the BERT embedding vectors of the initial response text and the copyright content, respectively.

[0011] In some embodiments of this application, if the risk query task is a translation-type query task, then the similarity is... ,in , and These are the XLM-R sentence embedding vectors of the initial response text and the copyright content, respectively.

[0012] In some embodiments of this application, a cache library is established and the hash value corresponding to the content whose copyright status validity has been determined is stored in the cache library, so as to allow subsequent determination of the copyright status validity of multimodal input content based on the cache library.

[0013] In some embodiments of this application, if a risk query task is identified, the large language model generates an alternative query task based on the risk query task and outputs it to the user.

[0014] In some embodiments of this application, the compliance response includes at least one of a rejection response, a guiding response, and a limited information response.

[0015] In a second aspect of this application, a copyright protection device based on a large visual language model includes: The input receiving module is used to acquire multimodal input content and user query text; The copyright statement recognition module is used to recognize copyright statements in the multimodal input content, so as to generate copyright risk information when a copyright statement is recognized; The copyright status verification module is used to determine the validity of the copyright status of the multimodal input content through network search, and to generate copyright risk information when the copyright status of the multimodal input content is determined to be valid; The query risk analysis module is used to identify risk query tasks in the user query text, so as to generate query risk information when a risk query task is identified. The copyright status alert module is used to generate warning messages that include copyright risk information and query risk information; The response generation guidance module is used to input the warning information into the visual language big model and guide it to generate a compliance response.

[0016] The beneficial effects of this invention are as follows: This invention employs a dual mechanism of copyright statement recognition and copyright status validity assessment to achieve complete copyright information verification for multimodal input content, including both explicitly marked and unmarked copyright statements. This significantly improves the proportion of copyright infringement requests correctly rejected by the visual language big data model, while also avoiding erroneous rejections of legitimate queries that comply with fair use principles, thus achieving a good balance between copyright protection and user experience. Attached Figure Description

[0017] Figure 1 This is a flowchart of a copyright protection method based on a large visual language model in an embodiment of the present invention; Figure 2 This is a schematic diagram of a copyright protection device based on a large visual language model in an embodiment of the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0019] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0020] Example: This embodiment provides a copyright protection method based on a large visual language model, including the following steps: Step 101: Obtain the multimodal input content and user query text.

[0021] The multimodal input content can include images and text, specifically materials that may be protected by copyright, such as book excerpts, news articles, music lyrics, and code documents. The sources of the multimodal input content can include direct user uploads, online retrieval by a multimodal retrieval enhancement generation system, and acquisition of webpage content by a multimodal webpage proxy.

[0022] User queries typically express the user's desired action from the visual language model on the multimodal input content, such as "repeat this text," "extract text from the image," "rewrite this content in your own words," or "translate this English text into Chinese."

[0023] Step 102: Recognize the copyright statement of the multimodal input content.

[0024] First, this embodiment uses OCR technology to extract text from the image in the multimodal input content. For example, this text extraction can be implemented using the PaddleOCR open-source tool, which supports multilingual text recognition and can accurately extract text information (including copyright notices that may be embedded in the image) from the image.

[0025] The algorithm retrieves and identifies copyright notices from the original text in the multimodal input content and the text extracted using OCR technology, based on a predefined keyword list. This keyword list includes, but is not limited to, "Copyright," "©," "All Rights Reserved," "(c)," "Copyright owned," "Authority reserved," and "All rights reserved." The retrieval algorithm combines regular expression matching and string similarity calculation to adapt to different writing formats and potential OCR recognition errors.

[0026] Step 103: If a copyright statement is detected, generate copyright risk information.

[0027] Specifically, if a copyright notice is detected, the corresponding part of the multimodal input content that is protected by copyright will be marked. Accordingly, the copyright risk information may include the specific location and content of the corresponding part of the multimodal input content that is protected by copyright.

[0028] In form, define a copyright notice identification function:

[0029] Where I represents the image in the multimodal input content, T represents the text in the multimodal input content, and K represents the set of copyright keywords. This represents the text extracted from image I using OCR.

[0030] Step 104: If no copyright statement is identified, at least the copyright status of the multimodal input content shall be determined by searching the web.

[0031] Specifically, the original text in the multimodal input content and the text extracted through OCR technology are used to perform a web search using a search engine API to identify the source of the corresponding content of the two texts. In this embodiment, the API may specifically be the Google Search API Serper, which can return web search results that are relatively relevant to the corresponding content of the two texts. The web search results include the title, URL, and summary fragment of the relevant web pages.

[0032] Preferably, the API can perform searches on only the first 100 characters of the text or sentences containing key information, rather than full-text searches.

[0033] By analyzing the relevant information fragments in the above web search results, the source of the original text in the multimodal input content and the content in the text extracted by OCR technology is determined, and the validity of the copyright status of the multimodal input content is judged based on the source information.

[0034] The analysis of relevant information fragments in web search results can specifically involve extracting book titles, authors, publishers, news websites, music album names, code repository names, etc.

[0035] As a preferred approach, metadata related to the text content can be further extracted from the two texts mentioned above (the original text in the multimodal input content and the text extracted through OCR technology), such as publication year and copyright holder. Then, the similarity between the metadata and the relevant information fragments in the webpage search results can be calculated to confirm the credibility of the webpage search results.

[0036] For example, the latest copyright status of the multimodal input content can be obtained through a large language model with integrated search engine functionality, thereby enabling the determination of the validity of the copyright status of the multimodal input content. In this embodiment, the large language model with integrated search engine functionality is specifically the DeepSeek-R1-all model, which can obtain content type (such as books / news articles / music lyrics / code documents), content source, publication year, and author / creator information based on relevant information fragments in webpage search results. Simultaneously, it combines this with the copyright protection period (such as the lifetime of the author plus 70 years as stipulated in US copyright law, or 95 years after publication, etc.) to determine the validity of the copyright status of the multimodal input content and provide detailed evidence.

[0037] Step 105: If the copyright status of the multimodal input content is determined to be valid, then copyright risk information is generated.

[0038] When the DeepSeek-R1-all model determines that the multimodal input content is still under copyright protection, it marks the relevant parts of the multimodal input content and records the copyright source information, copyright holder information, and judgment basis during the judgment process. The copyright risk information is the information recorded above.

[0039] Formally, define a copyright status verification function:

[0040] in, Indicates a search engine query. This indicates the validity of the copyright status of the large model.

[0041] Based on the combined identification of copyright statements and the determination of the validity of copyright status, the final copyright risk function is defined as follows:

[0042] Step 106: Perform risk query task identification on the user query text. If a risk query task is identified, generate query risk information.

[0043] The risk query task can be a repetitive query task, an extraction query task, a definition query task, or a translation query task.

[0044] Among them, repetitive query tasks require directly copying the original text, such as "repeat this text", "output all the text in the image", and "completely restate this content"; extraction query tasks require extracting specific segments of the original text, such as "extract the first paragraph", "tell me the lyrics in the image", and "output lines 10-20 of the code"; paraphrasing query tasks require rewriting or explaining the original text, such as "say it in your own words", "rewrite this text", and "explain the function of this code"; and translation query tasks require translating the original text, such as "translate into Chinese" and "translate this English text into French". These types of queries usually contain keywords such as "translate" and specify the target language.

[0045] Other types of queries (such as summaries, comments, background inquiries, etc.) are classified as non-risk query tasks.

[0046] In form, the infringement risk assessment function is defined as follows:

[0047] Where Q represents the user query and C represents the multimodal input content. This indicates a large segment in the risk query task.

[0048] When a risky query task is identified, the large language model can generate an alternative query task based on the risky query task and output it to the user. Specifically, alternative query tasks can be generated through prompting engineering techniques. The alternative query task should maintain the user's core information needs and intent, while avoiding requests to directly copy, extract, rewrite, or translate protected content, and obtaining information in a legal manner.

[0049] For example, if the risk query task is described as "repeating this text", then the alternative query task can be described as "summarizing the main points of this text"; if the risk query task is described as "extracting lyrics from an image", then the alternative query task can be described as "telling me the name of this song, the artist, and the year of its release"; if the risk query task is described as "rewriting this code", then the alternative query task can be described as "explaining the function and design concept of this code"; if the risk query task is described as "translating into Chinese", then the alternative query task can be described as "telling me the theme and gist of this text".

[0050] For example, the large language model in this embodiment may specifically be GPT-4.

[0051] In some other embodiments, when a user obtains an alternative query task, they can replace the risk query task with the alternative query task.

[0052] Step 107: Generate a warning message containing copyright risk information and query risk information.

[0053] Copyright risk information may specifically include an explanation of the source of copyright risk and an explanation of infringement risk.

[0054] The source of copyright risk can be, for example, the copyright notice “© 2023 ABC Publishing” identified in the text extracted by OCR technology, or the fact that at least part of the multimodal input content found through online search comes from the book “XXX” (author: XXX, publication year: 2020), and the work is still under copyright protection (protection period until 2090).

[0055] The infringement risk statement specifically indicates the types of infringement that the current user's query may result in and provides compliance suggestions. For example, the infringement risk statement may indicate that the current query is a repetitive query task, which requires directly outputting copyrighted text content, constituting unauthorized copying and carrying a high risk of infringement, violating the copyright holder's exclusive right of reproduction as defined in 17 USCode § 106. It may also suggest that the user replace the query task with something like "summarize the main points and core ideas of the content," "inquire about the author, publication information, and background of the work," "comment on or analyze the writing style and impact of the content," or "purchase or authorize access to the content through official channels."

[0056] Step 108: Input the warning information into the visual language big model so that the visual language big model outputs a compliance response.

[0057] The warning information is input into the visual language big model in the form of context enhancement to ensure that the visual language big model fully considers copyright constraints when generating compliant responses.

[0058] In this embodiment, copyrighted content in the multimodal input is marked based on the copyright risk information. After the warning information is input into the visual language model, the visual language model first outputs an initial response text, then calculates the similarity between the copyrighted content in the multimodal input and the initial response text; the visual language model outputs a compliant response based on the similarity.

[0059] If the risk query task is a repetitive query task or an extraction query task, then the similarity is ROUGE-L, where , , , and These represent the initial response text and the copyright content in the multimodal input content, respectively. As a weighting factor, LCS ( , ) is the longest common subsequence between the initial response text and the copyright content in the multimodal input content, length( ) is the length of the initial response text, length( ROUGE-L represents the length of the copyrighted content in the multimodal input. If ROUGE-L is greater than 0.7, the initial response text is considered to have a high risk of copyright infringement.

[0060] If the risk query task is a descriptive query task, then the similarity is F. BERT ,in, , , , and These are the BERT embedding vectors of the initial response text and the copyright content, respectively. If F BERT If the value is greater than 0.85, the initial response text is considered to have a high risk of semantic infringement.

[0061] If the risk query task is a translation-type query task, then the similarity is ,in , and These are the XLM-R sentence embedding vectors of the initial response text and the copyright content, respectively. If... If the value is greater than 0.8, the initial response text is considered to have a high risk of infringement at the cross-language level.

[0062] When the initial response text poses a high risk of infringement, it can be intercepted, preventing its output to the user. Simultaneously, a detection log is recorded, containing the aforementioned similarity, infringement type, and infringing content. This also triggers a compliance response mechanism, injecting stronger copyright constraint prompts into the visual language model, causing it to regenerate the initial response text until the infringement risk of the initial response text meets the requirements. If multiple regenerated initial response texts all pose a high risk of infringement, the visual language model outputs a rejection response to the user.

[0063] Formally, define a risk function for the initial response text:

[0064] in, This represents the initial response text. This indicates copyright content within the multimodal input. The similarity calculation function is represented (select ROUGE-L or F according to the task type). BERT or ), This indicates the corresponding threshold.

[0065] Compliance responses include at least one of the following: rejection response, guiding response, and limited information response.

[0066] Among them, a rejection response explicitly refuses to process a high-risk query; a guiding response provides a legitimate alternative, such as suggesting that the user visit official channels to purchase or authorize access to the content; and a limited information response provides non-infringing information about the content, such as metadata information like author, publication date, and topic overview, but does not copy or rewrite the protected substantive content.

[0067] Since there is no strict data dependency between copyright statement identification, copyright status validity judgment, and risk query task identification, a parallel execution strategy can be adopted.

[0068] Specifically, upon receiving a user's query text, the following three processing flows can be initiated simultaneously: OCR technology is used to extract text and perform keyword retrieval from images in the multimodal input content. The original text in the multimodal input content and the text extracted using OCR technology are then used to perform a web search using a search engine API to determine the validity of copyright status. A risk query task is performed on the user's query text. When a clear copyright statement is identified, the copyright status validity determination can be terminated early, saving computational resources. The results of the three processes are summarized to generate an alert message. Through parallel processing, the average response latency is reduced from the sum of the serial execution times to the maximum single-module processing time, improving system response efficiency.

[0069] Preferably, this embodiment can also establish a cache library to cache and store content for which copyright status validity has been determined.

[0070] Specifically, each cached record contains the hash value of the copyrighted content (for fast indexing), the source of the copyrighted content (such as book title, author, publisher), copyright status (protected or public domain), copyright expiration date, verification timestamp, and cache validity period.

[0071] When a new user enters text as a query, its content hash value is first calculated, and the cache is checked to see if a matching cache record exists. If the cache is hit and has not expired, the historical verification result is reused directly; if the cache is missed or has expired, the complete copyright status validity judgment process is executed, and the result is written to the cache.

[0072] The cache validity period policy is as follows: the default cache validity period is 7 days, which is suitable for content with relatively stable copyright status; for content that is close to the expiration date (less than 1 year away from the expiration date), the cache validity period is shortened to 1 day; when a change in copyright status is detected (such as copyright transfer, early entry into the public domain, etc.), the relevant cache is immediately invalidated.

[0073] The system uses an LRU (Least Recently Used) policy to manage cache capacity. When the cache capacity reaches its limit, the least frequently accessed cache records are evicted first. The system also supports a cache preheating mechanism, which can preload copyright status information for frequently accessed content.

[0074] Formally, define the cache lookup function:

[0075] Where C represents the user query text, The hash function representing the user's query text. This indicates a cache validity check. This function represents a function to determine the validity of copyright status. This indicates cached content.

[0076] This embodiment has been comprehensively experimentally verified on several mainstream visual language models, including closed-source models (GPT-4o, GPT-4o-mini, Gemini-2.0, Claude-3.7), open-source models (LLaVA series, Qwen2.5-VL series, DeepSeek-VL, GLM-4v), and unified architecture models (Janus-Pro).

[0077] It should be noted that this embodiment adopts a plug-in tool enhancement architecture, which does not require fine-tuning or retraining of any large visual language model. Therefore, it does not involve the use of copyrighted content as training data, fundamentally avoiding compliance risks caused by training data.

[0078] The external tools used in this embodiment all have clear sources of compliance: (1) PaddleOCR, an open-source optical character recognition tool, is licensed under the Apache 2.0 open-source license, allowing both commercial and non-commercial use; (2) Google Search APISerper is a commercially licensed public search engine interface, and legal access was obtained through official channels; (3) Large language models such as DeepSeek-R1-all are called through official API interfaces, and are all compliant commercial or open-source services. The test dataset used in the experiment was constructed for academic research purposes and is only used to evaluate the model's performance. It does not involve updating model parameters and complies with the relevant provisions of copyright law regarding fair use.

[0079] The experimental dataset contains 50,000 multimodal query-content pairs, covering four types of copyrighted material (book excerpts, news articles, music lyrics, code documents) and four infringement scenarios (repetition, extraction, interpretation, translation). The key to the experimental design is to distinguish between the compliance of the data source and the infringement risk of the query type: the content samples in the experiment are themselves copyrighted materials, but they are obtained through the following compliant channels for testing: (1) publicly available datasets and test materials with explicit authorization from the copyright holder; (2) short text excerpts that comply with the fair use principle, with the length of the excerpt strictly controlled within the scope of fair use (such as not exceeding 10% of the original text or within 200 words), used only to construct test samples; (3) example content publicly released by the copyright holder, such as fragments publicly shared by the author on personal websites or social media; (4) historical documents and works whose copyright protection period has expired (used for comparative testing). Although the data acquisition method is compliant, the user query types in the experiment (such as "completely repeat this text", "extract all content", "complete translation", etc.) themselves require operations that may constitute infringement. The experiment tested the system's ability to identify and correctly reject infringement risks associated with these query types, even if the data itself was obtained through compliant means. All experimental data usage complied with relevant copyright laws, ensuring the legality and compliance of the experimental process itself.

[0080] The evaluation metric is the rejection rate: the proportion of copyright infringement requests that the model correctly rejects. The higher the value, the better the copyright protection effect.

[0081] Table 1 shows the changes in rejection rates of each model in a copyright-free scenario before and after applying the method of this embodiment, where the unit of rejection rate is .

[0082] Table 1

[0083] As can be seen from Table 1, this embodiment achieves a significant improvement in rejection rate across all models and all task types: Repeated tasks: The rejection rate increased from an average of 18.78% to 94.42%, an improvement of 75.64 percentage points. For models with previously low rejection rates (such as Gemini-2.0 at 0.09%), the improvement was close to 100 percentage points.

[0084] Retrieval Task: The rejection rate increased from an average of 2.60% to 49.62%, an increase of 47.02 percentage points. The increase in the rejection rate for this task was relatively small because the boundaries of retrieval tasks are somewhat blurred, and some retrieval behaviors may fall within the scope of fair use.

[0085] Task Explanation: The rejection rate increased from an average of 2.23% to 43.89%, an increase of 41.66 percentage points. This significant improvement demonstrates that this embodiment can effectively identify and prevent copyright infringement in derivative works.

[0086] Translation task rejection rate increased from an average of 13.86% to 68.59%, an increase of 54.73 percentage points. This significant increase in the rejection rate for translation tasks indicates that this implementation method provides effective protection against cross-language copyright infringement.

[0087] This embodiment demonstrates good performance on both closed-source and open-source models, proving the cross-model versatility of its plug-in architecture.

[0088] This invention was tested on a non-copyright content dataset to verify whether there was an excessive rejection problem. The test dataset included: (1) public domain content: a total of 80 samples, including classic literary works (such as the Book of Songs, Shakespeare's works, etc.) and historical documents whose copyright protection period has expired; (2) fair use queries: a total of 80 samples, including academic reviews, educational citations, factual citations in news reports, and other query scenarios that conform to the fair use principle. During the test, the system identified the copyright status and assessed the query risk for each sample, and recorded whether the system incorrectly identified non-copyright content as copyright content and whether it incorrectly rejected legitimate queries that conform to the fair use principle. The experimental results showed that the false alarm rate of this invention was 0%, that is, no erroneous rejections were generated. Specifically, for public domain content, the system correctly identified that it had entered the public domain through the copyright status verification module and did not mark it as copyright-protected content; for queries that conform to the fair use principle, the query risk analysis module correctly identified its legitimate nature and did not generate a rejection response. This proves that this invention can accurately distinguish: (1) copyright-protected content and public domain content; (2) infringement and fair use. This embodiment effectively avoids the problem of excessive rejection, protecting copyright without compromising the normal practical function of the model in legal scenarios, and achieving a good balance between copyright protection and user experience.

[0089] This embodiment further provides a copyright protection device based on a large visual language model, including: The input receiving module 201 is used to acquire multimodal input content and user query text; The copyright statement recognition module 202 is used to recognize the copyright statement of the multimodal input content, so as to generate copyright risk information when a copyright statement is recognized; The copyright status verification module 203 is used to determine the validity of the copyright status of the multimodal input content through network search, and generate copyright risk information when the copyright status of the multimodal input content is determined to be valid; The query risk analysis module 204 is used to identify risk query tasks in the user query text, so as to generate query risk information when a risk query task is identified. The copyright status reminder module 205 is used to generate warning messages containing copyright risk information and query risk information; The response generation guidance module 206 is used to input the warning information into the visual language big model and guide it to generate a compliance response.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A copyright protection method based on a large visual language model, characterized in that, include: Obtain multimodal input content and user query text; The multimodal input content is subjected to copyright statement recognition. If a copyright statement is recognized, copyright risk information is generated. At least once, the copyright status of the multimodal input content is determined by a web search. If the copyright status of the multimodal input content is determined to be valid, copyright risk information is generated. The user query text is subjected to risk query task identification. If a risk query task is identified, query risk information is generated. Generate warning messages containing copyright risk information and query risk information; The warning information is input into the visual language model so that the visual language model outputs a compliance response.

2. The copyright protection method based on a large visual language model according to claim 1, characterized in that, The copyright status of the multimodal input content is determined by a large language model that integrates search engine functionality.

3. The copyright protection method based on a large visual language model according to claim 1, characterized in that, The copyrighted content in the multimodal input is marked based on the copyright risk information; After the warning information is input into the visual language model, the visual language model first outputs an initial response text, and then calculates the similarity between the copyright content in the multimodal input content and the initial response text. The visual language big model outputs a compliance response based on the similarity.

4. The copyright protection method based on a large visual language model according to claim 3, characterized in that, The risk query task can be a repetitive query task, an extraction query task, a definition query task, or a translation query task.

5. The copyright protection method based on a large visual language model according to claim 4, characterized in that, If the risk query task is a repetitive query task or an extraction query task, then the similarity is ROUGE-L, where , , , and These represent the initial response text and the copyright content in the multimodal input content, respectively. As a weighting factor, LCS ( , ) is the longest common subsequence between the initial response text and the copyright content in the multimodal input content, length( ) is the length of the initial response text, length( ) represents the length of the copyright content in the multimodal input content.

6. The copyright protection method based on a large visual language model according to claim 4, characterized in that, If the risk query task is a descriptive query task, then the similarity is F. BERT ,in, , , , and These are the BERT embedding vectors of the initial response text and the copyright content, respectively.

7. The copyright protection method based on a large visual language model according to claim 4, characterized in that, If the risk query task is a translation-type query task, then the similarity is ,in , and These are the XLM-R sentence embedding vectors of the initial response text and the copyright content, respectively.

8. The copyright protection method based on a large visual language model according to claim 3, characterized in that, A cache library is established, and the hash values ​​corresponding to the content whose copyright status validity has been determined are stored in the cache library, so that subsequent copyright status validity determinations can be performed on multimodal input content based on the cache library.

9. The copyright protection method based on a large visual language model according to claim 1, characterized in that, If a risk query task is identified, the large language model generates an alternative query task based on the risk query task and outputs it to the user.

10. A copyright protection device based on a large visual language model, characterized in that, include: The input receiving module is used to acquire multimodal input content and user query text; The copyright statement recognition module is used to recognize copyright statements in the multimodal input content, so as to generate copyright risk information when a copyright statement is recognized; The copyright status verification module is used to determine the validity of the copyright status of the multimodal input content through network search, and to generate copyright risk information when the copyright status of the multimodal input content is determined to be valid; The query risk analysis module is used to identify risk query tasks in the user query text, so as to generate query risk information when a risk query task is identified. The copyright status alert module is used to generate warning messages that include copyright risk information and query risk information; The response generation guidance module is used to input the warning information into the visual language big model and guide it to generate a compliance response.

Images