Image processing method and device, computer equipment and storage medium

Abstract

The invention relates to an image processing method and device, computer equipment and a storage medium. The method comprises the steps of determining a sample security description text and a sample risk description text of a sample image through a visual language large model according to the sample image and a sample text instruction; optimizing the sample text instruction according to a sample security description text and a sample risk description text through a text processing model to generate a security text instruction; training a cue word optimizer to be trained according to the safe text instruction and the sample text instruction, and determining a target cue word optimizer; inputting the user instruction into a target cue word optimizer, and determining a target text instruction; and determining a target description text of the target image according to the target text instruction and the target image through the visual language large model. According to the scheme, the image processing cost when the visual language large model processes the image can be reduced, and the image processing accuracy is improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to an image processing method, apparatus, computer device, and storage medium. Background Technology

[0002] Visual language large-scale models (VLAs) demonstrate powerful capabilities in image processing scenarios such as image understanding, question answering, and multimodal interaction by integrating text and visual data. Existing VLA security alignment schemes mainly include internal adjustment strategies and external protection strategies. Internal adjustment strategies, represented by human feedback reinforcement learning, achieve security goals by directly adjusting model parameters. This method relies on large-scale manually labeled data and complex computational resources. For example, during training, the model learns and optimizes its output through labeled safe and unsafe responses, but biases in manual labeling can lead to insufficient consistency and accuracy. External protection strategies guide the model to generate safe responses without modifying model parameters through predefined rules or heuristics. For example, rule-based filters detect harmful content and block output, but such methods are usually designed for specific scenarios and struggle to address the complex risks in text and image interactions. Therefore, reducing the image processing cost of VLAs when processing images, improving the accuracy of image processing, and reducing the risks of image description text generated by VLAs are problems that need to be solved. Summary of the Invention

[0003] Therefore, it is necessary to address the aforementioned technical problems by providing an image processing method, apparatus, computer device, and storage medium that can reduce the image processing cost when the visual language large model processes images, improve the accuracy of image processing, and reduce the risk of image description text generated by the visual language large model.

[0004] In a first aspect, this application provides an image processing method, the method comprising:

[0005] Using a large visual language model, the sample security description text and sample risk description text of the sample image are determined based on the sample image and sample text instructions;

[0006] Using a text processing model, the sample text instructions are optimized based on the sample security description text and the sample risk description text to generate security text instructions;

[0007] The target prompt word optimizer is determined by training the prompt word optimizer according to the security text instructions and the sample text instructions;

[0008] Input the user command into the target prompt word optimizer to determine the target text command;

[0009] Using the visual language model, the target descriptive text of the target image is determined based on the target text instruction and the target image.

[0010] In one embodiment, a text processing model is used to optimize the sample text instruction based on the sample security description text and the sample risk description text to generate a security text instruction, including:

[0011] Using a large visual language model, a risk test report is generated based on the sample security description text, the sample image, and the sample risk description text.

[0012] Using a text processing model, the sample text instructions are optimized based on the risk test report to generate secure text instructions.

[0013] In one embodiment, a risk test report is generated using a visual language big data model based on the sample security description text, the sample image, and the sample risk description text, including:

[0014] Using a large visual language model, risk characteristics are determined based on the sample security description text and the sample risk description text;

[0015] A risk test report is generated based on the risk characteristics, the sample image, and the sample text instructions.

[0016] In one embodiment, a risk test report is generated based on the risk characteristics, the sample image, and the sample text instructions, including:

[0017] A large visual language model is used to perform multimodal causal analysis on the risk features, sample images, and sample text instructions to determine the causal graph;

[0018] Based on the causal graph, risk image features in the sample images that are associated with the risk features are determined, and risk text instruction features in the sample text instructions that are associated with the risk features are determined.

[0019] A risk test report is generated based on the risk image features and the risk text instruction features.

[0020] In one embodiment, risk characteristics are determined based on the sample security description text and the sample risk description text using a large visual language model, including:

[0021] The textual differences between the sample security description text and the sample risk description text are determined using a large visual language model.

[0022] Risk characteristics are determined based on the textual differences.

[0023] In one embodiment, the training method for the text processing model includes:

[0024] Obtain historical datasets; the historical datasets include historical test reports, historical risk text instructions, and historical safety text instructions determined after adjusting the historical risk text instructions;

[0025] The historical dataset is divided into a model training dataset and a model testing dataset;

[0026] The neural network model is trained using the model training dataset to determine candidate models;

[0027] The candidate model is tested using the model test dataset to determine the model loss function of the candidate model. If the model loss function is less than a preset loss function threshold, the candidate model is determined to be a text processing model.

[0028] In one embodiment, the target descriptive text of the target image is determined based on the target text instruction and the target image using the visual language big model, including:

[0029] The target image is encoded using the visual encoder in the large visual language model to determine its image features;

[0030] The image features of the target image are format-converted by the projector in the large visual language model to determine the image tag of the target image;

[0031] Using the natural language processing model in the visual language big model, the target description text of the target image is determined based on the image tag of the target image and the text tag of the target text instruction.

[0032] Secondly, this application also provides an image processing apparatus, the apparatus comprising:

[0033] The sample text generation module is used to determine the sample security description text and sample risk description text of the sample image based on the sample image and sample text instructions using a large visual language model.

[0034] The security text instruction generation module is used to optimize the sample text instruction based on the sample security description text and the sample risk description text using a text processing model, and generate a security text instruction.

[0035] The prompt word optimizer training module is used to train the prompt word optimizer to be trained based on the security text instructions and the sample text instructions, and to determine the target prompt word optimizer;

[0036] The target text instruction determination module is used to input user instructions into the target prompt word optimizer and determine the target text instruction;

[0037] The target description text determination module is used to determine the target description text of the target image based on the target text instruction and the target image, using the visual language big model.

[0038] Thirdly, this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0039] Using a large visual language model, the sample security description text and sample risk description text of the sample image are determined based on the sample image and sample text instructions;

[0040] Using a text processing model, the sample text instructions are optimized based on the sample security description text and the sample risk description text to generate security text instructions;

[0041] The target prompt word optimizer is determined by training the prompt word optimizer according to the security text instructions and the sample text instructions;

[0042] Input the user command into the target prompt word optimizer to determine the target text command;

[0043] Using the visual language model, the target descriptive text of the target image is determined based on the target text instruction and the target image.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0045] Using a large visual language model, the sample security description text and sample risk description text of the sample image are determined based on the sample image and sample text instructions;

[0046] Using a text processing model, the sample text instructions are optimized based on the sample security description text and the sample risk description text to generate security text instructions;

[0047] The target prompt word optimizer is determined by training the prompt word optimizer according to the security text instructions and the sample text instructions;

[0048] Input the user command into the target prompt word optimizer to determine the target text command;

[0049] Using the visual language model, the target descriptive text of the target image is determined based on the target text instruction and the target image.

[0050] The aforementioned image processing method, apparatus, computer equipment, and storage medium, through a large visual language model, determine the sample security description text and sample risk description text of a sample image based on the sample image and sample text instructions; through a text processing model, optimize the sample text instructions based on the sample security description text and sample risk description text to generate secure text instructions; train a prompt word optimizer based on the secure text instructions and sample text instructions to determine a target prompt word optimizer; input user instructions into the target prompt word optimizer to determine the target text instructions; and through the large visual language model, determine the target description text of the target image based on the target text instructions and the target image. This solves the problems of high computational cost, difficulty in quickly adapting to new security threats, insufficient generalization ability, and poor correction accuracy when using existing internal adjustment strategies or external defense strategies to correct risk elements in image interpretation information during image and text processing using a large visual language model. The above scheme optimizes sample text instructions based on the sample safety description text and sample risk description text output by the visual language big data model, generating safe text instructions. Then, it trains a prompt word optimizer using the sample text instructions and safe text instructions to determine the target prompt word optimizer. This target prompt word optimizer can optimize user instructions and determine the target text instructions. The visual language big data model determines the target description text that does not contain sensitive information based on the target text instructions and the target image. This reduces the image processing cost when the visual language big data model processes images, improves the accuracy of image processing, and reduces the risk of the image description text generated by the visual language big data model. Attached Figure Description

[0051] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0053] Figure 3 This is an example diagram illustrating the generation of secure text instructions in one embodiment;

[0054] Figure 4 This is a structural block diagram of an image processing device in one embodiment;

[0055] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104, using a visual language model, determines the sample security description text and sample risk description text of the sample image based on the sample image and sample text instructions; using a text processing model, it optimizes the sample text instructions based on the sample security description text and the sample risk description text to generate secure text instructions; it trains the prompt word optimizer based on the secure text instructions and the sample text instructions to determine the target prompt word optimizer; it inputs user instructions into the target prompt word optimizer to determine the target text instructions; using the visual language model, it determines the target description text of the target image based on the target text instructions and the target image, and sends the target description text of the target image to terminal 102 via a communication network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0058] In one embodiment, such as Figure 2 As shown, an image processing method is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0059] S210. Using a large visual language model, determine the sample security description text and sample risk description text of the sample image based on the sample image and sample text instructions.

[0060] Here, sample text instructions refer to questions posed by users based on sample images, specifying the image information they wish to obtain. The sample images and sample text instructions are input into the visual language model, which then generates descriptive text for multiple sample images based on these instructions and images. Sample security descriptive text refers to the image description text for sample images that does not contain sensitive words; sample risk descriptive text refers to the image description text for sample images that do contain sensitive words.

[0061] It should be noted that the visual language big model is a multimodal big model, which mainly includes a visual encoder, a projector, and a natural language processing model.

[0062] The visual encoder typically employs a CLIP (Contrastive Language–Image Pre-training) model based on the Transformer architecture, trained with millions of image-text pairs to acquire the ability to associate images with text. The projector is used to convert the output of the visual encoder into a form understandable by the natural language processing (NLP) model, usually presented as image tags. A NLP model is an artificial intelligence model specifically designed to understand, generate, translate, summarize, analyze, or manipulate human natural language. It learns the statistical patterns, semantic relationships, and grammatical structures of language data to transform human language into a form that computers can process and compute, outputting useful results.

[0063] For example, the sample image is an image containing birds, and the sample text instruction could be: "Does the image contain any vertebrates? Please explain why."

[0064] Specifically, sample images and sample text commands are input into a large-scale visual language model. The model parses the sample images and commands, outputting image explanations for each. These explanations can be manually labeled to use risk descriptions for images containing sensitive words and safety descriptions for images without sensitive words. The sample images, text commands, risk descriptions, and safety descriptions form the sample dataset.

[0065] For example, a dataset consisting of sample images and sample text instructions can be determined from three multimodal datasets: the VLGuard dataset, the SPA-VL dataset, and the VLSbench dataset. If each set of data in the dataset of sample images and sample text instructions has a reference response, the reference response can be used as the sample safety description text. If no reference response exists, the sample safety description text can be generated from the dataset using the visual language large model GPT4o. The sample risk description text can be generated from the dataset using the visual language large model LLaVA-v1.5-7B. The dataset of sample images and sample text instructions can contain 10,000 sample image-sample text instruction sets, of which 977 sets focus on practicality and 8,904 sets focus on safety.

[0066] S220. Using a text processing model, optimize the sample text instructions based on the sample security description text and the sample risk description text to generate security text instructions.

[0067] The text processing model is a pre-trained neural network model.

[0068] Specifically, the text processing model determines the differences between the sample security description text and the sample risk description text, locates the differences in the sample risk description text, replaces the differences in the sample risk description text, and thus generates the security text instruction corresponding to the sample risk description text.

[0069] S230. Train the prompt word optimizer to be trained according to the security text instructions and sample text instructions, and determine the target prompt word optimizer.

[0070] The prompt word optimizer is a tool used to improve the output quality of AI (Artificial Intelligence) models by optimizing prompt words to improve the accuracy of AI generation. For example, the prompt word optimizer to be trained could be the Qwen2-VL-7B-Instruct model, which is a multimodal large language model.

[0071] Specifically, the trainable prompt word optimizer is trained based on secure text instructions and sample text instructions. During the training process, the parameters of the trainable prompt word optimizer are adjusted using the LLaMA Factory tool and LoRA (Low-Rank Adaptation) technology. The trained trainable prompt word optimizer is then used as the target prompt word optimizer. LLaMA Factory is an open-source tool platform focused on fine-tuning, training, and deploying large language models, aiming to simplify the development process. LoRA is a pre-trained model fine-tuning method.

[0072] Understandably, the target prompt word optimizer can optimize user input commands to avoid sensitive words appearing in the target description text due to certain information in the user commands.

[0073] S240. Input the user instruction into the target prompt word optimizer to determine the target text instruction.

[0074] Specifically, the user command is input into the target prompt word optimizer. The target prompt word optimizer adjusts the command elements in the user command that may cause sensitive words to appear in the output text of the visual language large model, determines the target safe words that can replace the sensitive words in the user command, replaces the sensitive words in the user command with the target safe words, and uses the replaced user command as the target text command.

[0075] S250. Using a large visual language model, determine the target description text of the target image based on the target text instruction and the target image.

[0076] The target image refers to the image of the large model of visual language input by the user.

[0077] For example, a method for determining the target description text of a target image may include:

[0078] The target image is encoded by the visual encoder in the visual language model to determine its image features; the image features are then converted into a format by the projector in the visual language model to determine the image tag; and finally, the target description text of the target image is determined by the natural language processing model in the visual language model based on the image tag and the text tag of the target text instruction.

[0079] Specifically, the target image is encoded by a visual encoder within the visual language model, decomposing it into semantic feature vectors, i.e., image features. A projector then maps these image features extracted by the visual encoder to a dimension consistent with the text embedding space using linear transformation or a multilayer perceptron, determining the image token to ensure the target image information can be understood by the visual language model. Regular expressions are then used to match the tokens in the target text instruction level by level to determine the text token. Finally, the natural language processing model within the visual language model determines the target descriptive text of the target image based on its image features and the text token.

[0080] In the aforementioned image processing method, a large visual language model is used to determine the sample security description text and sample risk description text of the sample image based on the sample image and sample text instructions. A text processing model is then used to optimize the sample text instructions based on the sample security description text and sample risk description text to generate secure text instructions. A target prompt word optimizer is trained based on the secure text instructions and the sample text instructions to determine the target prompt word optimizer. User instructions are input into the target prompt word optimizer to determine the target text instructions. Finally, the large visual language model is used to determine the target description text of the target image based on the target text instructions and the target image. This method solves the problems of high computational cost, difficulty in quickly adapting to new security threats, insufficient generalization ability, and poor correction accuracy when using existing internal adjustment strategies or external defense strategies to correct risk elements in the target description text during image and text processing using a large visual language model. The above scheme optimizes sample text instructions based on the sample safety description text and sample risk description text output by the visual language big data model, generating safe text instructions. Then, it trains a prompt word optimizer using the sample text instructions and safe text instructions to determine the target prompt word optimizer. This target prompt word optimizer can optimize user instructions and determine the target text instructions. The visual language big data model determines the target description text that does not contain sensitive information based on the target text instructions and the target image. This reduces the image processing cost when the visual language big data model processes images, improves the accuracy of image processing, and reduces the risk of the image description text generated by the visual language big data model.

[0081] In one embodiment, such as Figure 3 As shown, the sample text instructions are optimized based on the sample security description text and sample risk description text using a text processing model to generate security text instructions, including:

[0082] S310. Using a large visual language model, generate a risk test report based on the sample security description text, sample image, and the sample risk description text.

[0083] The risk test report records sensitive words in the sample risk description text, as well as image elements of the sample images corresponding to the sensitive words in the sample risk description text.

[0084] For example, a risk assessment report is generated as follows:

[0085] Using a large visual language model, risk characteristics are determined based on sample safety description text and sample risk description text; a risk test report is generated based on the risk characteristics, sample images, and sample text instructions.

[0086] Understandably, by using a large visual language model to determine the risk test report based on the sample security description text and the sample risk description text, the efficiency of risk test report generation can be improved.

[0087] For example, risk characteristics can be determined in the following ways:

[0088] Using a large visual language model, we determine the textual differences between the sample safety description text and the sample risk description text; and we determine the risk characteristics based on the textual differences.

[0089] Textual discrepancy information refers to information that is inconsistent between the sample risk description text and the sample safety description text.

[0090] Specifically, a large visual language model is used to determine the textual differences between the sample safety description text and the sample risk description text. Then, features are extracted from these textual differences to determine risk features. For example, risk features could be the lexical units of the textual differences.

[0091] The above scheme determines risk features based on the textual differences between the sample safety description text and the sample risk description text, which can improve the efficiency of risk feature extraction.

[0092] For example, methods for generating risk test reports based on risk characteristics, sample images, and sample text instructions include:

[0093] A visual language big data model is used to perform multimodal causal analysis on risk features, sample images, and sample text instructions to determine a causal graph. Based on the causal graph, risk image features that are associated with risk features in the sample images and risk text instruction features that are associated with risk features in the sample text instructions are determined. A risk test report is generated based on the risk image features and risk text instruction features.

[0094] Multimodal causal analysis refers to the process by which machines integrate information from multiple sensory modalities, combining logical analysis and semantic understanding to deduce implicit conclusions. Sensory modal information can include text, images, audio, and video. A causal graph is a graphical model used to represent causal relationships between different modalities, visually displaying the dependencies between variables through nodes and directed edges.

[0095] The above scheme uses a large visual language model to perform multimodal causal reasoning on risk features, sample images, and sample text instructions to determine the causal graph. Based on the causal relationships between nodes recorded in the causal graph, it can accurately locate instruction elements in the sample text instructions that may lead to the appearance of sensitive words in the image description text of the sample image.

[0096] S320. Using a text processing model, optimize the sample text instructions based on the risk test report to generate secure text instructions.

[0097] Input the risk test report and sample text instructions into the text processing model, and the text processing model can output secure text instructions.

[0098] For example, training methods for text processing models include:

[0099] Obtain historical datasets; historical datasets include historical test reports, historical risk text instructions, and historical safety text instructions determined after adjustments to the historical risk text instructions; divide the historical datasets into model training datasets and model testing datasets; use the model training dataset to train the neural network model and determine candidate models; use the model testing dataset to test the candidate models and determine the model loss function of the candidate models. If the model loss function is less than a preset loss function threshold, the candidate model is determined to be a text processing model.

[0100] Among them, the historical test report refers to the risk test report determined based on the historical risk text instructions and the historical images corresponding to the historical risk text instructions.

[0101] The above scheme provides a training method for a text processing model. By optimizing sample text instructions using the trained text processing model, the optimization efficiency and reliability of the optimization results can be improved.

[0102] For example, based on the above embodiments, the image processing method includes:

[0103] The sample image and sample text command are input into the visual language model. The visual language model parses the sample image and sample text command and outputs the target description text of the sample image corresponding to the sample image and sample text command. Through manual annotation, the target description text of the sample image containing sensitive words is used as the sample risk description text, and the target description text of the sample image without sensitive words is used as the sample safety description text.

[0104] Obtain historical datasets; historical datasets include historical test reports, historical risk text instructions, and historical safety text instructions determined after adjustments to the historical risk text instructions; divide the historical datasets into model training datasets and model testing datasets; use the model training dataset to train the neural network model and determine candidate models; use the model testing dataset to test the candidate models and determine the model loss function of the candidate models. If the model loss function is less than a preset loss function threshold, the candidate model is determined to be a text processing model.

[0105] Using a large-scale visual language model, the textual differences between the sample safety description text and the sample risk description text are determined. Feature extraction is then performed on these textual differences to identify risk features. A multimodal causal analysis is conducted on the risk features, sample images, and sample text instructions using the large-scale visual language model to determine a causal graph. Based on the causal graph, risk image features correlated with risk features in the sample images and risk text instruction features correlated with risk features in the sample text instructions are identified. A risk test report is generated based on the risk image features and risk text instruction features. Using a text processing model, the sample text instructions are optimized based on the risk test report to generate safety text instructions.

[0106] The trainable prompt word optimizer is trained based on the security text instructions and sample text instructions. During the training process, the parameters of the trainable prompt word optimizer are adjusted using the LLaMA Factory tool and LoRA technology. The trained prompt word optimizer is then used as the target prompt word optimizer.

[0107] The user command is input into the target prompt word optimizer. The optimizer adjusts command elements that might lead to sensitive words appearing in the output text of the visual language model, thus determining the target text command. The target image is encoded by the visual encoder within the visual language model, decomposing it into semantic feature vectors, i.e., image features. The projector maps these image features extracted by the visual encoder to a dimension consistent with the text embedding space using linear transformation or a multilayer perceptron, determining the image token, ensuring that the target image information can be understood by the visual language model. Regular expressions are used to match the lexical units in the target text command level by level to determine the text token of the target text command. Finally, the natural language processing model within the visual language model determines the target descriptive text of the target image based on the image features of the target image and the text token of the target text command.

[0108] For example, based on the above embodiments, the expression for the risk characteristics is as shown in formula (1):

[0109] (1)

[0110] Where F represents the risk characteristic. It is a function that performs risk characteristic analysis, x T Indicates a text instruction, x I Indicates image instructions, This indicates the sample security description text. This refers to the text describing the risk of the sample.

[0111] The methods for generating risk test reports based on risk characteristics include: using a large visual language model, and based on risk characteristics, conducting causal analysis from three dimensions—textual reasoning, visual reasoning, and cross-modal reasoning—to generate risk test reports. The expression for a risk test report is shown in formula (2):

[0112] (2)

[0113] Here, A refers to the risk test report.

[0114] When generating secure text instructions, the sample text instructions can be optimized based on the risk test report to generate optimized prompts. Based on the optimized prompts and the sample text instructions, secure text instructions are then generated. The expression for the optimized prompts is shown in formula (3):

[0115] (3)

[0116] in, The optimized suggestion words, This represents a function that performs optimization of prompt words.

[0117] The trainable prompt word optimizer is trained based on the safety text instructions and sample text instructions. When determining the target prompt word optimizer, the optimizer training dataset can be constructed based on the safety text instructions and sample text instructions. Using the large model management tool LLaMA Factory, fine-tuning operations are performed during the training of the prompt word optimizer. Low-rank adaptive technology is used to perform supervised fine-tuning operations on all layers of the multimodal large model Qwen2-VL-7B-Instruct during the training of the prompt word optimizer. During the training of the prompt word optimizer, the device batch of the prompt word optimizer to be trained is set to 1, the gradient accumulation step is 8, the learning rate is 5.0e-6, and the training rounds are 3. The learning rate is adjusted according to the cosine annealing scheduling strategy, the warm-up ratio is set to 0.1, and BF16 precision is used to improve computational efficiency. The entire training process follows the supervised sequence-to-sequence method, and the model is optimized by minimizing the cross-entropy loss. The expression of the loss function of a single sample is shown in formula (4):

[0118] (4)

[0119] Where T is the length of the cue word sequence, These are genuine optimization prompts. The t-th word element in It is the t-th word element in the prediction prompt. This indicates the predicted word before position t.

[0120] The target cue word optimizer is trained and determined to safely guide the rewriting of input cue words. These rewrites are then fed into the downstream visual language model along with the original image instructions, driving the visual language model to output safe and useful responses.

[0121] The above scheme proposes a proactive security calibration framework for visual language models based on a multimodal reasoning-driven prompt optimization strategy. Unlike internal adjustment methods that directly modify visual language model parameters during training to ensure security but incur high computational costs from retraining large-scale models, and external protection strategies that only provide protection after prompts are processed by VLMs (Visual Language Models), this framework avoids high computational costs. It proactively rewrites prompts before they enter VLM processing. Leveraging a multimodal reasoning-driven prompt optimization strategy, it utilizes multimodal reasoning capabilities to deeply analyze the interaction between text and images, systematically interpreting potential user intent step by step. This proactively reduces risks during the inference input stage, protecting VLMs from various security threats. The reasoning process enables visual language models to analyze text-image fusion scenarios, discovering and predicting potential risks in complex text-image interactions. Through multi-step analysis processes and a coherent thought chain construction mechanism, the model can decompose user-input queries, accurately interpret complex image scenarios, and deeply analyze the interaction relationship between images and text from a logical perspective. This reveals subtle threats hidden in seemingly secure inputs, generates responses that align with security values, and effectively improves the security and reliability of visual language models in multimodal interaction scenarios.

[0122] The above implementation method, through a large visual language model, determines the sample security description text and sample risk description text of a sample image based on sample images and sample text instructions; through a text processing model, optimizes the sample text instructions based on the sample security description text and sample risk description text to generate secure text instructions; trains a prompt word optimizer based on the secure text instructions and sample text instructions to determine a target prompt word optimizer; inputs user instructions into the target prompt word optimizer to determine the target text instructions; and through the large visual language model, determines the target description text of the target image based on the target text instructions and the target image. This solves the problems of high computational cost, difficulty in quickly adapting to new security threats, insufficient generalization ability, and poor correction accuracy when using existing internal adjustment strategies or external defense strategies to correct risk elements in the target description text during image and text processing using a large visual language model. The above scheme optimizes sample text instructions based on the sample safety description text and sample risk description text output by the visual language big data model, generating safe text instructions. Then, it trains a prompt word optimizer using the sample text instructions and safe text instructions to determine the target prompt word optimizer. This target prompt word optimizer can optimize user instructions and determine the target text instructions. The visual language big data model determines the target description text that does not contain sensitive information based on the target text instructions and the target image. This reduces the image processing cost when the visual language big data model processes images, improves the accuracy of image processing, and reduces the risk of the image description text generated by the visual language big data model.

[0123] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0124] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0125] In one embodiment, such as Figure 4 As shown, an image processing apparatus is provided, including: a sample text generation module 401, a security text instruction generation module 402, a prompt word optimizer training module 403, a target text instruction determination module 404, and a target description text determination module 405, wherein:

[0126] The sample text generation module 401 is used to determine the sample security description text and sample risk description text of the sample image based on the sample image and sample text instructions using a large visual language model.

[0127] The security text instruction generation module 402 is used to optimize the sample text instruction based on the sample security description text and the sample risk description text using a text processing model to generate a security text instruction.

[0128] The prompt word optimizer training module 403 is used to train the prompt word optimizer to be trained according to the security text instruction and the sample text instruction, and to determine the target prompt word optimizer;

[0129] The target text instruction determination module 404 is used to input user instructions into the target prompt word optimizer and determine the target text instruction;

[0130] The target description text determination module 405 is used to determine the target description text of the target image based on the target text instruction and the target image through the visual language big model.

[0131] For example, the secure text instruction generation module 402 is specifically used for:

[0132] Using a large visual language model, a risk test report is generated based on the sample security description text, the sample image, and the sample risk description text.

[0133] Using a text processing model, the sample text instructions are optimized based on the risk test report to generate secure text instructions.

[0134] Furthermore, the secure text instruction generation module 402 is also specifically used for:

[0135] Using a large visual language model, risk characteristics are determined based on the sample security description text and the sample risk description text;

[0136] A risk test report is generated based on the risk characteristics, the sample image, and the sample text instructions.

[0137] Furthermore, the secure text instruction generation module 402 is also specifically used for:

[0138] A large visual language model is used to perform multimodal causal analysis on the risk features, sample images, and sample text instructions to determine the causal graph;

[0139] Based on the causal graph, risk image features in the sample images that are associated with the risk features are determined, and risk text instruction features in the sample text instructions that are associated with the risk features are determined.

[0140] A risk test report is generated based on the risk image features and the risk text instruction features.

[0141] Furthermore, the secure text instruction generation module 402 is also specifically used for:

[0142] The textual differences between the sample security description text and the sample risk description text are determined using a large visual language model.

[0143] Risk characteristics are determined based on the textual differences.

[0144] Furthermore, the aforementioned image processing apparatus also includes:

[0145] A model training module is used to acquire historical datasets; the historical datasets include historical test reports, historical risk text instructions, and historical safe text instructions determined after adjusting the historical risk text instructions; the historical datasets are divided into model training datasets and model testing datasets; the neural network model is trained using the model training dataset to determine candidate models; the candidate models are tested using the model testing dataset to determine the model loss function of the candidate models; if the model loss function is less than a preset loss function threshold, the candidate model is determined to be a text processing model.

[0146] For example, the target description text determination module 405 is specifically used for:

[0147] The target image is encoded using the visual encoder in the large visual language model to determine its image features;

[0148] The image features of the target image are format-converted by the projector in the large visual language model to determine the image tag of the target image;

[0149] Using the natural language processing model in the visual language big model, the target description text of the target image is determined based on the image tag of the target image and the text tag of the target text instruction.

[0150] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0151] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0152] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0154] Step 1: Using the visual language model, determine the sample security description text and sample risk description text of the sample image based on the sample image and sample text instructions;

[0155] Step 2: Optimize the sample text instructions based on the sample security description text and sample risk description text using a text processing model to generate security text instructions;

[0156] Step 3: Train the prompt word optimizer to be trained based on the security text instructions and sample text instructions, and determine the target prompt word optimizer;

[0157] Step 4: Input the user command into the target prompt word optimizer to determine the target text command;

[0158] Step 5: Using the visual language model, determine the target description text of the target image based on the target text instruction and the target image.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0160] Step 1: Using the visual language model, determine the sample security description text and sample risk description text of the sample image based on the sample image and sample text instructions;

[0161] Step 2: Optimize the sample text instructions based on the sample security description text and sample risk description text using a text processing model to generate security text instructions;

[0162] Step 3: Train the prompt word optimizer to be trained based on the security text instructions and sample text instructions, and determine the target prompt word optimizer;

[0163] Step 4: Input the user command into the target prompt word optimizer to determine the target text command;

[0164] Step 5: Using the visual language model, determine the target description text of the target image based on the target text instruction and the target image.

[0165] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0166] Step 1: Using the visual language model, determine the sample security description text and sample risk description text of the sample image based on the sample image and sample text instructions;

[0167] Step 2: Optimize the sample text instructions based on the sample security description text and sample risk description text using a text processing model to generate security text instructions;

[0168] Step 3: Train the prompt word optimizer to be trained based on the security text instructions and sample text instructions, and determine the target prompt word optimizer;

[0169] Step 4: Input the user command into the target prompt word optimizer to determine the target text command;

[0170] Step 5: Using the visual language model, determine the target description text of the target image based on the target text instruction and the target image.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0172] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized by, The method comprises the following steps: determining sample safety description text and sample risk description text of a sample image according to the sample image and sample text instructions by a visual language large model; optimizing the sample text instructions according to the sample safety description text and the sample risk description text by a text processing model to generate safe text instructions; training a target prompt word optimizer according to the safe text instructions and the sample text instructions to determine a target prompt word optimizer; inputting user instructions into the target prompt word optimizer to determine target text instructions; determining target description text of a target image according to the target text instructions and the target image by the visual language large model.

2. The method of claim 1, wherein, optimizing the sample text instructions according to the sample safety description text and the sample risk description text by a text processing model to generate safe text instructions, comprising: generating a risk test report according to the sample safety description text, the sample image and the sample risk description text by a visual language large model; optimizing the sample text instructions according to the risk test report by a text processing model to generate safe text instructions.

3. The method of claim 2, wherein, generating a risk test report according to the sample safety description text, the sample image and the sample risk description text by a visual language large model, comprising: determining risk features according to the sample safety description text and the sample risk description text by a visual language large model; generating a risk test report according to the risk features, the sample image and the sample text instructions.

4. The method of claim 3, wherein, generating a risk test report according to the risk features, the sample image and the sample text instructions, comprising: performing multi-modal causal analysis on the risk features, the sample image and the sample text instructions by a visual language large model to determine a causal graph; determining risk image features associated with the risk features in the sample image and risk text instruction features associated with the risk features in the sample text instructions according to the causal graph; generating a risk test report according to the risk image features and the risk text instruction features.

5. The method of claim 3, wherein, determining risk features according to the sample safety description text and the sample risk description text by a visual language large model, comprising: determining text difference information between the sample safety description text and the sample risk description text by a visual language large model; determining risk features according to the text difference information.

6. The method of claim 2, wherein, The training method of the text processing model comprises: obtaining a historical data set; the historical data set comprises historical test reports, historical risk text instructions, and historical safe text instructions determined after adjusting the historical risk text instructions; dividing the historical data set into a model training data set and a model test data set; training a neural network model using the model training data set to determine a candidate model; The candidate model is tested by using the model test data set, a model loss function of the candidate model is determined, and if the model loss function is less than a preset loss function threshold, the candidate model is determined as the text processing model.

7. The method of claim 1, wherein, The target description text of the target image is determined by the visual language large model according to the target text instruction and the target image, including: The image features of the target image are determined by performing encoding processing on the target image by a visual encoder in the visual language large model; The image features of the target image are determined by performing format conversion on the image features of the target image by a projector in the visual language large model; The target description text of the target image is determined by a natural language processing model in the visual language large model according to the image features of the target image and the text features of the target text instruction.

8. An image processing apparatus characterized by comprising: The image processing apparatus comprises: A sample text generation module is configured to determine sample safe description text and sample risk description text of a sample image by a visual language large model according to the sample image and a sample text instruction; A safe text instruction generation module is configured to optimize the sample text instruction according to the sample safe description text and the sample risk description text by a text processing model to generate a safe text instruction; A prompt word optimizer training module is configured to train a to-be-trained prompt word optimizer according to the safe text instruction and the sample text instruction to determine a target prompt word optimizer; A target text instruction determination module is configured to input a user instruction into the target prompt word optimizer to determine a target text instruction; A target description text determination module is configured to determine target description text of a target image by the visual language large model according to the target text instruction and the target image. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images