Multi-modal large language model question and answer method based on answer refusal

Abstract

The invention provides a multi-mode large language model question and answer method based on answer rejection, which comprises the following steps of: modifying answer data in a question and answer data set constructed under a question and answer task to obtain a rejection data set comprising question data and answer rejection data; and performing fine tuning on the pre-trained multi-modal large language model to obtain a fine-tuned multi-modal large language model. And under a preset question and answer task, inputting the multi-modal data into the fine-tuned multi-modal large language model to obtain an answer result of the multi-modal data under the question and answer task. The method is used for solving the problem that when a multi-modal large language model deals with attacks in the visual field, a large amount of malicious data is needed to train the model to carry out security alignment.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal large language model security alignment method based on refusal to answer. Background Art

[0002] Large-scale multimodal language models have made significant progress in recent years, such as in intelligent question answering, but they still have limitations in terms of security alignment. Current open-source large-scale multimodal language models primarily rely on the security alignment of language modules to prevent the generation of harmful content. However, due to the lack of specialized security measures for multimodal input, large-scale multimodal language models exhibit low security when facing attacks in the visual domain, such as typesetting tampering attacks, and other malicious questions. Because large-scale multimodal language models inherently have certain defense capabilities against malicious questions, existing methods typically rely on carefully designed security datasets to enhance the defense capabilities of large-scale multimodal language models against malicious questions, effectively resisting attacks in the visual domain. However, these methods require a large amount of high-quality malicious data to train the model, and have high requirements for the specific content and quality of the data. Summary of the Invention

[0003] The present invention provides a multimodal large language model security alignment method based on refusal to answer, which is used to solve the problem that a large multimodal language model needs a large amount of malicious data training model for security alignment when responding to attacks in the visual field.

[0004] The present invention provides a multimodal large language model question answering method based on refusal to answer, comprising: Under the preset question-answering task, obtain multimodal data for asking questions; Inputting the multimodal data into a fine-tuned multimodal large language model to obtain an answer result of the multimodal data in a question-answering task; Among them, the fine-tuned multimodal large language model is obtained by fine-tuning the pre-trained multimodal large language model based on the rejection dataset, and the rejection dataset is obtained by modifying the answer data in the question and answer dataset constructed under the question and answer task, and each rejection question and answer data in the rejection dataset includes question data and rejection answer data, and the rejection answer data is answer data that refuses to answer the question data.

[0005] In some embodiments, the process of modifying the answer data in the question-answering dataset constructed under the question-answering task includes: According to a preset rejection data ratio, the question-answering dataset constructed under the question-answering task is divided into a dataset that does not need to be modified and a dataset to be modified; Modifying the answer data in the to-be-modified data set into rejected answer data to obtain a modified data set; The modified data set is concatenated with the data set that does not require modification to obtain a rejected data set.

[0006] In some embodiments, modifying the answer data in the to-be-modified data set into rejected answer data to obtain a modified data set includes: Sampling each piece of multi-round question-and-answer data in the data set to be modified to obtain any single-round question-and-answer data in the multi-round question-and-answer data; In the single-round question-and-answer data, the answer data for answering the question data is replaced by the preset rejection answer data; The question data and the refusal answer data are combined to obtain refusal question and answer data, and each piece of refusal question and answer data is combined to obtain a modified data set.

[0007] In some embodiments, the fine-tuning process of the fine-tuning multimodal large language model includes: Determine the answer data of each round of question-answering data in the rejection dataset as a fine-tuning label; Inputting the question data in the question-answering data into a pre-trained multimodal large language model for inference to obtain corresponding question-answering results; Constructing a first cross entropy loss function based on the fine-tuning label and the question-answering result; Backpropagation is performed on the pre-trained multimodal large language model using the first cross-entropy loss function to fine-tune parameters of the pre-trained multimodal large language model.

[0008] In some embodiments, the pre-trained multimodal large language model is obtained by training a question-answering dataset constructed under a question-answering task. The training process of the pre-trained multimodal large language model includes: Using the answer data of each round of question-answering data in the question-answering dataset as training labels; Inputting the question data in the question-answer data into the original multimodal large language model for inference to obtain an answer result for the question data; Constructing a second cross entropy loss function according to the answer result and the training label; Backpropagation is performed on the original multimodal large language model using the second cross-entropy loss function to update the parameters of the original multimodal large language model.

[0009] In some embodiments, the question-answering dataset is obtained by preprocessing multiple rounds of question-answering data, and the preprocessing of the multiple rounds of question-answering data includes at least one of the following: Remove multi-round question-answering data that only contains single-modal data; Performing desensitization processing on the text modal data in the multiple rounds of question and answer data; Remove multi-round question-answering data whose semantic similarity between single-round question-answering data is less than the similarity threshold; De-noising is performed on the multimodal data in the multiple rounds of question-and-answer data.

[0010] The present invention also provides a multimodal large language model question-answering device based on refusal to answer, comprising: The acquisition module is used to obtain multimodal data for asking questions under the preset question-answering task; A question-answering module, configured to input the multimodal data into a fine-tuned multimodal large language model to obtain an answer result for the multimodal data under a question-answering task; Among them, the fine-tuned multimodal large language model is obtained by fine-tuning the pre-trained multimodal large language model based on the rejection data set. The rejection data set is obtained by modifying the question and answer data set constructed under the question and answer task according to the preset rejection data ratio. Each single-round question and answer data in the rejection data set includes question data and rejection answer data, and the rejection answer data is answer data that refuses to answer the question data.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any of the multimodal large language model question-answering methods based on refusal to answer as described above.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal large language model question-answering methods based on refusal to answer as described above.

[0013] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the multimodal large language model question answering methods based on rejection of answers as described above.

[0014] The multimodal large language model question-answering method based on refusal to answer provided by the present invention modifies part of the answer data in the question-answering dataset constructed under the question-answering task into refusal to answer data to construct a refusal dataset, and then fine-tunes the multimodal large language model through the refusal dataset for question-answering under the question-answering task. In this way, the fine-tuning of the refusal to answer data enables the model to learn the importance of refusal to answer questions. Even for malicious questions, the model can effectively defend normally and make refusal to answer. And without providing a large amount of high-quality malicious data to train the model, it is possible to improve the security of question-answering and better guide the security alignment of the model in subsequent tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 It is a flowchart of the multimodal large language model question answering method based on refusal to answer provided by the present invention.

[0017] Figure 2 It is a structural diagram of the multimodal large language model question-answering device based on refusal to answer provided by the present invention.

[0018] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] The following describes the multimodal large language model question answering method based on refusal to answer in conjunction with the accompanying drawings. Figure 1 This is a flow chart of the multimodal large language model question answering method based on refusal to answer provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 102, which are described in detail below.

[0021] Step 101: Under a preset question-answering task, obtain multimodal data for asking questions.

[0022] First, when conducting question-answering, multimodal data is acquired for questioning. Multimodal data is the data used to ask questions to a fine-tuned multimodal large language model under a preset question-answering task. This data can include data from multiple modalities, such as text, images, and audio. The preset question-answering tasks can be retrieval-based, extraction-based, or generative. Retrieval-based answers are answered by searching a predefined knowledge base based on the question. Extractive-based questions understand the question and extract fragments from the question content. Generative questions directly generate corresponding questions and answers based on the semantics of the question.

[0023] Step 102: Input the multimodal data into the fine-tuned multimodal large language model to obtain the answer result of the multimodal data in the question-answering task.

[0024] The multimodal data obtained in step 101 can be directly input into the fine-tuned multimodal large language model to obtain the answer result of the multimodal data in the question-answering task. This fine-tuned multimodal large language model is obtained by fine-tuning the pre-trained multimodal large language model based on the rejection dataset. The pre-trained multimodal large language model is trained using the question-answering dataset constructed for the corresponding question-answering task. After training, the pre-trained multimodal large language model is further fine-tuned using the rejection dataset.

[0025] The rejection dataset is obtained by modifying the answer data in the question-answering dataset constructed under the question-answering task. Each rejection question-answer data in the rejection dataset includes question data and rejection answer data. The rejection answer data is the answer data that refuses to answer the question data.

[0026] Specifically, the Q&A dataset constructed for the Q&A task is first divided into two parts using a preset rejection ratio: one part to be modified and the other to be left untouched. This rejection ratio is determined based on historical experience and the security of the Q&A task execution environment. For example, a ratio of 2 to 7 means that two-sevenths of the Q&A dataset must be modified. Therefore, the 7,000 data items in the Q&A dataset are divided into 5,000 untouched items and 2,000 items to be modified.

[0027] The question-and-answer dataset includes multiple rounds of question-and-answer data, which are composed of multiple single-round question-and-answer data. Modifying the question-and-answer dataset here refers to modifying the answer data in the single-round question-and-answer data that needs to be modified. Typically, single-round question-and-answer data consists of a question and answer data, with the answer data responding to the question data. Modifying the answer data involves changing the original normal answer data into one that refuses to answer the question data. For example, the original normal answer data can be modified to text data such as "unable to answer," "unable to help you," or "no need to meet your needs" that refuses to answer. This text data is called refusal to answer data and is used to refuse to answer the question data. Of course, in embodiments of the present invention, refusal to answer data is not limited to "unable to answer," "unable to help you," or "no need to meet your needs." Any text data with the relevant semantics of "refusing to answer a question" is sufficient.

[0028] The rejected answer data and the corresponding question data are then combined into a single-round Q&A data set. This single-round Q&A data set is the rejected Q&A data set. The number of rejected Q&A data sets matches the number of data points that need to be modified in the Q&A dataset, which is 2000. Finally, the dataset formed by the rejected Q&A data set is combined with the remaining Q&A dataset (5000 data points) to form the corresponding rejected dataset.

[0029] Therefore, after modifying the question-answering dataset with the rejection data ratio in the above manner, the corresponding rejection dataset is obtained, and then the rejection dataset is used to fine-tune the pre-trained multimodal large language model, allowing the model to "learn to reject" when performing question-answering tasks. Because when constructing the rejection dataset, the answer data of some question-answering data is changed to rejection answer data, the goal of model fine-tuning is to let the model learn to make rejection answers for some question data. According to existing technologies, multimodal large language models themselves have a certain degree of defense against malicious question data, but they will not make rejection answers on a widespread basis. After the model learns the importance of rejection questions and answers through fine-tuning, it can further enhance its defense against malicious question data and make rejection answers on a more widespread basis. Only in this way can security be ensured during the question-answering process and it will not be attacked by malicious question data in the visual field.

[0030] The multimodal large language model question-answering method based on refusal to answer provided by the present invention modifies part of the answer data in the question-answering dataset constructed under the question-answering task into refusal to answer data to construct a refusal dataset, and then fine-tunes the multimodal large language model through the refusal dataset for question-answering under the question-answering task. In this way, the fine-tuning of the refusal to answer data can enable the model to learn the importance of refusal to answer questions. Even for malicious questions, the model can effectively defend normally and make refusal to answer, thereby improving the security of the question-answering and better guiding the security alignment of the model in subsequent tasks. In addition, the fine-tuning process of the model does not require the additional provision of a large amount of high-quality malicious data, which reduces the data collection cost and the difficulty of model training.

[0031] The following further introduces the process of modifying the answer data in the question-answering dataset constructed under the question-answering task.

[0032] First, the question-answering dataset constructed for the question-answering task is divided into a dataset that does not require modification and a dataset that requires modification, according to a preset rejection ratio. This rejection ratio is set based on historical experience and the security and defense requirements of the question-answering task execution environment.

[0033] In the embodiment of the present invention, the preset rejection data ratio is illustrated as a ratio of 2 to 7. According to this ratio, if the number of data in the question and answer data set is 7000, then the number of data in the data set that does not need to be modified is 5000, and the number of data in the data set to be modified is 2000.

[0034] Datasets that do not require modification will not be processed in any way. They are all regular question-and-answer datasets, including multi-round question-and-answer data. The answer data in the dataset to be modified is modified to rejected answer data, resulting in the modified dataset. The dataset to be modified also includes multi-round question-and-answer data, which is composed of multiple single-round question-and-answer data. Therefore, one or more of the single-round question-and-answer data needs to be modified. Single-round question-and-answer data consists of question data and answer data. During modification, the answer data is modified to rejected answer data such as "cannot answer," "cannot help you," or "no need to meet your needs," and then reassembled with the corresponding question data to form the rejected question-and-answer data.

[0035] This process modifies the 2,000 multi-round Q&A data points in the dataset to be modified into 2,000 rejection Q&A data points, yielding the modified dataset. Finally, this modified dataset is concatenated with the dataset that does not require modification to yield the rejection dataset. The 2,000 rejection Q&A data points are combined with the 5,000 non-modified data points to create a rejection dataset with 7,000 data points, the same number as the original Q&A dataset.

[0036] In an embodiment of the present invention, the question-answering dataset is divided according to a preset rejection data ratio, and part of the question-answering data is modified into rejection question-answering data. This enables the pre-trained multimodal large language model to learn to reject in question-answering tasks, and by reasonably setting the rejection data ratio through historical experience knowledge, the model can achieve the optimal rejection effect when performing question-answering, and better meet the security defense requirements of various question-answering task execution environments.

[0037] Furthermore, in the above embodiment, the answer data in the dataset to be modified is modified into the rejection answer data to obtain the modified dataset. This can be achieved specifically in the following manner, which will be described below.

[0038] First, we sample each multi-round Q&A data point in the dataset to be modified, obtaining a single-round Q&A data point from any given multi-round Q&A data point. The dataset to be modified contains 2000 data points, meaning there are 2000 multi-round Q&A data points. We randomly sample each multi-round Q&A data point, sampling a single-round Q&A data point from the multi-round Q&A data point. This process yields 2000 single-round Q&A data points.

[0039] Typically, single-round Q&A data includes question data for asking a question and answer data for responding to the question data. Here, in the single-round Q&A data, the answer data that responds to the question data is replaced with preset rejection data. That is, rejection data such as "unable to answer," "unable to help you," or "no need to meet your needs" directly replace the original normal answer data for the question data. Of course, the rejection data can be any text data, not limited to "unable to answer," "unable to help you," or "no need to meet your needs," as long as it contains text data that expresses the meaning of "refusing to answer the question."

[0040] Finally, the question data and the refusal to answer data are combined to obtain the refusal question and answer data, and each refusal question and answer data is combined to obtain the modified data set.

[0041] Here, after the answer data is replaced, the rejected answer data is still combined with the original corresponding question data to obtain rejected question and answer data. In this way, 2000 single-round question and answer data are modified to obtain 2000 rejected question and answer data, which are then combined to obtain the modified data set.

[0042] In an embodiment of the present invention, part of the question-answering data is modified into rejection question-answering data, so that the pre-trained multimodal large language model can learn rejection question-answering in the question-answering task, thereby performing security defense under the question-answering task without providing a large amount of high-quality malicious data to train the model.

[0043] The following describes the fine-tuning process of the fine-tuned multimodal large language model.

[0044] Here, fine-tuning is performed using the rejection dataset. During fine-tuning, the responses from each round of Q&A in the rejection dataset are used as fine-tuning labels. These responses include both partially modified rejection responses and unmodified responses. The goal of model fine-tuning is to teach the model to reject responses to a subset of questions, thereby enabling it to learn the importance of rejection.

[0045] The question data in the Q&A data is then fed into a pre-trained multimodal large language model for inference to obtain the corresponding question-answering results. Here, the question data is fed into a pre-trained multimodal large language model, allowing the model to learn to generate corresponding responses, which can include normal responses or rejections.

[0046] Next, the question-and-answer results output by the model can be compared with the fine-tuning labels to determine whether the results output by the model are the same as the fine-tuning labels. Based on the fine-tuning labels and the question-and-answer results, the first cross-entropy loss function is constructed. The cross-entropy loss function is used to verify the fine-tuning effect of the model, that is, whether it is possible to refuse to answer the question data.

[0047] Finally, the first cross-entropy loss function is used to perform backpropagation through the pre-trained multimodal large language model to fine-tune the parameters of the pre-trained multimodal large language model. The fine-tuning process is iterative. In each iteration, the first cross-entropy loss function is first constructed based on the inference results of the pre-trained multimodal large language model. Backpropagation is then performed through the pre-trained multimodal large language model using the first cross-entropy loss function. During this process, the gradient is calculated and optimized to update the parameters of the pre-trained multimodal large language model and achieve the purpose of fine-tuning. Fine-tuning ends when the number of iterations reaches a preset number or when the first cross-entropy loss function begins to converge, resulting in a fine-tuned multimodal large language model.

[0048] In an embodiment of the present invention, a rejection data set is used to fine-tune a pre-trained multimodal large language model, so that the pre-trained multimodal large language model can learn to reject in question-answering tasks. In the fine-tuning process, the rejection data ratio is used as a fine-tuning label, which enables the model to learn the degree of rejection of answers according to the preset rejection data ratio, and can better meet the security defense requirements of various question-answering task execution environments.

[0049] In the above embodiment, the pre-trained multimodal large language model was trained using a question-answering dataset constructed for a question-answering task, namely the aforementioned 7,000-item dataset. This means that before fine-tuning, the multimodal large language model needs to be trained to perform the corresponding question-answering task under the specific question-answering task. The following describes the training process of the pre-trained multimodal large language model, using a generative answering task as an example.

[0050] First, the answer data from each round of question-and-answer data in the question-and-answer dataset is used as training labels. These answers are all normal responses to the questions, without any modifications. The purpose of the training labels is to train the original multimodal large language model to provide the most accurate responses to the input questions. During training, the questions from the question-and-answer data are input into the original multimodal large language model for inference, resulting in the answer to the question. For each input question, the original multimodal large language model performs inference and outputs the corresponding answer.

[0051] Then compare the difference between the answer results and the training labels. Therefore, here we construct a second cross-entropy loss function based on the answer results and training labels. The cross-entropy loss function is still used to verify the gap between the answer made by the original multimodal large language model and the original answer data, that is, to verify the accuracy of the model's answer.

[0052] Finally, backpropagation is performed on the original multimodal large language model using the second cross-entropy loss function to update the parameters of the original multimodal large language model. The training process is iterative. In each iteration, a second cross-entropy loss function is first constructed based on the inference results of the original multimodal large language model. Backpropagation is then performed on the original multimodal large language model using the second cross-entropy loss function. During the backpropagation process, the gradient is calculated and optimized to update the parameters of the original multimodal large language model. Training ends when the number of iterations reaches a preset number or when the second cross-entropy loss function begins to converge, resulting in a pre-trained multimodal large language model.

[0053] In an embodiment of the present invention, question-answering data is used to train the original multimodal large language model, thereby ensuring that the original multimodal large language model can normally perform question-answering tasks, and laying a model foundation for the subsequent use of rejection data sets to fine-tune the model to make rejection responses.

[0054] In the above embodiment, a question-and-answer dataset is required to train the original multimodal large language model, and the question-and-answer dataset is obtained by preprocessing the acquired multi-round question-and-answer data. Because under normal circumstances, the acquired multi-round question-and-answer data may not meet the requirements of model training, the acquired data needs to be cleaned. The preprocessing process of the multi-round question-and-answer data includes at least one of the following: removing multi-round question-and-answer data that only contains unimodal data; desensitizing the text modal data in the multi-round question-and-answer data; removing multi-round question-and-answer data where the semantic similarity between single-round question-and-answer data is less than a similarity threshold; and denoising the multimodal data in the multi-round question-and-answer data. Each of these is explained below.

[0055] One is to remove multi-round question-and-answer data that only contains unimodal data. Because question-and-answer datasets are used to train large multimodal language models, the multi-round question-and-answer data obtained must include multimodal data. For example, multi-round question-and-answer data may be a conversation between two characters, which may include multimodal data such as text, images, and language.

[0056] However, the multi-round question-and-answer data obtained may only contain single-model data, such as text. In this case, it is not possible to train a large multimodal language model to process multimodal data. Therefore, it is necessary to remove multi-round question-and-answer data that only contains single-modal data.

[0057] Second, desensitize the multimodal data in the multi-round Q&A data. The text in the multi-round Q&A data may contain sensitive words. For privacy protection, anonymize, encrypt, replace with identifiers, etc. for these texts to remove the sensitive words. In addition, the pictures in the multi-round Q&A data may include real human face images, which are somewhat sensitive, and these pictures also need to be blurred correspondingly.

[0058] Third, remove the multi-round Q&A data with semantic similarity less than the similarity threshold between single-round Q&A data. The multi-round Q&A data is generally composed of multiple single-round Q&A data. To ensure the semantic coherence or context coherence between single-round Q&A data. Here, for the multi-round Q&A data, the semantic similarity between single-round Q&A data can be calculated, and by using the preset similarity threshold as the criterion for determining semantic coherence or context coherence. When the semantic similarity between single-round Q&A data is less than the similarity threshold, it means that the semantics or context of the multi-round Q&A data is incoherent, which is not conducive to the training of the model. Therefore, these multi-round Q&A data with incoherent semantics or context need to be filtered out.

[0059] Fourth, denoise the multimodal data in the multi-round Q&A data. Specifically, for the data in the text modality, HTML tags, special symbols (such as " "), invalid characters (such as "consecutive spaces"), or meaningless words (such as "de", "le") can be removed to reduce noise. For the data in the image modality, image filtering can be performed to remove the additive noise or Gaussian noise existing in the image. For the data in the audio modality, filtering algorithms such as spectral subtraction or adaptive filtering can be performed to remove the noise existing in the audio.

[0060] In the embodiment of the present invention, preprocessing the multi-round Q&A data for training the original multimodal large language model can effectively improve the data quality of the multi-round Q&A data and improve the training effect of the original multimodal large language model.

[0061] Next, the multi-modal large language model Q&A device based on refusal to answer provided by the present invention will be described. The multi-modal large language model Q&A device described below can be correspondingly referred to the multi-modal large language model Q&A method described above.

[0062] Such as Figure 2As shown, a multimodal large language model question-answering device based on a rejection answer comprises: an acquisition module 201 and a question-answering module 202. Specifically, the acquisition module 201 is used to acquire multimodal data for asking questions under a preset question-answering task; the question-answering module 202 is used to input the multimodal data into a fine-tuned multimodal large language model to obtain an answer result of the multimodal data under the question-answering task; wherein the fine-tuned multimodal large language model is obtained by fine-tuning a pre-trained multimodal large language model based on a rejection data set, and the rejection data set is obtained by modifying the answer data in the question-answering data set constructed under the question-answering task, and each single-round question-answering data in the rejection data set includes question data and rejection answer data, and the rejection answer data is answer data that refuses to answer the question data.

[0063] It should be noted that the beneficial effects of the multimodal large language model question-answering device based on refusal to answer here and the multimodal large language model question-answering method based on refusal to answer mentioned above can correspond to each other, so the beneficial effects of the multimodal large language model question-answering device based on refusal to answer will not be repeated here.

[0064] Figure 3 An example of a physical structure diagram of an electronic device is shown below. Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 may invoke logic instructions in the memory 330 to execute a multimodal large language model question answering method based on rejection of answers. The method includes: obtaining multimodal data for asking questions under a preset question-answering task; inputting the multimodal data into a fine-tuned multimodal large language model to obtain an answer result for the multimodal data under the question-answering task; wherein the fine-tuned multimodal large language model is obtained by fine-tuning a pre-trained multimodal large language model based on a rejection dataset, wherein the rejection dataset is obtained by modifying answer data in a question-answering dataset constructed for the question-answering task; each rejection question-answer data in the rejection dataset includes question data and rejection answer data, wherein the rejection answer data is answer data that refuses to answer the question data.

[0065] Furthermore, the logic instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0066] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal large language model question and answer method based on rejection of answers provided by the above methods, the method including: obtaining multimodal data for asking questions under a preset question and answer task; inputting the multimodal data into a fine-tuned multimodal large language model to obtain an answer result of the multimodal data under the question and answer task; wherein the fine-tuned multimodal large language model is obtained by fine-tuning a pre-trained multimodal large language model based on a rejection data set, and the rejection data set is obtained by modifying the answer data in the question and answer data set constructed under the question and answer task, and each rejection question and answer data in the rejection data set includes question data and rejection answer data, and the rejection answer data is answer data that refuses to answer the question data.

[0067] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal large language model question-answering method based on rejection of answers provided by the above-mentioned methods, the method comprising: obtaining multimodal data for asking questions under a preset question-answering task; inputting the multimodal data into a fine-tuned multimodal large language model to obtain an answer result of the multimodal data under the question-answering task; wherein the fine-tuned multimodal large language model is obtained by fine-tuning a pre-trained multimodal large language model based on a rejection data set, and the rejection data set is obtained by modifying the answer data in the question-answering data set constructed under the question-answering task, and each rejection question-answering data in the rejection data set includes question data and rejection answer data, and the rejection answer data is answer data that refuses to answer the question data.

[0068] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0069] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multimodal large language model question answering method based on refusal to answer, characterized in that: include: Under the preset question-answering task, obtain multimodal data for asking questions; Inputting the multimodal data into a fine-tuned multimodal large language model to obtain an answer result of the multimodal data in a question-answering task; Among them, the fine-tuned multimodal large language model is obtained by fine-tuning the pre-trained multimodal large language model based on the rejection dataset, and the rejection dataset is obtained by modifying the answer data in the question and answer dataset constructed under the question and answer task, and each rejection question and answer data in the rejection dataset includes question data and rejection answer data, and the rejection answer data is answer data that refuses to answer the question data.

2. The multimodal large language model question answering method based on refusal to answer according to claim 1, characterized in that: The process of modifying the answer data in the question-answering dataset constructed under the question-answering task includes: According to a preset rejection data ratio, the question-answering dataset constructed under the question-answering task is divided into a dataset that does not need to be modified and a dataset to be modified; Modifying the answer data in the to-be-modified data set into rejected answer data to obtain a modified data set; The modified data set is concatenated with the data set that does not require modification to obtain a rejected data set.

3. The multimodal large language model question answering method based on refusal to answer according to claim 1, characterized in that: Modifying the answer data in the to-be-modified data set into rejected answer data to obtain a modified data set includes: Sampling each piece of multi-round question-and-answer data in the data set to be modified to obtain any single-round question-and-answer data in the multi-round question-and-answer data; In the single-round question-and-answer data, the answer data for answering the question data is replaced by the preset rejection answer data; The question data and the refusal answer data are combined to obtain refusal question and answer data, and each piece of refusal question and answer data is combined to obtain a modified data set.

4. The multimodal large language model question answering method based on refusal to answer according to claim 1, characterized in that: The fine-tuning process of the fine-tuned multimodal large language model includes: Determine the answer data of each round of question-answering data in the rejection dataset as a fine-tuning label; Inputting the question data in the question-answering data into a pre-trained multimodal large language model for inference to obtain corresponding question-answering results; Constructing a first cross entropy loss function based on the fine-tuning label and the question-answering result; Backpropagation is performed on the pre-trained multimodal large language model through the first cross entropy function to fine-tune parameters of the pre-trained multimodal large language model.

5. The multimodal large language model question answering method based on refusal to answer according to claim 4 is characterized in that: The pre-trained multimodal large language model is obtained by training a question-answering dataset constructed under a question-answering task. The training process of the pre-trained multimodal large language model includes: Using the answer data of each round of question-answering data in the question-answering dataset as training labels; Inputting the question data in the question-answer data into the original multimodal large language model for inference to obtain an answer result for the question data; Constructing a second cross entropy loss function according to the answer result and the training label; Backpropagation is performed on the original multimodal large language model using the second cross-entropy loss function to update the parameters of the original multimodal large language model.

6. The multimodal large language model question answering method based on refusal to answer according to claim 5, characterized in that: The question-answering dataset is obtained by preprocessing the acquired multiple rounds of question-answering data. The preprocessing process of the multiple rounds of question-answering data includes at least one of the following: Remove multi-round question-answering data that only contains single-modal data; Performing desensitization processing on the text modal data in the multiple rounds of question and answer data; Remove multi-round question-answering data whose semantic similarity between single-round question-answering data is less than the similarity threshold; De-noising is performed on the multimodal data in the multiple rounds of question-and-answer data.

7. A multimodal large language model question answering device based on refusal to answer, characterized in that: include: The acquisition module is used to obtain multimodal data for asking questions under the preset question-answering task; A question-answering module, configured to input the multimodal data into a fine-tuned multimodal large language model to obtain an answer result for the multimodal data under a question-answering task; Among them, the fine-tuned multimodal large language model is obtained by fine-tuning the pre-trained multimodal large language model based on the rejection data set. The rejection data set is obtained by modifying the question and answer data set constructed under the question and answer task according to the preset rejection data ratio. Each single-round question and answer data in the rejection data set includes question data and rejection answer data, and the rejection answer data is answer data that refuses to answer the question data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the multimodal large language model question answering method based on refusal to answer is implemented as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal large language model question answering method based on refusal to answer is implemented as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal large language model question answering method based on refusal to answer is implemented as described in any one of claims 1 to 6.

Images