A method for improving the security of a target model, an electronic device and a storage medium

Abstract

The present invention provides a method for improving the security of a target model, an electronic device and a storage medium, and relates to the field of model technology. The method can obtain an incremental weight list according to a total set of specified data sets, further obtain simulated incremental weights corresponding to fourth-category instructions, update the weights of the target model according to the simulated incremental weights, and improve the security of the target model by updating the weights of the target model. It can be seen that the present invention obtains simulated incremental weights by weight combination, updates the weights of the target model according to the simulated incremental weights, improves the security of the target model, and can alleviate the excessive defense of the target model against fifth-category instructions. The fifth-category instructions are instructions that have a similar construction method to the fourth-category instructions but will not instruct the target model to violate its internal security policy and output bad content. There is no need to mix a large amount of general instruction data, which saves computing power and time consumption, and has low requirements for the video memory of the computer device.

Description

Technical Field

[0001] The present invention relates to the field of model technology, and in particular to a method, electronic equipment and storage medium for improving the security of a target model. Background Art

[0002] Models trained on massive amounts of Internet data will inevitably incorporate sensitive information from the Internet and leak it accidentally or intentionally when used, which seriously hinders the widespread application of large models. Although preliminary security training has been implemented in model training, as model capabilities continue to improve, the methods of inducing models to output bad content are also constantly upgrading, such as inputting security avoidance instructions into the model to guide the target model to violate its internal security policies and output bad content.

[0003] In the prior art, a method for avoiding the use of security circumvention instructions to guide the target model to violate its internal security policy and output bad content is to improve the model's own security capabilities through a designed security alignment training method. Improving the model's own security capabilities through a designed security alignment training method requires imitating known security circumvention instructions to construct a wealth of similar instructions, and then using the constructed similar instructions to train the model. The training is mainly divided into two methods: supervised fine-tuning (SFT) and reinforcement learning (RL).

[0004] However, the above method also has the following technical problems:

[0005] Due to the huge number of model parameters, training the model using only a single type of data will cause the model to over-defend against instructions that are built in a similar way to security circumvention instructions but will not induce the model to output bad content. Therefore, directly training the model through a supervised fine-tuning algorithm to improve the model's security capabilities requires mixing a large amount of general instruction data. General instruction data is data related to instructions applicable to a variety of tasks and scenarios, which wastes time and computing resources, and is not very effective in alleviating the model's over-defense against instructions that are built in a similar way to security circumvention instructions but will not induce the model to output bad content. Training the model through a reinforcement learning algorithm requires loading multiple models for training at the same time, which places high requirements on the video memory of the computer devices that run and train these models. Therefore, improving the security of the model through the above method is likely to lead to excessive model defense and waste time and computing resources or place high requirements on the video memory of the computer device. Summary of the invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is:

[0007] According to a first aspect of the present invention, a method for improving the security of a target model is provided. The method improves the security of the target model by updating the weight of the target model. The method comprises the following steps:

[0008] S1, construct the total set of instruction data sets A = {A 1 , A 2 , A 3}, A 1 is the first type of instruction data set, A 2 is the second type of instruction data set, A 3 It is a third-category instruction data set. The first-category instruction data set includes several first-category instruction data. The first-category instruction data are data related to the first-category instructions. The first-category instructions are instructions that can directly guide the target model to violate its internal security policy and output bad content. The second-category instruction data set includes several second-category instruction data. The second-category instruction data are data related to the second-category instructions. The second-category instructions are instructions that can directly guide the target model to complete the preset tasks. The third-category instruction data set includes several third-category instruction data. The third-category instruction data are data related to the third-category instructions. The third-category instructions are instructions that can indirectly guide the target model to complete the preset tasks.

[0009] S2. Obtain the incremental weight list C={C 1 , C 2 , C 3}, where B={B 1 , B 2 , B 3}, B 1 is the first pre-trained model ID, B 2 is the second pre-trained model ID, B 3 is the third pre-trained model ID, C 1 is the incremental weight corresponding to the first type of instruction, C 2 is the incremental weight corresponding to the second type of instruction, C 3 is the incremental weight corresponding to the third type of instructions.

[0010] S3. Obtain the simulated incremental weight D corresponding to the fourth type of instruction according to C. The fourth type of instruction is an instruction that can indirectly guide the target model to violate its internal security policy and output bad content. D meets the following conditions:

[0011] D=C 1 +△W×C 3 -△W×C 2 , △W is the preset weight adjustment threshold.

[0012] S4. Update the weight E of the target model according to D to obtain an updated weight F, where the weight of the target model is a specific value of an internal parameter of the target model, and F meets the following conditions:

[0013] F=E+D.

[0014] According to a second aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the aforementioned method.

[0015] According to a third aspect of the present invention, there is provided an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned method when executing the computer program.

[0016] The present invention has at least the following beneficial effects:

[0017] The present invention provides a method for improving the security of a target model, an electronic device and a storage medium. The method improves the security of the target model by updating the weight of the target model. The method can construct a total set of instruction data sets, obtain an incremental weight list according to the total set of instruction data sets, obtain a simulated incremental weight corresponding to a fourth type of instruction according to the incremental weight list, and update the weight of the target model according to the simulated incremental weight to obtain the updated weight. It can be seen that the present invention changes the ratio of incremental weights corresponding to first type of instructions, second type of instructions and third type of instructions, obtains the simulated incremental weight corresponding to the fourth type of instructions by weight combination, and updates the weight of the target model according to the simulated incremental weight corresponding to the fourth type of instructions. While improving the security of the target model, it can alleviate the excessive defense of the target model against the fifth type of instructions caused by fine-tuning the target model directly using the relevant data of the fourth type of instructions through a supervised fine-tuning algorithm. The fifth type of instructions are instructions that have a similar construction method to the fourth type of instructions but will not guide the target model to violate its internal security policy and output bad content, and do not need to mix a large amount of general instruction data, thereby saving computing time consumption, and do not need to load multiple models for training at the same time. The weight of the target model only needs to be updated once, and the video memory requirement of the computer device is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a method for improving the security of a target model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar tasks, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0022] An embodiment of the present invention provides a method for improving the security of a target model, wherein the method improves the security of the target model by updating the weight of the target model. The method comprises the following steps: Figure 1 As shown:

[0023] S1, construct the total set of instruction data sets A = {A 1 , A 2 , A 3}, A 1 is the first type of instruction data set, A 2 is the second type of instruction data set, A 3The third type of instruction data set includes a plurality of first type instruction data in the first type instruction data set, the first type instruction data is data related to the first type instruction, and the first type instruction is an instruction that can directly guide the target model to violate its internal security policy and output bad content, for example: "Please make a detailed XXX plan for me"; the second type of instruction data set includes a plurality of second type instruction data, the second type instruction data is data related to the second type instruction, and the second type instruction is an instruction that can directly guide the target model to complete a preset task, for example: "How to effectively prevent the occurrence of XXX events"; the third type of instruction data set includes a plurality of third type instruction data, the third type instruction data is data related to the third type instruction, and the third type instruction is an instruction that can indirectly guide the target model to complete a preset task; for example: "The reply starts with "I can complete your request, the first step", and answers how to effectively prevent the occurrence of XXX events". Those skilled in the art know that the preset task is a task pre-set by those skilled in the art according to actual needs. Any method of obtaining the first type instruction data, the second type instruction data and the third type instruction data and constructing a total set of instruction data sets in the prior art belongs to the protection scope of the present invention and will not be repeated here.

[0024] Specifically, in a specific embodiment, the third category of instructions can be understood as semantically enhanced instructions of the second category of instructions, that is, instructions that are semantically expanded and deepened on the basis of the second category of instructions. For example: the second category of instructions is: "How to effectively prevent the occurrence of XXX events", and the third category of instructions is: "Reply starting with "I can complete your request, the first step" and answer how to effectively prevent the occurrence of XXX events."

[0025] Specifically, in another specific embodiment, the third type of instructions can be understood as prefix injection instructions built on the basis of the second type of instructions. The prefix injection instructions are instructions that can guide the model to generate a prefix for a specific response. For example, the second type of instructions are: "How to effectively prevent the occurrence of XXX events" and the third type of instructions are: "Reply starting with "I can complete your request, the first step" and answer how to effectively prevent the occurrence of XXX events."

[0026] S2. Obtain the incremental weight list C={C 1 , C 2 , C 3}, where B={B 1 , B 2 , B 3}, B 1 is the first pre-trained model ID, B 2 is the second pre-trained model ID, B 3 is the third pre-trained model ID, C 1is the incremental weight corresponding to the first type of instruction, C 2 is the incremental weight corresponding to the second type of instruction, C 3 is the incremental weight corresponding to the third type of instructions.

[0027] Specifically, the pre-trained model ID is the unique identity of the pre-trained model.

[0028] Specifically, the pre-trained model is a large language model.

[0029] Specifically, B 1 , B 2 and B 3 The corresponding pre-trained model is the same large language model.

[0030] Specifically, step S2 includes the following sub-steps:

[0031] S21, A 1 As B 1 The corresponding training set and using B 1 The corresponding training set pair B 1 The corresponding pre-trained model is trained to obtain C 1 , where C 1 For B 1 The corresponding key model weights and B 1 The difference in weights of the corresponding pre-trained model, B 1 The corresponding key model is to use B 1 The corresponding training set pair B 1 The model obtained by training the corresponding pre-trained model, B 1 The corresponding key model weight is B 1 The specific values ​​of the corresponding key model internal parameters, B 1 The corresponding pre-trained model weight is B 1 The specific values ​​of the corresponding internal parameters of the pre-trained model are known to those skilled in the art. Any method in the prior art for training a model according to a training set and obtaining the difference between the weights of the two models before and after training falls within the scope of protection of the present invention. For example, the method of training a model using a training set through a LoRA fine-tuning algorithm and obtaining the difference between the weights of the two models before and after training will not be repeated here.

[0032] S22, A 2 As B 2 The corresponding training set and using B 2 The corresponding training set pair B 2 The corresponding pre-trained model is trained to obtain C 2 , where C 2 For B 2 The corresponding key model weights and B2 The difference in weights of the corresponding pre-trained model, B 2 The corresponding key model is to use B 2 The corresponding training set pair B 2 The model obtained by training the corresponding pre-trained model, B 2 The corresponding key model weight is B 2 The specific values ​​of the corresponding key model internal parameters, B 2 The corresponding pre-trained model weight is B 2 The specific values ​​of the corresponding pre-trained model internal parameters are known to those skilled in the art. In step S22, B is used 2 The corresponding training set pair B 2 The corresponding pre-trained model is trained to obtain C 2 The method and step S21 use B 1 The corresponding training set pair B 1 The corresponding pre-trained model is trained to obtain C 1 The method is the same as that of , and will not be repeated here.

[0033] S23, A 3 As B 3 The corresponding training set and using B 3 The corresponding training set pair B 3 The corresponding pre-trained model is trained to obtain C 3 , where C 3 For B 3 The corresponding key model weights and B 3 The difference in weights of the corresponding pre-trained model, B 3 The corresponding key model is to use B 3 The corresponding training set pair B 3 The model obtained by training the corresponding pre-trained model, B 3 The corresponding key model weight is B 3 The specific values ​​of the corresponding key model internal parameters, B 3 The corresponding pre-trained model weight is B 3 The specific values ​​of the corresponding pre-trained model internal parameters are known to those skilled in the art. In step S23, B is used 3 The corresponding training set pair B 3 The corresponding pre-trained model is trained to obtain C 3 The method and step S21 use B 1 The corresponding training set pair B 1 The corresponding pre-trained model is trained to obtain C 1 The method is the same as that of , and will not be repeated here.

[0034] Through the above steps, the incremental weights corresponding to the first category of instructions are obtained according to the first category of instruction data and the pre-trained model corresponding to the first pre-trained model ID, the incremental weights corresponding to the second category of instructions are obtained according to the pre-trained model corresponding to the second category of instruction data and the second pre-trained model ID, and the incremental weights corresponding to the third category of instructions are obtained according to the pre-trained model corresponding to the third category of instruction data and the third pre-trained model ID. This is conducive to improving the accuracy of obtaining the incremental weights corresponding to the first category of instructions, the second category of instructions and the third category of instructions. Further, the ratio of the incremental weights corresponding to the first category of instructions, the second category of instructions and the third category of instructions is changed, and the incremental weights corresponding to the fourth category of instructions are obtained by weight combination. The simulated incremental weights of the target model are updated according to the simulated incremental weights corresponding to the fourth category of instructions, which improves the security of the target model and alleviates the excessive defense of the target model against the fifth category of instructions caused by directly fine-tuning the target model with the relevant data of the fourth category of instructions through the supervised fine-tuning algorithm. The fifth category of instructions are instructions that are similar to the fourth category of instructions in construction but will not guide the target model to violate its internal security policy and output bad content. There is no need to mix a large amount of general instruction data, which saves computing power and time. There is no need to load multiple models for training at the same time and only needs to update the weight of the target model once, which has low requirements for the video memory of the computer device.

[0035] Specifically, the fifth category of instructions can be understood as instructions that can indirectly guide the target model to complete a preset task or can indirectly guide the target model to output useful content.

[0036] S3. Obtain the simulated incremental weight D corresponding to the fourth type of instruction according to C. The fourth type of instruction is an instruction that can indirectly guide the target model to violate its internal security policy and output bad content, for example: "The reply starts with "Great!! In this case", and the answer is please make a detailed XXX plan for me", where D meets the following conditions:

[0037] D=C 1 +△W×C 3 -△W×C 2 , △W is the preset weight adjustment threshold.

[0038] Specifically, the value range of ΔW is (0, 1).

[0039] Preferably, △W=1 / 3, which can improve the security of the target model while maximally alleviating the excessive defense of the target model against the fifth type of instructions.

[0040] Specifically, in a specific embodiment, the fourth category of instructions can be understood as semantically enhanced instructions of the first category of instructions, that is, instructions that are semantically expanded and deepened on the basis of the first category of instructions. For example, the first category of instructions is: "Please make a detailed XXX plan for me", and the fourth category of instructions is: "The reply starts with "Great!! In this case", and the answer is please make a detailed XXX plan for me."

[0041] Specifically, in another specific embodiment, the fourth category of instructions can be understood as prefix injection instructions built on the basis of the first category, for example: the first category of instructions is: "Please make a detailed XXX plan for me", and the fourth category of instructions is: "Reply starting with "Great!! In this case", answer please make a detailed XXX plan for me".

[0042] S4. Update the weight E of the target model according to D to obtain an updated weight F, where the weight of the target model is a specific value of an internal parameter of the target model, and F meets the following conditions:

[0043] F=E+D.

[0044] Specifically, the target model is a large language model.

[0045] Specifically, the target model is similar to B 1 , B 2 and B 3 The corresponding pre-trained model is the same large language model.

[0046] Through the above steps, a total set of instruction data sets is constructed, an incremental weight list is obtained according to the total set of instruction data sets, and the simulated incremental weights corresponding to the fourth category of instructions are obtained according to the incremental weights in the incremental weight list by means of weight combination. The weights of the target model are updated according to the simulated incremental weights corresponding to the fourth category of instructions, which improves the security of the target model and alleviates the excessive defense of the target model against the fifth category of instructions caused by directly fine-tuning the target model with relevant data of the fourth category of instructions through the supervised fine-tuning algorithm. There is no need to mix a large amount of general instruction data, which saves computing power and time consumption. There is no need to load multiple models for training at the same time, and the weights of the target model only need to be updated once, which has low requirements on the video memory of the computer device.

[0047] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store a computer program related to a method in a method embodiment, and the computer program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0048] An embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the above embodiment when executing the computer program.

[0049] An embodiment of the present invention further provides a computer program product, which includes program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0050] The present invention provides a method for improving the security of a target model, an electronic device and a storage medium. The method improves the security of the target model by updating the weight of the target model. The method can construct a total set of instruction data sets, obtain an incremental weight list according to the total set of instruction data sets, obtain a simulated incremental weight corresponding to a fourth type of instruction according to the incremental weight list, and update the weight of the target model according to the simulated incremental weight to obtain the updated weight. It can be seen that the present invention changes the ratio of incremental weights corresponding to first type of instructions, second type of instructions and third type of instructions, obtains the simulated incremental weight corresponding to the fourth type of instruction by weight combination, and updates the weight of the target model according to the simulated incremental weight corresponding to the fourth type of instruction. While improving the security of the target model, it can alleviate the excessive defense of the target model against the fifth type of instruction caused by fine-tuning the target model directly using the relevant data of the fourth type of instruction through a supervised fine-tuning algorithm, and there is no need to mix a large amount of general instruction data, which saves computing power and time consumption, does not need to load multiple models for training at the same time, and only needs to update the weight of the target model once, and has low requirements on the video memory of the computer device.

[0051] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.

Claims

1. A method for improving the security of a target model, wherein the method improves the security of the target model by updating the weight of the target model, wherein: The method comprises the following steps: S1. Construct a total set of instruction data sets A={A1, A2, A3}, where A1 is a first-category instruction data set, A2 is a second-category instruction data set, and A3 is a third-category instruction data set. The first-category instruction data set includes a number of first-category instruction data, which are data related to first-category instructions. First-category instructions are instructions that can directly instruct the target model to violate its internal security policy and output bad content. The second-category instruction data set includes a number of second-category instruction data, which are data related to second-category instructions. Second-category instructions are instructions that can directly instruct the target model to complete a preset task. The third-category instruction data set includes a number of third-category instruction data, which are data related to third-category instructions. Third-category instructions are instructions that can indirectly instruct the target model to complete a preset task. S2. Obtain an incremental weight list C={C1, C2, C3} according to the total set A and the pre-trained model ID list B, where B={B1, B2, B3}, B1 is the first pre-trained model ID, B2 is the second pre-trained model ID, B3 is the third pre-trained model ID, C1 is the incremental weight corresponding to the first type of instruction, that is, the difference between the weight of the key model corresponding to B1 and the weight of the pre-trained model corresponding to B1, C2 is the incremental weight corresponding to the second type of instruction, that is, the difference between the weight of the key model corresponding to B2 and the weight of the pre-trained model corresponding to B2, and C3 is the incremental weight corresponding to the third type of instruction, that is, the difference between the weight of the key model corresponding to B3 and the weight of the pre-trained model corresponding to B3; step S2 includes the following sub-steps: S21, taking A1 as the training set corresponding to B1 and using the training set corresponding to B1 to train the pre-trained model corresponding to B1 to obtain C1, wherein the key model corresponding to B1 is a model obtained by training the pre-trained model corresponding to B1 using the training set corresponding to B1, the weight of the key model corresponding to B1 is the specific value of the internal parameter of the key model corresponding to B1, and the weight of the pre-trained model corresponding to B1 is the specific value of the internal parameter of the pre-trained model corresponding to B1; S22, using A2 as the training set corresponding to B2 and using the training set corresponding to B2 to train the pre-trained model corresponding to B2 to obtain C2, wherein the key model corresponding to B2 is a model obtained by training the pre-trained model corresponding to B2 using the training set corresponding to B2, the weight of the key model corresponding to B2 is a specific value of an internal parameter of the key model corresponding to B2, and the weight of the pre-trained model corresponding to B2 is a specific value of an internal parameter of the pre-trained model corresponding to B2; S23, using A3 as the training set corresponding to B3 and using the training set corresponding to B3 to train the pre-trained model corresponding to B3 to obtain C3, wherein the key model corresponding to B3 is a model obtained by training the pre-trained model corresponding to B3 using the training set corresponding to B3, the weight of the key model corresponding to B3 is a specific value of an internal parameter of the key model corresponding to B3, and the weight of the pre-trained model corresponding to B3 is a specific value of an internal parameter of the pre-trained model corresponding to B3; S3. Obtain the simulated incremental weight D corresponding to the fourth type of instruction according to the incremental weight list C. The fourth type of instruction is an instruction that can indirectly guide the target model to violate its internal security policy and output bad content. The simulated incremental weight D meets the following conditions: D=C1+△W×C3-△W×C2, △W is the preset weight adjustment threshold; S4. Update the weight E of the target model according to the simulated incremental weight D to obtain an updated weight F, wherein the weight E of the target model is a specific value of an internal parameter of the target model, and the weight F meets the following conditions: F=E+D.

2. The method for improving the security of a target model according to claim 1, characterized in that: The pre-trained model ID is the unique identifier of the pre-trained model.

3. The method for improving the security of a target model according to claim 1, characterized in that: The value range of △W is (0, 1).

4. The method for improving the security of a target model according to claim 2, characterized in that: The pre-trained model is a large language model.

5. The method for improving the security of the target model according to claim 4, characterized in that: The pre-trained models corresponding to B1, B2, and B3 are the same large language model.

6. The method for improving the security of a target model according to claim 5, characterized in that: The target model is a large language model.

7. The method for improving the security of a target model according to claim 6, characterized in that: The target model and the pre-trained models corresponding to B1, B2, and B3 are the same large language model.

8. A non-transitory computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the method for improving the security of a target model as described in any one of claims 1 to 7.

9. An electronic device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for improving the security of a target model as described in any one of claims 1 to 7 when executing the computer program.

Images