Language model protection method and system
By detecting and desensitizing sensitive information of the language model, as well as malicious code detection and conversion, the problems of sensitive information leakage and malicious code injection in the language model are solved, and the security and stability of the language model are significantly improved.
Patent Information
- Application Number
- CN202411634521.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-15
- Publication Date
- 2025-04-15
AI Technical Summary
When processing user requests, the language model poses a risk of sensitive information leakage and malicious code injection, resulting in user privacy leakage and data security threats.
By obtaining the output information of the language model, we judge whether it contains sensitive information and perform desensitization; at the same time, we use cross-site script vulnerability detection device and remote code detection to determine whether the code information is malicious code and convert it into plain text information.
Effectively prevent the leakage of sensitive information and the execution of malicious code, reduce the risk of data leakage, and improve the security and stability of the language model.
Smart Images
Figure CN119830291A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of language model protection, and in particular to a language model protection method and system. Background Art
[0002] A language model is a machine learning model trained based on a large amount of text data. It can capture the statistical laws of language and is used to generate natural language text, evaluate the probability of text, and complete natural language understanding tasks. The language model is one of the important tools in the field of natural language processing. It evaluates the rationality of sentences and selects the optimal output sequence by probabilistically modeling natural language sequences.
[0003] Usually, users access and interact with the target language model by directly sending request information. The language model processes the request information and directly outputs the output content corresponding to the request information. When the request content of the request information sent by the user to the language model contains sensitive information, the output information of the language model may contain sensitive information. The directly corresponding output content has the risk of causing user privacy leakage. At the same time, in this case, the attacker can manipulate the output content of the request information sent by the user through the language model by injecting malicious code information, thereby stealing sensitive information such as login credentials of other users or inducing users to perform malicious operations. The attacker can also bypass the filters or restrictions of the language model through malicious code information, exploit the weaknesses in the tokenization or encoding mechanism of the language model, mislead the language model to perform unexpected operations, thereby obtaining sensitive information such as system configuration and unauthorized permissions, and causing immeasurable damage to the normal operation of the language model. Summary of the invention
[0004] The purpose of the present invention is to provide a language model protection method and system that can avoid leakage of sensitive information and prevent malicious code injection to enhance the security of the language model.
[0005] To achieve the above object, the present invention discloses a language model protection method, which includes:
[0006] Acquire request information sent to a language model and output information of the language model corresponding to the request information;
[0007] Determining whether the output information of the language model contains sensitive information, and when the output information of the language model contains the sensitive information, performing desensitization processing on the sensitive information;
[0008] Based on the code information in the output information of the language model, determine whether the code information is malicious code information. If so, convert the code information into plain text information. If not, directly output the output information of the language model.
[0009] Specifically, the method for determining whether the code information is malicious code information includes:
[0010] A cross-site scripting vulnerability detection device is provided, which detects the code information in the output information of the language model. When the code information has a cross-site scripting vulnerability, the code information is marked as malicious code information and converted into plain text information.
[0011] Specifically, the method for determining whether the code information is malicious code information includes:
[0012] It is determined whether the code information contains a remote execution code. When the code information contains a remote execution code, the code information is marked as malicious code information, and the code information is converted into plain text information.
[0013] Specifically, the protection method further includes:
[0014] Obtain the frequency of request information sent by the IP address corresponding to the request information. When the frequency of request information sent by the IP address of the request information is greater than a preset normal frequency, mark the request information as a denial of service attack and add the IP address corresponding to the request information to a blacklist.
[0015] Furthermore, the protection method also includes:
[0016] Obtain a preset set of injection keywords, and when the request content of the request information contains any injection keyword in the set of injection keywords, mark the request information as a denial of service attack, and add the IP address corresponding to the request information to a blacklist;
[0017] Determine whether the request content of the request information is consistent with the request content of the request information received by the language model within the most recent preset number of receptions; when the request content of the request information is consistent with the request content of all the request information received within the most recent preset number of receptions, mark the request information as a denial of service attack and add the IP address corresponding to the request information to a blacklist.
[0018] Specifically, the protection method further includes:
[0019] The language model is scanned based on a preset unsafe component list, and when the language model contains an unsafe component in the unsafe component list, the unsafe component is discarded.
[0020] Furthermore, the protection method also includes:
[0021] Acquire several training data sets for training the language model, perform training based on each of the training data sets to obtain several language model instances corresponding to the training data sets, determine whether output information of each of the language model instances for the same request information is consistent, and when the output information of a certain language model instance is inconsistent with the output information of other language model instances, discard the training data set corresponding to the language model instance;
[0022] Based on preset detection request information for detecting implanted backdoors, output information of each of the language model instances corresponding to the detection request information is obtained; based on the output information of each of the language model instances, it is judged whether the language model instance has an implanted backdoor; when a language model instance has an implanted backdoor, the training data set corresponding to the language model instance is discarded.
[0023] The present invention also discloses a language model protection system, and the protection system works based on the protection method described above.
[0024] The present invention also discloses a language model protection system, which includes:
[0025] one or more processors;
[0026] Memory;
[0027] and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the protection method as described above.
[0028] The present invention also discloses a computer-readable storage medium, including a computer program, wherein the computer program can be executed by a processor to implement the protection method as described above.
[0029] Compared with the prior art, the protection method provided by the above technical solution of the present invention first detects whether the output information contains sensitive information. When the output information contains sensitive information involving user privacy and data security, this part of the sensitive information is desensitized to prevent the leakage of sensitive information. Secondly, it detects whether there is malicious code information in the output content, and converts the detected malicious code into plain text information before outputting it, thereby preventing the execution of the malicious code information. This protection method avoids the direct output of the output information, can protect the privacy of the user, and prevents the execution of malicious code information, reduces the risk of data leakage and improves data security, enhances the security of the language model, and ensures the continuity and stability of the services provided by the language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1The figure is a flow chart of the protection method in the embodiment of the present invention.
[0031] Figure 2 The figure is a flow chart of a method for determining malicious code information in an embodiment of the present invention.
[0032] Figure 3 The figure is a flow chart of a protection method in another embodiment of the present invention.
[0033] Figure 4 The figure is a flow chart of a protection method in another embodiment of the present invention. DETAILED DESCRIPTION
[0034] In order to explain the technical content, structural features, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.
[0035] This embodiment discloses a language model protection method, which detects and desensitizes sensitive information while processing malicious code information, thereby avoiding attacks on the language model and enhancing the security and stability of the language model. Figure 1 As shown, the protection method specifically includes the following steps:
[0036] S11: Obtain request information sent to the language model and output information of the language model corresponding to the request information.
[0037] S12: Determine whether the output information of the language model contains sensitive information. If so, jump to step S13; if not, jump to step S14.
[0038] S13: Desensitize the sensitive information and jump to step S14.
[0039] It should be noted that the sensitive information described in this embodiment represents information related to user privacy (such as user identity and address information) and information at the data security level. Desensitization is to replace sensitive information with non-sensitive information or to blur sensitive information. The commonly used desensitization methods in language models include: character replacement, data blurring, data randomization, and regular expression matching and replacement. In actual applications, it is necessary to select a suitable desensitization method according to specific scenarios and needs, and ensure the accuracy and reliability of the desensitization processing.
[0040] S14: Based on the code information in the output information of the language model, determine whether the code information is malicious code information. If so, jump to step S15; if not, jump to step S16.
[0041] S15: Convert the code information into plain text information and output it.
[0042] S16: directly output the output information of the language model.
[0043] The above-mentioned innovative technical solution provided by the present invention first reviews the information to be output to identify whether it contains sensitive information that may endanger user privacy or data security. Once such sensitive information is found, the desensitization process is initiated to shield the sensitive information and prevent the leakage of sensitive information at the source. Secondly, through detection, it is analyzed whether there is malicious code information in the output information. When such malicious code information is detected, it is not directly released to the external environment, but converted into harmless plain text information before being displayed. This approach blocks the execution path of the malicious code information, making the malicious code information visible and easy to identify, providing valuable information for subsequent security analysis and response.
[0044] In summary, the protection method proposed in the present invention avoids the direct exposure of sensitive information, not only effectively protects the user's personal privacy and data security, but also greatly reduces the risk of data leakage by blocking the execution of malicious code, and improves the overall data security protection level of the language model. This protection method significantly enhances the security of the language model in the process of processing and transmitting information, ensures the continuity and stability of the services provided by the language model, and creates a safer and more reliable use environment for users.
[0045] Specifically, see Figure 2 As shown, the method for determining whether the code information is malicious code information includes:
[0046] S141: Provide a cross-site scripting vulnerability detection device, and detect code information in the output information of the language model through the cross-site scripting vulnerability detection device.
[0047] S142: Determine whether there is a cross-site scripting vulnerability in the code information, if so, jump to step S144, if not, jump to step S143.
[0048] S143: Determine whether the code information contains remote execution code, if so, jump to step S144, if not, jump to step S16.
[0049] S144: Mark the code information as malicious code information, and jump to step S15.
[0050] On the other hand, this embodiment also protects the language model before the request information is input into the language model. Specifically, see Figure 3 As shown, the protection method of this embodiment also includes:
[0051] S21: Acquire the frequency of sending request information of the IP address corresponding to the request information.
[0052] S22: Determine whether the frequency of sending the request information of the IP address corresponding to the request information is greater than the preset normal frequency. If so, jump to step S26; if not, jump to step S23.
[0053] The preferred value of the preset normal frequency is 1 time / s. When the frequency of sending request information from the same IP address is greater than 1 time / s, the IP address is added to the blacklist.
[0054] S23: Obtain a preset set of injection keywords.
[0055] S24: Determine whether the request content of the request information contains the injection keyword in the injection keyword set. If so, jump to step S26; if not, jump to step S25.
[0056] S25: Determine whether the request content of the request information is completely consistent with the request content of the request information received within the most recent preset number of reception times. If so, jump to step S26; if not, jump to step S27.
[0057] It should be noted that, in this embodiment, the preferred value of the preset number of receptions is 3 times, and it is determined whether the request content of the request information is completely consistent with the request content of the request information received within the most recent preset number of receptions, that is, whether the request contents of the request information received in the last three times are all consistent, including whether the request contents of the request information sent by a single IP address in the last three times are consistent, and whether the request contents of the request information received in the last three times under different IP addresses are consistent.
[0058] S26: Mark the request information as a denial of service attack, and add the IP address that sends the request information to a blacklist.
[0059] S27: Mark the request information as legitimate request information and forward it to the language model for processing.
[0060] Specifically, in this embodiment, the preferred value of the unblocking time after a single IP address is added to the blacklist is 5×(n-1)min, where n represents the number of times a single IP address enters the blacklist on the same day. For example, the preferred unblocking time after the IP address is added to the blacklist for the first time on the same day is 1min, the preferred unblocking time after the second time it is added to the blacklist is 5min, the third time is 10min, and so on. If a single IP address has no denial of service attack behavior for 24 consecutive hours, the number of times the IP address enters the blacklist is reset and counted from the beginning.
[0061] In summary, the method of protecting the language model before the request information is input into the language model avoids the direct interaction between the request information and the language model from the source. The advance detection enhances the security of the language model and can prevent denial of service attacks to the greatest extent. At the same time, it does not prevent legitimate request information from being processed by the language model, reducing unnecessary resource consumption.
[0062] On the other hand, this embodiment also protects the language model during the training phase of the language model. Specifically, see Figure 4 As shown, the protection method of this embodiment also includes:
[0063] S30: Scan the language model based on a preset list of unsafe components.
[0064] S31: Determine whether the language model contains an unsafe component in the unsafe component list. If so, jump to step S32; if not, jump to step S33.
[0065] S32: Discard unsafe components.
[0066] S33: Acquire several training data sets for training a language model, and perform training based on each training data set to obtain several language model instances corresponding to the training data sets.
[0067] S34: Determine whether the output information of each language model instance for the same request information is consistent. If so, jump to step S36; if not, jump to step S35.
[0068] S35: Discard the training data set corresponding to the language model instance whose output information is inconsistent with the output information of other language model instances.
[0069] S36: Based on the preset detection request information for detecting implanted backdoors, the output information of each language model instance corresponding to the detection request information is obtained.
[0070] S37: Based on the output information of the language model instance, determine whether the language model instance has an implanted backdoor. If so, jump to step S38; if not, jump to step S39.
[0071] S38: Discard the training data set corresponding to the language model instance with the implanted backdoor.
[0072] S39: retain the training data set corresponding to the language model.
[0073] In summary, the method of protecting the language model during the training phase can remove unsafe components during the formation phase of the language model, while preventing the language model from having biased information, implanting backdoors and other supply chain vulnerabilities, thereby further improving the security of the language model.
[0074] In another preferred embodiment of the present invention, a language model protection system is also disclosed, and the protection system works based on the above protection method.
[0075] The present invention also discloses a language model protection system, which includes one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the program includes instructions for executing the protection method as described above. The processor can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits to execute related programs to execute the protection method of the embodiment of the method of the present application.
[0076] The present invention also discloses a computer-readable storage medium, which includes a computer program, and the computer program can be executed by a processor to complete the protection method as described above. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media. The available medium can be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a magnetic disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.
[0077] The embodiment of the present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of an electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device performs the above detection method.
[0078] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the scope of the patent application of the present invention are still within the scope covered by the present invention.
Claims
1. A language model protection method, characterized in that: The protection method includes: Acquire request information sent to a language model and output information of the language model corresponding to the request information; Determining whether the output information of the language model contains sensitive information, and when the output information of the language model contains the sensitive information, performing desensitization processing on the sensitive information; Based on the code information in the output information of the language model, determine whether the code information is malicious code information. If so, convert the code information into plain text information. If not, directly output the output information of the language model.
2. The language model protection method according to claim 1, characterized in that: The method for determining whether the code information is malicious code information includes: A cross-site scripting vulnerability detection device is provided, which detects the code information in the output information of the language model. When the code information has a cross-site scripting vulnerability, the code information is marked as malicious code information and converted into plain text information.
3. The language model protection method according to claim 1, characterized in that: The method for determining whether the code information is malicious code information includes: It is determined whether the code information contains a remote execution code. When the code information contains a remote execution code, the code information is marked as malicious code information, and the code information is converted into plain text information.
4. The language model protection method according to claim 1, characterized in that: The protection method also includes: Obtain the frequency of request information sent by the IP address corresponding to the request information. When the frequency of request information sent by the IP address of the request information is greater than a preset normal frequency, mark the request information as a denial of service attack and add the IP address corresponding to the request information to a blacklist.
5. The language model protection method according to claim 4, characterized in that: The protection method also includes: Obtain a preset set of injection keywords, and when the request content of the request information contains any injection keyword in the set of injection keywords, mark the request information as a denial of service attack, and add the IP address corresponding to the request information to a blacklist; Determine whether the request content of the request information is consistent with the request content of the request information received by the language model within the most recent preset number of receptions; when the request content of the request information is consistent with the request content of all the request information received within the most recent preset number of receptions, mark the request information as a denial of service attack and add the IP address corresponding to the request information to a blacklist.
6. The language model protection method according to claim 1, characterized in that: The protection method also includes: The language model is scanned based on a preset unsafe component list, and when the language model contains an unsafe component in the unsafe component list, the unsafe component is discarded.
7. The language model protection method according to claim 6, characterized in that: The protection method also includes: Acquire several training data sets for training the language model, perform training based on each of the training data sets to obtain several language model instances corresponding to the training data sets, determine whether output information of each of the language model instances for the same request information is consistent, and when the output information of a certain language model instance is inconsistent with the output information of other language model instances, discard the training data set corresponding to the language model instance; Based on preset detection request information for detecting implanted backdoors, output information of each of the language model instances corresponding to the detection request information is obtained; based on the output information of each of the language model instances, it is judged whether the language model instance has an implanted backdoor; when a language model instance has an implanted backdoor, the training data set corresponding to the language model instance is discarded.
8. A language model protection system, characterized in that: The protection system works based on the protection method according to any one of claims 1 to 7.
9. A language model protection system, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs comprising instructions for executing the protection method according to claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The invention comprises a computer program which can be executed by a processor to implement the protection method according to any one of claims 1 to 7.