Security control method and device for model generated content, storage medium and equipment

Abstract

The embodiment of the specification discloses a kind of security control method, device, storage medium and equipment to model generation content, comprising: by the security control module of service application built-in in local terminal equipment obtains the original content generated by LLM and at least one text to be detected is obtained according to original content, information detector built-in in security control module is called to execute information detection to text to be detected, in response to information detection result meets security conversion trigger condition, original content is executed security conversion operation and obtains content to be shown, content to be shown is output to the user interface of service application.In the limited computing resources of terminal local, low delay, low power consumption pure offline content security review is realized, user privacy is protected and it is guaranteed that model output content meets security specification.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium and device for security control of model-generated content. Background Technology

[0002] Currently, while local large language models (LLMs) deployed on terminal devices offer advantages such as offline operation, low latency, and privacy protection, they also face unique security challenges. The content output of large models is entirely automatically generated, exhibiting characteristics such as uncontrollability, illusion, and potential harm; therefore, real-time review of the model's output is necessary.

[0003] However, in resource-constrained offline scenarios requiring low latency and privacy protection, it is difficult to use a dedicated content moderation model to detect the content output of a local large model, and it is impossible to effectively guarantee that the model output complies with security specifications. Summary of the Invention

[0004] This specification provides a method, apparatus, storage medium, and device for security control of model-generated content, in order to partially solve the problems existing in the prior art.

[0005] The embodiments in this specification adopt the following technical solutions: This specification provides a method for security control of model-generated content, executed by a security control module. This security control module is built into a local business application on the terminal device. The business application also includes a built-in Large Language Model (LLM). The method includes: Obtain the original content generated by the LLM, and obtain at least one text to be detected based on the original content; The information detector built into the security control module is invoked to perform information detection on the at least one text to be detected; wherein, the information detection includes at least one of: predefined privacy information detection, predefined illegal operation instruction detection, and predefined harmful information detection; In response to the information detection result meeting the security conversion trigger condition, a security conversion operation is performed on the original content to obtain the content to be displayed; The content to be displayed is output to the user interface of the business application.

[0006] This specification provides a security control device for model-generated content. The device is deployed within a security control module, which is built into a local business application on a terminal device. The business application also includes a Large Language Model (LLM). The device comprises: The content acquisition unit acquires the original content generated by the LLM and obtains at least one text to be detected based on the original content; The detection unit is invoked, and the information detector built into the security control module is invoked to perform information detection on the at least one text to be detected; wherein, the information detection includes at least one of: predefined privacy information detection, predefined illegal operation instruction detection, and predefined harmful information detection; The security conversion unit, in response to the information detection result meeting the security conversion trigger condition, performs a security conversion operation on the original content to obtain the content to be displayed; The content output unit outputs the content to be displayed to the user interface of the business application.

[0007] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for security control of model-generated content.

[0008] This specification provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned security control method for the generated model content.

[0009] This specification provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned security control method for the generated model content.

[0010] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: This specification discloses a method for security control of model-generated content. This method integrates a security control module within the local business application of the terminal device, performing detection after the LLM generates the original content and before it is output to the user interface. Because the security control module calls the built-in information detector to perform local lightweight detection such as predefined privacy information detection, predefined illegal operation instruction detection, and predefined harmful information detection, it does not rely on a large cloud-based model review architecture. Therefore, it achieves low-latency, low-power, purely offline content security review with limited local computing resources on the terminal, protecting user privacy while ensuring that the model-generated content displayed to the user complies with security standards. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the security control method for model-generated content provided in the embodiments of this specification; Figure 2This specification provides a schematic diagram of a security control process for model-generated content, as illustrated in an embodiment of the present specification. Figure 3 This is a schematic diagram of a security control device for model-generated content provided in the embodiments of this specification; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0013] The embodiments provided in this specification can include a security control module, which is built into the local business application of the terminal device. This business application also includes an LLM (Local Management Model). It should be understood that the terminal device can be a resource-constrained computing device such as a mobile phone, laptop, or edge intelligent device. The business application can be various applications integrating a local large model, such as intelligent dialogue, code generation, or office assistance applications. The security control module can be embedded into the business application in the form of a Software Development Kit (SDK), running as a purely edge-side, purely offline output fence, without cloud deployment. The security control module execution scheme provided in this embodiment can achieve real-time review of content generated by the local large model, with a fast response. It performs a security transformation on the original content generated by the model before outputting it to the user interface. The entire security control process is imperceptible to the user and does not affect the user's immediate experience of obtaining feedback through the business application. Furthermore, since the security control module uses a built-in lightweight information detector for information detection, it operates in low-power edge scenarios and does not significantly increase the operational burden of the business application or the device's power consumption.

[0014] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0015] Figure 1 This is a schematic diagram of a text-based business recommendation method provided in the embodiments of this specification, which specifically includes the following steps: Step S100: Obtain the original content generated by the LLM, and obtain at least one text to be detected based on the original content.

[0016] One implementation approach is to intercept the original LLM-generated content before it is output to the user interface of the business application, in response to the detection of such content. This interception mechanism is a prerequisite for the effectiveness of the edge-side security fence; the transmission channel must be cut off before the data stream reaches the user interface display layer. Otherwise, if the original content is rendered on the screen, it cannot be retracted even if a violation is subsequently discovered, thus missing the opportunity for security control. As an example, the interception operation can be implemented by rewriting the content distribution logic of the business application or by attaching an output stream listening hook.

[0017] After intercepting the original content, it is necessary to perform text normalization processing on the original content to obtain normalized text. "Text normalization processing" can be the process of converting the original content of LLM output, which may contain multimodal or unstructured characters, into structured encoded text (such as UTF-8 encoding) and performing word segmentation and vector mapping so that the detector can parse it later.

[0018] Because the security control module has limited memory and computing power for single processing, directly inputting excessively long text can lead to processing timeouts or memory overflows. Therefore, the text length corresponding to the standardized text can be determined; in response to the text length exceeding a preset length threshold, the standardized text is segmented to obtain multiple texts to be detected. The preset length threshold is determined based on the processing resource limitations of the security control module. Alternatively, in response to the text length not exceeding the preset length threshold, the standardized text can be treated as a single text to be detected.

[0019] In some embodiments, the standardized text can be segmented based on sentence boundaries, paragraph boundaries, and the preset length threshold, ensuring that the length of each text to be detected does not exceed the preset length threshold and conforms to a preset semantic coherence standard. For example, the preset length threshold can be set to 512 tokens. When the length of the standardized text exceeds this threshold, it is first segmented at the paragraph boundary. If a single paragraph still exceeds the limit, it is segmented again at the sentence boundary. This ensures that each text to be detected does not exceed the SDK's single processing capacity while preserving complete contextual semantics, avoiding semantic fragmentation caused by abrupt truncation and resulting in false detections.

[0020] Step S101: Invoke the information detector built into the security control module to perform information detection on the at least one text to be detected.

[0021] The information detection includes at least one of predefined privacy information detection, predefined violation operation instruction detection, and predefined harmful information detection.

[0022] In some embodiments, this embodiment deploys three lightweight information detectors in parallel within the edge security control module, instead of employing a large-scale model review architecture. The reason for using only these lightweight methods on the edge instead of a large-scale model review is that terminal devices have extremely limited computing resources, making it difficult to run two LLMs locally simultaneously. Furthermore, due to the high real-time requirements on the edge, it is also difficult to utilize a large model deployed in the cloud to review the content generated by the local large model on the edge in real time.

[0023] Therefore, one or more embodiments of this disclosure employ a lightweight detector to achieve the technical objective of local large model output fencing.

[0024] First, the security control module invokes its built-in privacy information detector to match predefined privacy information in the text to be detected. This privacy information detector incorporates regular expression matching and / or a named entity recognition model. Based on the matching results, it is determined whether the text to be detected contains predefined privacy information. Predefined privacy information can cover various sensitive data types, such as personal identification (e.g., ID card), contact information (e.g., mobile phone number), account credentials (e.g., bank card number, API key), network identifiers (e.g., IP address, MAC address), and location.

[0025] Regular expression matching can efficiently match privacy information with fixed format features (such as the 11-digit pattern of a mobile phone number), while named entity recognition (NER) models are usually lightweight and can extract privacy information that lacks a fixed format but has contextual semantic features (such as personal names and company names). The two can be used together to take the union of the detection results, or one can be used separately, thereby achieving a wide range of privacy information detection with low computing power consumption.

[0026] Secondly, the built-in violation instruction detector of the security control module is invoked to match the text to be detected with preset instruction detection rules. Based on the matching results, it is determined whether the text to be detected contains predefined violation instructions. The types of predefined violation instructions include procedural instructions and / or natural language instructions. Procedural instructions can be executable dangerous commands sent to the operating system (such as code keywords like "DROP TABLE" and "rm -rf" in shell scripts), while natural language instructions refer to text commands constructed using human language (such as dangerous verb combinations like "teaching you how to make dangerous items"). Through script keyword detection and rule matching, high-risk operation instructions generated by the model can be detected in near real-time.

[0027] Third, the text classification model built into the security control module is invoked to classify the text to be detected, obtaining the probability of the text corresponding to each predefined harmful category; based on the category probabilities, it is determined whether the text to be detected contains predefined harmful information. The "text classification model" is not a large language model with a large number of parameters, but a lightweight binary or multi-class classification model. Its output is a probability distribution across each predefined harmful category (such as violence, hatred, illegality, etc.), and the category with the highest probability is usually selected as the identified category. The lightweight model has extremely low inference latency on the edge, effectively filtering content with deep semantic harm and compensating for the shortcomings of pure rule matching in semantic understanding.

[0028] Step S102: In response to the information detection result meeting the security conversion trigger condition, a security conversion operation is performed on the original content to obtain the content to be displayed.

[0029] The security conversion trigger condition can be set as needed. Its function is to trigger the security conversion operation of the original content when the information detection result shows that there is a risk in the original content generated by the model, so that the content displayed to the user complies with security standards.

[0030] Security conversion is a means of handling risky content. This embodiment provides at least three operations: The first type is desensitization, used to hide at least one of the following in the original content: predefined privacy information, predefined illegal operation instructions, and predefined harmful information. As an example, desensitization can be used to mask privacy information, such as replacing a real IP address with "***" or an identity verification code with "XXXXXXXX". Its purpose is to sever the mapping relationship between sensitive data and real entities while preserving the overall semantic structure of the original content.

[0031] The second type is partial replacement operation, used to replace at least one of the following in the original content: predefined privacy information, predefined illegal operation instructions, and predefined harmful information, with preset safe text. As an example, partial replacement operation can be used to correct high-risk instructions. For instance, it can partially replace the dangerous procedural instruction "DROP TABLE" in the original content with the preset safe text "SELECT," or replace dangerous verb combinations with harmless verbs. This not only blocks the execution path of dangerous operations but also ensures the functional consistency and usability of the output content.

[0032] The third type is the overall replacement operation, which is used to replace the original content with a preset security response. As an example, the overall replacement operation can be used in harmful content scenarios. When the original content is filled with serious violations such as violence, and partial modifications cannot eliminate its harmfulness, the entire original output can be directly replaced with a preset security response (such as "Sorry, I cannot answer this question"), thereby blocking the spread of harmful information.

[0033] It should be understood that the above three security conversion operations can be flexibly combined according to the actual type of information detected, and are not limited to a single operation.

[0034] Step S103: Output the content to be displayed to the user interface of the business application.

[0035] After the above interception, detection and conversion process, the original content that originally contained risks can be converted into compliant content to be displayed. At this point, the content to be displayed can be safely released to the rendering layer of the user interface of the business application for users to view, thus completing the security control of the output content of the large model in a closed loop on the terminal side.

[0036] The above solution integrates a security control module into the local business application of the terminal device, performing detection after the LLM generates the original content and before it is output to the user interface. Because the security control module calls the built-in information detector to perform lightweight local detection such as predefined privacy information detection, predefined illegal operation command detection, and predefined harmful information detection, it does not rely on a large cloud-based model review architecture. This achieves low-latency, low-power, purely offline content security review within the limited local computing resources of the terminal, protecting user privacy while ensuring that the model-generated content displayed to the user complies with security standards.

[0037] Furthermore, in some embodiments, before invoking the information detector built into the security control module, in response to detecting that the original content belongs to structured data, it can check whether the original content conforms to the structured data format specification; if the detection result is negative, a regeneration instruction is sent to the LLM. If the detection result is positive, subsequent steps can continue.

[0038] It should be noted that the structured data mentioned above can be, for example, JSON, forms, code, etc., and specific structured data usually has specific format specifications.

[0039] As an example, the above-mentioned regeneration instructions may include relevant information indicating that the original content does not conform to the format specifications.

[0040] In addition, Figure 1 Based on the method and flow shown, this embodiment also provides a multi-dimensional decision fusion quantification mechanism and a dynamic update mechanism for detection rules for the security control module.

[0041] Regarding the decision fusion mechanism: Each predefined information type can be associated with a decision contribution weight; the information detection result includes one or more target predefined information types contained in the text to be detected, and the decision contribution weight associated with each target predefined information type. "Predefined information type" can be a unified definition of the three detection dimensions, covering predefined privacy information (such as personal identification information, contact information, account credentials, etc.), predefined illegal operation instructions (such as procedural instructions, natural language instructions), and predefined harmful information (such as violence, hatred, etc.). The assignment logic of the "decision contribution weight" can follow the principle of differentiated risk severity, that is, different types of information contribute differently to security threats, therefore the weight assignment should also be differentiated. For example, high-risk operation commands (such as procedural commands containing "DROP TABLE" or "rm -rf") will cause irreversible damage to the system once executed, and their decision contribution weight should be assigned a high value (e.g., 0.7); while the risk of leakage of predefined privacy information (such as ordinary mobile phone numbers) is serious but usually not immediately destructive, its decision contribution weight can be assigned a relatively low value (e.g., 0.3); the weight of predefined harmful information (such as hate speech) can be between the two (e.g., 0.5). This can avoid the problem of misjudgment caused by over-triggering risk control when a single dimension is hit.

[0042] In some embodiments, a decision score can be calculated based on the decision contribution weight associated with each predefined information type in the information detection results. For example, the decision score can be calculated using a weighted summation method. Assuming a piece of text to be detected hits k predefined information types after passing through three detectors, the calculation formula for the decision score S is as follows: S = Σ(Wi * Ci), where i ranges from 1 to k, Wi represents the decision contribution weight associated with the i-th predefined information type, and Ci represents the hit confidence of the i-th predefined information type (for example, the confidence of privacy regularization matching can be set to 1.0, and the category probability output by the micro-text classification model can be directly used as the confidence). This formula normalizes the multi-dimensional heterogeneous detection results into a quantifiable comprehensive score.

[0043] In some embodiments, a preset scoring threshold is a condition for triggering a security transition. A decision score can be calculated based on the decision contribution weight associated with each target predefined information type in the information detection result; in response to the decision score exceeding the preset scoring threshold, it is determined that the information detection result meets the security transition triggering condition.

[0044] In some embodiments, the business control module in a business application can issue threshold adjustment instructions to the security control module based on the risk tolerance of the current business scenario. For example, in financial business applications, the tolerance for privacy leaks is extremely low, and the preset scoring threshold can be dynamically lowered to 0.3, so that even a slight privacy violation can trigger a switch. In contrast, in entertainment business applications, the tolerance for harmful content is low, but the tolerance for instruction risk is relatively high, and the threshold can be raised to 0.6, thereby preventing normal conversations from being frequently blocked.

[0045] Regarding the dynamic update mechanism between the security control module and the business control module, this embodiment demonstrates the data interaction between the security control party and the business party within the business application data domain. It allows the injection and updating of custom data from the business party into various rule bases and model tags on the client side, improving the scalability and business adaptability of the solution.

[0046] In some embodiments, custom privacy types defined by the business user can be obtained from the business control module built into the business application to update the regular expression matching and / or named entity recognition model in the privacy information detector. For example, a business user (such as a local medical application) often has specific privacy concerns that differ from general scenarios (such as medical record numbers, prescription information, etc.). When the application starts or is updated, the business control module pushes a configuration file containing these custom privacy types to the security control module via local inter-process communication. After receiving the configuration file, the security control module parses it and automatically generates the corresponding regular expression matching, or injects these custom types as new tags into the recognition dictionary of the lightweight named entity recognition model, thereby enabling the privacy information detector to accurately identify sensitive data specific to the business, rather than being limited to preset privacy types.

[0047] In some embodiments, the business application's built-in business control module can be used to obtain custom violation operation types for updating the pre-defined instruction detection rules in the security control module. Similar to privacy type updates, different business applications may have different definitions of violations. For example, a native code compiler application may focus more on specific dangerous compilation instructions, while a smart home control application may focus more on dangerous natural language instructions for controlling hardware devices. The business control module sends the custom violation operation types and their corresponding instruction characteristics (such as specific shell keywords or dangerous verb combinations) to the security control module. The security control module then updates its built-in instruction detection rule library accordingly, enabling the violation operation instruction detector to effectively intercept high-risk instructions deeply bound to the specific logic of the business.

[0048] In some embodiments, custom harmful categories defined by the business user can be obtained from the business control module built into the business application to update the text classification model. The predefined harmful categories in the micro-text classification model are typically general (e.g., violence). However, in specific business contexts, it may be necessary to identify specific harmful content. For example, a children's education application may need to additionally identify the "negative values" category, while a financial consulting application may need to additionally identify the "investment fraud" category. The business control module sends the custom harmful category labels and a small amount of sample data to the security control module. The security controller uses this data to fine-tune or expand and update the labels of the micro-text classification model, ensuring that its classification output covers the business-specific harmful category probabilities, thereby achieving more accurate semantic risk control.

[0049] In some embodiments, custom-defined security text and / or custom-defined security response content can be obtained and stored from the business control module built into the business application. Businesses often want to display reassuring messages that align with their brand image or business logic when content is blocked or replaced, rather than generic rejection statements. Therefore, the business control module can send custom-defined security text (for partial replacement operations, such as replacing dangerous instructions with business-specific security prompt codes) and custom-defined security response content (for overall replacement operations, such as "According to the security specifications of XX application, I cannot provide this information") to the security control module. The security control module stores and updates these custom corpora in its local replacement corpus. When a security transition is triggered, these custom corpora from the business are prioritized for replacement, thus ensuring security while also respecting the business's interaction guidelines.

[0050] Furthermore, it should be understood that in actual large-scale model interaction scenarios, the vast majority of model outputs are normal and compliant, and the normal operating mode of the security fence is "no risk, then allow." Specifically, in response to the information detection result not meeting the security conversion trigger condition, the original content can be output to the user interface.

[0051] The triggering conditions for this access path can include two scenarios. The first scenario is when the decision score calculated based on the decision contribution weight associated with each predefined information type in the information detection results does not exceed a preset score threshold. This indicates that although there may be extremely slight sensitive traces in the original content (e.g., a word happens to match a low-weight privacy regular expression), its overall risk level is far from reaching the level requiring intervention. In this case, the information detection result is judged not to meet the security conversion triggering conditions. The second scenario is when the privacy information detector, the violation operation instruction detector, and the harmful information classification model all fail to detect any predefined information in the text to be detected. Naturally, the information detection result also does not meet the security conversion triggering conditions.

[0052] If any of the above triggering conditions are not met, the security control module will not perform any desensitization, partial replacement, or overall replacement conversion operations. Instead, it will directly output the original content generated by the LLM to the user interface of the business application.

[0053] also, Figure 2 This is a schematic diagram illustrating a security control process for model-generated content, provided as an embodiment of this specification. For example... Figure 2 As shown, the business application is built into the local terminal device, and it also includes an LLM and a security control module. The security control module intercepts the original content generated by the LLM before it is output to the business application's user interface upon detecting it.

[0054] After intercepting the original content, the security control module can determine whether the original content belongs to structured data. If the original content belongs to structured data, it checks whether the original content conforms to the structured data format specification; if the detection result is no, it sends a regeneration instruction to the LLM; if the detection result is yes, it obtains at least one text to be detected based on the original content. If the original content does not belong to structured data, the security control module can directly obtain at least one text to be detected based on the original content.

[0055] Subsequently, the security control module invokes its built-in information detector to perform information detection on at least one text to be detected, and obtains the information detection result. The information detection may include at least one of the following: predefined privacy information detection, predefined violation operation instruction detection, and predefined harmful information detection.

[0056] The security control module determines whether the information detection result meets the security conversion trigger conditions. If the information detection result meets the security conversion trigger conditions, a security conversion operation is performed on the original content to obtain the content to be displayed; if the information detection result does not meet the security conversion trigger conditions, the original content is used as the content to be displayed. Finally, the security control module outputs the content to be displayed to the user interface of the business application.

[0057] The above is an embodiment of a method for security control of model-generated content provided in this specification. Based on the same idea, this specification also provides corresponding devices, storage media and electronic devices.

[0058] Figure 3 This is a schematic diagram of a security control method device for model-generated content provided in the embodiments of this specification. The device is deployed in a security control module, which is built into a local business application of the terminal device. The business application also has a built-in Large Language Model (LLM). The device includes: Content acquisition unit 301 acquires the original content generated by the LLM and obtains at least one text to be detected based on the original content; The detection unit 302 is invoked to call the information detector built into the security control module to perform information detection on the at least one text to be detected; wherein, the information detection includes at least one of: predefined privacy information detection, predefined illegal operation instruction detection, and predefined harmful information detection; The security conversion unit 303, in response to the information detection result meeting the security conversion trigger condition, performs a security conversion operation on the original content to obtain the content to be displayed; The content output unit 304 outputs the content to be displayed to the user interface of the business application.

[0059] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform embodiments of this specification.

[0060] This specification also provides a computer program product comprising a computer program that, when executed by a processor, implements the embodiments of this specification.

[0061] The embodiments in this specification also provide Figure 4 The diagram shows the structure of the electronic device. Figure 4 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile storage, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile storage into memory and then runs it to implement the text-based business recommendation method described above.

[0062] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for security control of model-generated content, executed by a security control module, wherein the security control module is built into a local business application of a terminal device, and the business application also has a built-in Large Language Model (LLM), the method comprising: Obtain the original content generated by the LLM, and obtain at least one text to be detected based on the original content; The information detector built into the security control module is invoked to perform information detection on the at least one text to be detected; wherein, the information detection includes at least one of: predefined privacy information detection, predefined illegal operation instruction detection, and predefined harmful information detection; In response to the information detection result meeting the security conversion trigger condition, a security conversion operation is performed on the original content to obtain the content to be displayed; The content to be displayed is output to the user interface of the business application.

2. The method as described in claim 1, wherein at least one text to be detected is obtained based on the original content, specifically including: The original content is subjected to text standardization processing to obtain standardized text; Determine the text length corresponding to the standardized text; In response to the text length exceeding a preset length threshold, the standardized text is segmented to obtain multiple texts to be detected; wherein, the preset length threshold is determined based on the processing resource limitations of the security control module; In response to the fact that the text length does not exceed the preset length threshold, the standardized text is treated as a text to be detected.

3. The method as described in claim 1, wherein the information detector built into the security control module is invoked to perform information detection on the at least one text to be detected, specifically including: The privacy information detector built into the security control module is invoked to match predefined privacy information in the text to be detected; wherein, the privacy information detector has built-in regular expression matching and / or, the privacy information detector has built-in named entity recognition model; Based on the matching results, it is determined whether the text to be detected contains predefined privacy information.

4. The method of claim 3, further comprising: The privacy type defined by the business party is obtained from the business control module built into the business application, so as to update the regular expression matching expression and / or named entity recognition model in the privacy information detector.

5. The method as described in claim 1, wherein the information detector built into the security control module is invoked to perform information detection on the at least one text to be detected, specifically including: The security control module's built-in violation operation command detector is invoked to match the text to be detected with preset command detection rules; Based on the matching results, determine whether the text to be detected contains predefined illegal operation instructions; The predefined violation operation instructions include procedural instructions and / or natural language instructions.

6. The method of claim 5, further comprising: The system obtains the business user's custom violation operation type from the business control module built into the business application, so as to update the pre-set instruction detection rules in the security control module.

7. The method as described in claim 1, wherein the information detector built into the security control module is invoked to perform information detection on the at least one text to be detected, specifically including: The text classification model built into the security control module is invoked to classify the text to be detected, thereby obtaining the probability of the text to be detected corresponding to each predefined harmful category; Based on the category probability, it is determined whether the text to be detected contains predefined harmful information.

8. The method of claim 1, wherein, Each predefined information type is associated with a decision contribution weight; the information detection result includes one or more target predefined information types contained in the text to be detected, and the decision contribution weight associated with each target predefined information type; The method further includes: Calculate the decision score based on the decision contribution weight associated with each predefined information type of the target in the information detection results; In response to the decision score exceeding a preset score threshold, it is determined that the information detection result meets the security conversion trigger condition.

9. The method as described in claim 1, wherein the security conversion operation includes at least one of a desensitization operation, a partial replacement operation, and a global replacement operation; in, The desensitization operation is used to hide at least one of the predefined privacy information, predefined illegal operation instructions, and predefined harmful information in the original content. The partial replacement operation is used to replace at least one of the predefined privacy information, predefined illegal operation instructions, and predefined harmful information in the original content with preset safe text. The overall replacement operation is used to replace the original content with preset security response content.

10. The method of claim 9, further comprising: The application obtains and stores the business-defined preset security text and / or preset security response content from the business control module.

11. The method of claim 1, further comprising, before invoking the information detector built into the security control module: In response to the detection that the original content belongs to structured data, the system checks whether the original content conforms to the structured data format specification; If the detection result is negative, a regeneration command is sent to the LLM.

12. A security control device for model-generated content, the device being deployed within a security control module, the security control module being built into a local business application of a terminal device, the business application also having a Large Language Model (LLM) built into it, the device comprising: The content acquisition unit acquires the original content generated by the LLM and obtains at least one text to be detected based on the original content; The detection unit is invoked, and the information detector built into the security control module is invoked to perform information detection on the at least one text to be detected; wherein, the information detection includes at least one of: predefined privacy information detection, predefined illegal operation instruction detection, and predefined harmful information detection; The security conversion unit, in response to the information detection result meeting the security conversion trigger condition, performs a security conversion operation on the original content to obtain the content to be displayed; The content output unit outputs the content to be displayed to the user interface of the business application.

13. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1-11.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method described in any one of claims 1-11.

Images