A message processing method and electronic device

Abstract

The application discloses a message processing method and an electronic device. The method comprises the following steps: obtaining a to-be-processed message; determining a message source of the to-be-processed message; generating prompt information according to the to-be-processed message and the corresponding message source; processing the prompt information based on a security policy through a target processing model to determine a processing result corresponding to the to-be-processed message, wherein the to-be-processed messages of different message sources correspond to different security policies.

Description

Technical Field

[0001] This application relates to the field of message processing technology, and in particular to a message processing method and an electronic device. Background Technology

[0002] With the development of artificial intelligence technology, many electronic devices have deployed models with message processing capabilities. However, these models often mix user input with system information automatically injected by third-party platforms, which lacks a reliable differentiation mechanism based on the source of the prompt and a dynamic permission control mechanism, leading to security risks. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] The first aspect of this application provides a message processing method, including:

[0005] Receive pending messages;

[0006] Determine the message source of the message to be processed;

[0007] Generate prompt information based on the message to be processed and its corresponding message source;

[0008] The prompt information is processed by the target processing model based on the security policy to determine the processing result corresponding to the message to be processed. The message to be processed from different message sources corresponds to different security policies.

[0009] Optionally, generating prompt information based on the message to be processed and the corresponding message source includes:

[0010] Obtain a source tag representing the source of each of the multiple messages to be processed;

[0011] A prompt message is generated based on the source tag and multiple messages to be processed. The source tag is used to divide the prompt message into information partitions corresponding to the message source, and the messages to be processed are located in the information partitions corresponding to the message source.

[0012] Optionally, the step of processing the prompt information based on a security policy using a target processing model to determine the processing result corresponding to the message to be processed includes:

[0013] Determine the security level of the message source of the message to be processed;

[0014] Obtain the security policy corresponding to the security level, wherein different security policies correspond to different operation permissions for the target processing model;

[0015] The target processing model processes the prompt information based on the operation permissions corresponding to the security policy, and determines the processing result corresponding to the message to be processed.

[0016] Optionally, the step of processing the prompt information based on the operation permissions corresponding to the security policy through the target processing model and determining the processing result corresponding to the message to be processed includes:

[0017] The target processing model is used to reason about the prompt information to obtain the operation instructions corresponding to the message to be processed.

[0018] Based on the operation instructions and the corresponding operation permissions of the security policy, the operation instructions are processed, and the processing result corresponding to the message to be processed is determined.

[0019] Optionally, the step of processing the operation instruction according to the operation permission corresponding to the operation instruction and the security policy, and determining the processing result corresponding to the message to be processed, includes:

[0020] Based on the operation permissions corresponding to the security policy, a target tool that meets the operation permissions is determined from the candidate tools corresponding to the operation instructions;

[0021] The operation instructions are executed based on the target tool to determine the processing result corresponding to the message to be processed.

[0022] Optionally, the step of processing the prompt information based on a security policy using a target processing model to determine the processing result corresponding to the message to be processed includes:

[0023] The usage method corresponding to the message source of the message to be processed is determined according to the security policy;

[0024] If the usage method indicates that the message to be processed cannot be used in the reasoning process, filter out the messages to be processed that are not used for reasoning from the prompt information to obtain the filtered prompt information;

[0025] The filtered prompt information is processed based on the security policy using the target processing model to determine the processing result corresponding to the message to be processed.

[0026] Optionally, determining the message source of the message to be processed includes at least one of the following:

[0027] Parse at least one of the message header field, event type field, sender identifier field, and system event identifier field of the message to be processed to determine the message source of the message to be processed;

[0028] The data structure of the message to be processed is detected, and the source of the message is determined based on the data structure.

[0029] Optionally, generating prompt information based on the message to be processed and the corresponding message source includes:

[0030] The message to be processed is subjected to security detection according to the detection rules corresponding to the message source of the message to be processed;

[0031] In response to the pending message passing the security check, a prompt message is generated based on the pending message and the corresponding message source.

[0032] Optional, also includes:

[0033] In response to the pending message failing the security check, a detection failure message is output, which includes at least one of the following: the pending message, the corresponding message source, and the reason for failing the security check.

[0034] A second aspect of this application also provides an electronic device, including a memory and a processor;

[0035] The memory is used to store computer programs;

[0036] The processor is used to execute the computer program to perform:

[0037] Receive pending messages;

[0038] Determine the message source of the message to be processed;

[0039] Generate prompt information based on the message to be processed and its corresponding message source;

[0040] The prompt information is processed by the target processing model based on the security policy to determine the processing result corresponding to the message to be processed. The message to be processed from different message sources corresponds to different security policies. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This is a flowchart of a message processing method provided in an embodiment of this application;

[0043] Figure 2 This is a flowchart of a method for generating prompt information provided in an embodiment of this application;

[0044] Figure 3 This is a flowchart of a method for obtaining processing results based on a security policy, provided in an embodiment of this application.

[0045] Figure 4 This is a flowchart illustrating a method for processing prompt information based on operation permissions, as provided in an embodiment of this application.

[0046] Figure 5 This is a flowchart of another method for obtaining processing results based on a security policy, provided in an embodiment of this application.

[0047] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] The executing entity of the message processing method in any embodiment of this application can be an electronic device with data processing capabilities, capable of automatically processing messages using a model, such as a smartphone, laptop, or intelligent robot. The electronic device may be equipped with one or more intelligent agents, through which interaction between the electronic device and the user is realized, and through which the intelligent agents invoke the model to process messages.

[0050] An agent is an application of artificial intelligence technology. It can be implemented based on a neural network model with reasoning capabilities (such as a large language model). The agent's behavior (or output) is determined by the model based on the current state and external inputs, and it can be used to specifically solve a certain type of problem. The agent's model provides the basis for decision-making through learning and training on large amounts of data; the agent can also call upon tools, plugins, and knowledge bases to provide reasoning, decision-making, and execution capabilities. Agents can be installed on electronic devices by embedding them in applications (such as browsers, app stores, etc.) and / or in the operating system, allowing the application or operating system to invoke the agent to provide intelligent functions in various ways.

[0051] With the development of Large Language Models (LLMs), more and more systems are adopting agent technology to realize intelligent interaction between electronic devices and users. However, there are certain security risks in the process of the model processing messages. These security risks are usually caused by potential attackers who construct special messages to be processed and input them into the model. Especially in agents with the ability to call software tools, these security risks are further amplified. Some common security risks include, but are not limited to: calling unauthorized software tools, reading sensitive information, and executing unexpected operation instructions.

[0052] Related technologies typically mitigate these security risks through keyword filtering. The problem with this approach is that keyword filtering can sometimes block compliant information actively entered by the user, causing the model to fail to meet the user's needs. Furthermore, keyword filtering is generally based on a static keyword library, making it easily circumvented.

[0053] This embodiment provides a message processing method; please refer to [link to relevant documentation]. Figure 1 The method may include the following steps.

[0054] S101, Receive message to be processed.

[0055] S102, Determine the message source of the message to be processed.

[0056] S103, Generate prompt information based on the message to be processed and the corresponding message source.

[0057] S104, the target processing model processes the prompt information based on the security policy, determines the processing result corresponding to the message to be processed, and the message to be processed from different sources corresponds to different security policies.

[0058] The target processing model can be any deep learning model with reasoning capabilities, capable of processing input information based on reasoning and generating corresponding processing results. For example, it can be any large language model in related technologies. The target processing model can be invoked by the intelligent agent of an electronic device to process information and can output operation instructions to invoke various software tools based on the operation instructions, such as invoking an email tool to send an email.

[0059] The pending message in this scheme can be obtained by the intelligent agent (also known as the intelligent agent or agent) that invokes the target processing model, and is used to input the target processing model for inference. The pending message can be obtained as follows: the user inputs an initial message on a third-party communication platform (such as an enterprise instant messaging tool or social platform). The third-party communication platform transmits the initial message to the intelligent agent through a platform interface or software development kit (SDK). The intelligent agent combines the message from the third-party communication platform with a preset system prompt and historical context information to form the pending message. That is, the transmission path of the pending message is: user - third-party communication platform - intelligent agent - target processing model.

[0060] Given the aforementioned transmission path, the messages to be processed may contain initial information input by the user, as well as additional messages added by third-party communication platforms, such as system messages from third-party platforms, robot identifiers, context of the user's group chat, user identity tags, platform template content, etc. These messages come from various sources, some reliable and some unreliable. Directly inputting these messages into the target processing model could easily lead to security risks.

[0061] In this embodiment, when processing messages through the target processing model, prompt information reflecting the message source is provided to the target processing model. As a result, the target processing model can process the message to be processed according to a security policy that matches the message source. For example, a stricter security policy can be used to process messages from sources with low credibility, thereby reducing the security risks in the process of the target processing model automatically processing messages.

[0062] Furthermore, compared to conventional keyword filtering solutions, this solution can distinguish the source of the message to be processed. For message sources with high credibility and low potential security risks, a more lenient security policy can be applied, thereby meeting the relevant needs of users and partially solving the problem that keyword filtering technology cannot distinguish between users' real needs and potential attack behaviors. Moreover, every message has its inherent source, so the security mechanism of this solution based on the message source is less likely to be bypassed and has better protection performance than keyword filtering solutions.

[0063] Optionally, the method for determining the message source of the message to be processed may include at least one of the following:

[0064] 1. Parse at least one of the message header field, event type field, sender identifier field, and system event identifier field of the message to be processed to determine the message source of the message to be processed;

[0065] 2. Detect the data structure of the message to be processed, and determine the source of the message based on the data structure.

[0066] The pending messages can contain several messages from one or more different message sources. That is, each time a pending message is obtained in S101, there can be one or more pending messages. The message source of each message can be, for example, any one of the following: user input, historical dialogue, third-party platform prompts, third-party platform metadata, group chat context, third-party agent generation, and agent prompts. The above message sources are just examples, and other message sources can also be used depending on the actual situation. There are no restrictions.

[0067] Agent prompts refer to messages that are preset prompts within the agent that invokes the target processing model. Third-party agent generation refers to messages generated by a third-party agent using its inference model; a third-party agent refers to an agent that is different from the agent belonging to the target processing model of this solution.

[0068] In the first method for determining the source of a message, the message source can be determined by parsing the message header field, event type field, sender identifier field, and system event identifier field of the received message to be processed, based on the pre-agreed communication protocol between the agent and the third-party communication platform.

[0069] For example, if the sender identifier field indicates that the sender of a message is a user using an agent, then the message source can be determined to be user input; if the sender identifier field indicates that the sender of a message is a group chat to which the user belongs, then the message source can be determined to be the group chat context; if the event type field or system event identifier field indicates that a message is generated based on a specific event of a third-party communication platform, then the message source can be determined to be a prompt from a third-party platform; if the message header field indicates that the corresponding message belongs to metadata (such as system time, geographical location, etc.), then the message source can be determined to be metadata from a third-party platform.

[0070] In the second method for determining the message source, some third-party communication platforms fill the messages to be transmitted to the agent into a preset message template before sending them. These messages to be processed generally have a fixed structure corresponding to their message templates. Therefore, the message templates used by the third-party communication platform can be obtained in advance, and then the data structure of the message to be processed can be checked to see if it matches these message templates, thereby determining the message source. For example, if a message to be processed is formed by piecing together a dialogue template, then the message source of the message to be processed can be determined to be a group chat context.

[0071] Alternatively, the following methods can be used to identify the source of a message.

[0072] Some third-party communication platforms will mix all messages into a single string. In this case, the messages in each part of the string can be compared with the historical dialogues, system templates, and event information stored locally by the agent. This will determine whether the message source of each message to be processed in the string is a historical dialogue or a prompt from a third-party platform.

[0073] Alternatively, a reasoning model with a small number of parameters (such as a large language model) can be pre-configured and used to reason about the obtained messages to be processed in order to determine the corresponding message source.

[0074] The above methods for determining the source of information can be used simultaneously in one or more ways, without limitation. For example, the first method for determining the source of information can be combined with a method based on inference models to jointly determine the source of information.

[0075] Using the methods described above, this solution can accurately identify the source of the message to be processed, providing a basis for selecting appropriate security strategies in subsequent steps. This not only improves security but also avoids overly strict security strategies that could negatively impact the user experience.

[0076] The method for generating the prompt message in S103 is not limited, as long as the prompt message can contain the message to be processed and can indicate the source of the message to be processed.

[0077] One alternative method is to sequentially concatenate all pending messages into a single prompt message, and insert a source tag indicating the source of each pending message at the end of the prompt message. The resulting combination of pending message and source tag serves as the prompt message. For example, the prompt message generated in this way could be in the form of: pending message 1, source tag 1; pending message 2, source tag 2... pending message X, source tag X.

[0078] Using this method, you only need to concatenate the message to be processed and the source tag to obtain the prompt information, which is highly efficient in generating prompt information.

[0079] See Figure 2 Another alternative method for generating prompt information based on the message to be processed and the corresponding message source may include the following steps.

[0080] S201, Obtain a source label representing the source of a message based on the message source of multiple pending messages.

[0081] S202, generate prompt information based on source tags and multiple pending messages. The source tags are used to divide the prompt information into information partitions corresponding to the message sources, and the pending messages are located in the information partitions corresponding to the message sources.

[0082] Optionally, a corresponding source label can be preset for each possible message source, and the corresponding source label can be read according to the identified message source when executing S201. Alternatively, the reasoning capability of the target processing model can be utilized to generate a corresponding source label based on the characteristics of each message source.

[0083] As an example, the source tags corresponding to some message sources can be represented by Table 1 below.

[0084] Table 1

[0085] Source Source tags User input User Third-party platform notification Platform Agent prompts System Group chat context Context

[0086] Based on the above source tags, a notification template containing multiple information sections can be created. Each information section corresponds to a message source, and different information sections are distinguished by their corresponding source tags. The resulting notification template can be represented as follows:

[0087] System:

[0088] ...;

[0089] Platform:

[0090] ...;

[0091] User:

[0092] ...

[0093] Among them, “System”, “Platform” and “User” are source labels that separate information partitions. The area between “System” and “Platform”, the area between “Platform” and “User”, and the area after “User” are the information partitions corresponding to the intelligent agent prompts, the information partitions corresponding to the third-party platform prompts, and the information partitions corresponding to the user input, respectively.

[0094] Based on the predefined information partitions, each message to be processed can be filled into its corresponding information partition according to its message source, thus obtaining the prompt information in S202. As an example, the obtained prompt information can be represented as follows:

[0095] System:

[0096] Intelligent agent system configuration prompts;

[0097] Platform:

[0098] Third-party auto-completion;

[0099] User:

[0100] User input.

[0101] The advantage of obtaining prompt information using the above method is that information partitioning can accurately represent the message source of the message to be processed, and can avoid inserting too many source tags into the prompt information, reducing the number of characters in the prompt information, thereby improving the reasoning efficiency of the target processing model when reasoning about the prompt information.

[0102] Optional, see Figure 3 The process of processing prompt information based on security policies using a target processing model and determining the processing result corresponding to the message to be processed may include the following steps.

[0103] S301, Determine the security level of the message source to be processed.

[0104] S302, obtain the security policy corresponding to the security level, where different security policies correspond to different operation permissions for the target processing model.

[0105] S303, based on the operation permissions corresponding to the security policy, the target processing model processes the prompt information and determines the processing result of the message to be processed.

[0106] The beneficial effect of this embodiment is that by processing the prompt information according to the operation permissions matched with the message source of the message to be processed, the operation that the target processing model may perform during the message processing process can be restricted while meeting the user's needs as much as possible. This avoids the target processing model from performing some high-privilege operations and causing security risks, thereby improving the security of electronic devices.

[0107] Security levels can include, for example, three categories: Trusted, Semi-Trusted, and Untrusted. Each message source can belong to any of these three security levels. As examples, a third-party platform might indicate an untrusted source, a user's input might indicate a semi-trusted source, and an agent might indicate a trusted source.

[0108] The correspondence between message sources and security levels can be pre-configured and dynamically adjusted. For example, if it is detected that the agent may be under network attack, the security level of the agent can be temporarily changed from trusted to partially trusted, and then restored to trusted after the potential network attack is ruled out. If the third-party communication platform currently connected to the agent is compliant and has its own security detection capabilities, the security level of the third-party platform can be changed to partially trusted.

[0109] To implement the method of this embodiment, a corresponding security policy can be pre-configured for each security level. Each security policy specifies the operation permissions of the target processing model when operating electronic devices (such as calling tools, reading and writing files, modifying system configuration, and installing / uninstalling software). The operation permissions of different security policies are different. Generally, the higher the security level, the higher the operation permissions of the corresponding security policy and the more operations that can be performed. Conversely, the lower the security level, the lower the operation permissions and the fewer operations that can be performed.

[0110] In addition, security policies can also include how the messages to be processed are used, i.e., whether the messages are used in the inference process of the target processing model. In other words, security policies can include the operation permissions corresponding to operation instructions and / or how the messages to be processed are used.

[0111] As an example, in the aforementioned security levels, the operation permissions of the security policy corresponding to trusted can be to allow all high-privilege operations, i.e., allow all operations; the operation permissions of the security policy corresponding to partially trusted can be to allow some high-privilege operations and prohibit the execution of other high-privilege operations, i.e. restrict some operations; and the operation permissions of the security policy corresponding to untrusted can be to prohibit high-privilege operations.

[0112] High-privilege operations are a predefined set of operations whose content can be set according to actual needs and is not limited. For example, high-privilege operations may include one or more of the following: reading and writing sensitive user information, accessing specific network addresses, installing or uninstalling applications, downloading files from the network, and invoking specific software tools. Electronic devices can be configured by the user to determine which operations are high-privilege and which are not.

[0113] In the security policy of restricting certain high-privilege operations, the specific high-privilege operations that are allowed or prohibited can be determined based on information such as the user's identity and the network environment of the electronic device. For example, if the user is currently logged into the administrator account of the electronic device, the security policy can allow the installation or uninstallation of applications; if the user is logged into a guest account, the security policy can prohibit the installation or uninstallation of applications. If the electronic device is connected to a public network with poor security, the security policy can prohibit the downloading of files from the network; if it is connected to a more secure corporate intranet, the security policy can allow the downloading of files from the network.

[0114] Optional, see Figure 4 The target processing model processes the prompt information based on the operation permissions corresponding to the security policy, and determines the processing result of the message to be processed, which may include:

[0115] S401, the prompt information is reasoned through the target processing model to obtain the operation instructions for the corresponding message to be processed.

[0116] S402: Based on the operation permissions corresponding to the operation instructions and security policies, process the operation instructions and determine the processing result corresponding to the message to be processed.

[0117] The target processing model can be any neural network model with reasoning capabilities (such as any large language model). The method for the target processing model to obtain operation instructions based on prompt information can be found in relevant technologies, which will not be elaborated here.

[0118] In order to accurately manage operation permissions according to different message sources, when an operation instruction is obtained in S401, the target processing model can also determine which pending messages these operation instructions were inferred from, that is, determine which pending messages the operation instructions originated from.

[0119] Based on this, when processing an operation instruction according to S402, the operation authority can correspond to the pending message from which the operation instruction originates. In other words, the operation authority applicable to an operation instruction is related to the message source from which the operation instruction is derived.

[0120] For example, in S401, two operation instructions are inferred, denoted as operation instruction 1 and operation instruction 2. Operation instruction 1 is inferred from the pending message in the aforementioned user-input information partition. That is, operation instruction 1 originates from the pending message input by the user. Combining the aforementioned correspondence between message source and operation permission, the way to process operation instruction 1 is to process operation instruction 1 according to the operation permission that allows high-privilege operation. Operation instruction 2 originates from the pending message prompted by the third-party platform. Therefore, the way to process operation instruction 2 can be to process operation instruction 2 according to the operation permission that prohibits high-privilege operation.

[0121] Optionally, if an operation instruction is derived from multiple pending messages, and each of these pending messages corresponds to a different operation permission, then the lowest operation permission can be applied to this operation instruction. For example, if operation instruction 3 is derived from two pending messages, one of which corresponds to restricting the operation permission of some high-privilege operations, and the other corresponds to allowing the execution of high-privilege operations, then operation instruction 3 can be processed according to the operation permission of restricting the operation permission of some high-privilege operations.

[0122] The advantage of processing operation commands in the above way is that appropriate operation permissions can be applied to process operation commands based on the different messages to be processed from the source of the operation command. For operation commands from relatively secure sources, such as those derived from user input messages, higher operation permissions can be applied to process them, thereby improving the convenience for users. For operation commands from less secure sources, lower operation permissions can be applied to process them, thereby improving the security of electronic devices.

[0123] S402 can be implemented in any one or more embodiments. One embodiment may include:

[0124] For each operation instruction, determine whether the operation corresponding to the instruction is within the scope of its applicable operation permissions. If it is within the scope, execute the instruction directly and use the result as the processing result. If it is not within the scope, refuse to execute the instruction and output the message "refuse to execute instruction" as the processing result.

[0125] For example, when processing operation instruction 1 based on the operation permission that restricts certain operations, the operation to be performed by operation instruction 1 is a high-privilege operation that is allowed to be executed. That is, the operation corresponding to operation instruction 1 is within the scope allowed by the applicable operation permission, so operation instruction 1 is executed directly and the processing result is obtained. When processing operation instruction 2 based on the operation permission that prohibits high-privilege operations, the operation to be performed by operation instruction 2 is a high-privilege operation. This operation is not within the scope allowed by the applicable operation permission for operation instruction 2, so operation instruction 2 is refused to be executed, and the information of executing operation instruction 2 is output as the processing result.

[0126] Another implementation of S402 may include:

[0127] Based on the operation permissions corresponding to the security policy, determine the target tool that meets the operation permissions from the candidate tools corresponding to the operation instructions;

[0128] The operation instructions are executed based on the target tool to determine the processing result corresponding to the message to be processed.

[0129] In this embodiment, the electronic device can pre-assign various callable software tools to the aforementioned operation permissions. For example, for operation permissions that allow all operations, all callable software tools on the electronic device can be called; for operation permissions that restrict some operations, only a portion of the software tools can be called; and for operation permissions that prohibit high-privilege operations, even fewer software tools can be called.

[0130] As an example, for operation permissions that allow all operations, the search tool, file reading tool, and system command execution tool can be invoked; for operation permissions that restrict some operations, only the search tool and file reading tool can be invoked; for operation permissions that prohibit high-privilege operations, only the search tool can be invoked.

[0131] When processing an operation command, the candidate tools for that operation command can be determined first. The candidate tools for an operation command refer to the software tools among the various software tools of the electronic device that can be used to implement the operation command. For example, if an operation command is to display an image, this operation command can be implemented using an image browsing tool or an image editing tool. Therefore, the candidate tools for this operation command include both image browsing tools and image editing tools.

[0132] For each operation instruction, the target processing model can be used to infer which candidate tools it corresponds to. The inference method is described in the relevant technology and will not be repeated here.

[0133] After identifying candidate tools, the target tool that matches the operation permissions corresponding to the operation instructions can be selected from the candidate tools based on the permission division of the software tools.

[0134] Methods for determining target tools can include:

[0135] If only one of the candidate tools meets the operation permissions corresponding to the operation command, then this candidate tool will be selected as the target tool.

[0136] If multiple candidate tools meet the operation permissions corresponding to the operation instructions, the one with the best performance among the candidate tools that meet the operation permissions is selected as the target tool.

[0137] Optimal performance is equivalent to achieving the highest expected execution efficiency when using this tool to execute the operation command. The performance of a software tool in executing operation commands can be determined by the average time taken to execute similar operation commands using the same software tool in the past; the longer the average time, the worse the performance.

[0138] As an example, suppose operation command 1 is an instruction to obtain a certain device parameter of an electronic device. Operation command 1 can be achieved by searching for the device parameters of similar devices on the network using a search tool, by reading the local configuration file of the electronic device using a file reading tool, or by directly executing the system command that outputs the device parameter using a system command execution tool. That is, the candidate tools for operation command 1 include search tools, file reading tools, and system command execution tools.

[0139] If the applicable operation permission for operation instruction 1 is to prohibit high-privilege operations, then only the search tool among these candidate tools meets the permission requirements. Therefore, the search tool is selected as the target tool, and operation instruction 1 is executed by the search tool to obtain the processing result. If the applicable operation permission for operation instruction 1 is to allow all operations, then all three tools meet the operation permission requirements. In this case, the performance of the system command execution tool is determined to be optimal through evaluation. Therefore, the system command execution tool is selected as the target tool, and operation instruction 1 is executed by the system command execution tool to obtain the processing result.

[0140] Optionally, if when processing an operation command, none of the candidate tools corresponding to the operation command meet its corresponding operation permissions, then the operation command is rejected, and the message "Rejected" is output as the processing result.

[0141] The advantage of processing operation instructions in the above manner is that when executing operation instructions, the permissions and performance of the tool used for execution are taken into account. Within the scope of permissions, the tool with the best performance is selected first to execute the operation instructions, which satisfies the needs of security and can maximize the working efficiency of electronic devices.

[0142] Optionally, in the aforementioned embodiments, the scheme of rejecting the operation command due to exceeding the operating permissions can also be changed to requiring user confirmation or executing it in a sandbox environment.

[0143] The method for requiring user confirmation is to display a prompt box, which shows the operation instructions to be performed and the potential security risks, and displays two controls: allow execution and deny execution. Depending on the user's selection of either control, the corresponding operation instruction can be allowed or denied.

[0144] The execution method in a sandbox environment refers to creating a sandbox environment in an electronic device. The sandbox environment can be regarded as a virtual machine isolated from the operating system of the electronic device. Then, the software tools required to execute operation instructions are run in this sandbox environment, and the operation instructions are executed based on these software tools to obtain the processing results.

[0145] Optional, see Figure 5 The target processing model processes the prompt information based on security policies to determine the processing result corresponding to the message to be processed, which may also include:

[0146] S501, determine the usage method corresponding to the message source of the message to be processed according to the security policy.

[0147] S502, if the usage mode indicates that the message to be processed cannot be used in the reasoning process, filter out the messages to be processed that are not used for reasoning in the prompt information to obtain the filtered prompt information.

[0148] S503 uses the target processing model to process the filtered prompt information based on the security policy and determines the processing result corresponding to the message to be processed.

[0149] The usage method corresponding to a security policy can be any of the following: "used in the inference process," "filtered and used for inference," or "not used in the inference process." Referring to the examples of the aforementioned security levels, the security policy corresponding to a trusted level can be used "for the inference process"; the security policy corresponding to a partially trusted level can be used "filtered and used for inference"; and the security policy corresponding to an untrusted level can be used "not used in the inference process." The usage method corresponding to each security level can be pre-configured in the electronic device or adjusted by the user as needed.

[0150] "Used for inference process" means that the corresponding message to be processed can be directly input into the target processing model for inference. "Filtered for inference" means that the corresponding message to be processed can be input into the target processing model for inference after being filtered by sensitive words. "Not used for inference process" means that the corresponding message to be processed may pollute the target processing model and reduce the accuracy of the target processing model, so it should not be input into the target processing model for inference.

[0151] In S502, pending messages "not used in the reasoning process" can be removed from the prompt information, and sensitive word filtering can be applied to pending messages "filtered for reasoning," thus obtaining filtered prompt information. This filtered prompt information can then be input into the target processing model for processing based on security policies. The method for processing filtered prompt information in S503 can be found in [reference needed]. Figure 3 and Figure 4 The corresponding implementation examples are not described in detail.

[0152] The advantage of processing the prompts in the above way is that some information is masked before the prompts are input into the target processing model, thus avoiding any adverse effects on the model's performance after the prompts are input.

[0153] Optionally, a prompt message can be generated based on the message to be processed and its corresponding source, including:

[0154] Based on the detection rules corresponding to the message source of the message to be processed, perform security checks on the message to be processed.

[0155] In response to a pending message passing a security check, a prompt message is generated based on the pending message and its corresponding message source.

[0156] In this application, the security checks performed on the messages to be processed include, but are not limited to, detection of prompt injection, detection of illegal tool call instructions, and detection of privilege escalation requests, or any one or more of these. Prompt injection can be implemented using any one or more methods such as rule-based detection methods, semantic detection methods, and classification model detection methods, and the implementation methods of these detection methods can be found in related technologies. Detection of privilege escalation requests can be performed based on a pre-established privilege escalation feature library (which records common keywords that privilege escalation requests may carry, such as admin, root, etc.) and / or a sensitive resource library (which records resources related to permission control, such as system configuration files, user credential files, private key files, etc.), that is, detecting whether the message to be processed contains the corresponding keywords or whether it requests access to sensitive resources to detect whether it is a privilege escalation request. Detection of illegal tool call instructions can be implemented using any one or more methods such as detecting whether it belongs to a compliance whitelist, detecting whether the called tool is compliant, detecting whether the call parameters are compliant, and detecting whether the context of the message to be processed is compliant (i.e., determining whether the tool call intent matches the user's true intent), and the corresponding implementation methods can be found in related technologies.

[0157] In this embodiment, the security level corresponding to the message source can be determined, and then the message to be processed can be subjected to security detection according to the detection rules corresponding to the security level.

[0158] For example, for a trusted level, the detection rules could be to detect only illegal tool invocation commands and privilege escalation requests, without detecting injection hints; for a partially trusted level, the detection rules could include detecting illegal tool invocation commands, privilege escalation requests, and injection hints; for an untrusted level, the detection rules could include detecting illegal tool invocation commands, privilege escalation requests, and injection hints, and could also include detecting data leakage, malicious links, etc. Each detection requires enhanced detection methods, meaning each detection uses multiple methods mentioned above for cross-verification of the results. Furthermore, stricter thresholds can be used to filter the detection results; for example, a probability of 0.6 for injection hints might be considered a detected injection hint, while in a partially trusted level, only a probability of 0.8 might be considered a detected injection hint.

[0159] For each pending message, if the pending message passes the security check, the pending message can be used to generate corresponding prompt information in accordance with the aforementioned method, such as filling in the aforementioned prompt information template that has been divided into information partitions;

[0160] For each pending message, if the pending message fails the security check, it may not be included in the notification message; that is, the notification message does not contain pending messages that have failed the security check.

[0161] The advantages of applying the above embodiments are: on the one hand, security detection further enhances the security of message processing; on the other hand, different detection rules are applied to different security levels, and simplified detection rules can be used for messages with higher security levels, thereby improving detection efficiency.

[0162] Optional, also includes:

[0163] In response to a pending message failing the security check, output a check failure message. The check failure message includes at least one of the following: the pending message, the corresponding message source, and the reason for failing the security check.

[0164] After outputting a detection failure message, the system can further prompt the user whether to input these pending messages that failed the security check into the target processing model for inference. If the user confirms inputting these messages into the target processing model, the pending messages that failed the security check can be added to the prompt message. If the user confirms removing these messages, the pending messages can be removed from the prompt message.

[0165] Using the methods described above, reasoning can be performed on some pending messages that have failed security checks, based on user needs, thus meeting the user's personalized requirements.

[0166] This embodiment also provides an electronic device, such as... Figure 6 As shown, it includes a memory 601 and a processor 602;

[0167] Memory 601 is used to store computer programs;

[0168] Processor 602 is used to execute computer programs to perform:

[0169] Receive pending messages;

[0170] Determine the source of the message to be processed;

[0171] Generate prompt information based on the pending message and its corresponding message source;

[0172] The target processing model processes the prompt information based on the security policy to determine the processing result of the message to be processed. Messages from different sources correspond to different security policies.

[0173] The working principle of electronic devices can be found in the relevant steps of the aforementioned message processing method, and will not be repeated here.

[0174] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. The systems or devices described above are divided into various modules or units based on their functions. In implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0175] In this document, relational terms such as first, second, and third are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0176] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A message processing method, comprising: Receive pending messages; Determine the message source of the message to be processed; Generate prompt information based on the message to be processed and its corresponding message source; The prompt information is processed by the target processing model based on the security policy to determine the processing result corresponding to the message to be processed. The message to be processed from different message sources corresponds to different security policies.

2. The method according to claim 1, wherein generating prompt information based on the message to be processed and the corresponding message source includes: Obtain a source tag representing the source of each of the multiple messages to be processed; A prompt message is generated based on the source tag and multiple messages to be processed. The source tag is used to divide the prompt message into information partitions corresponding to the message source, and the messages to be processed are located in the information partitions corresponding to the message source.

3. The method according to claim 1, wherein processing the prompt information based on a security policy using a target processing model to determine the processing result corresponding to the message to be processed includes: Determine the security level of the message source of the message to be processed; Obtain the security policy corresponding to the security level, wherein different security policies correspond to different operation permissions for the target processing model; The target processing model processes the prompt information based on the operation permissions corresponding to the security policy, and determines the processing result corresponding to the message to be processed.

4. The method according to claim 3, wherein processing the prompt information based on the operation permissions corresponding to the security policy through the target processing model and determining the processing result corresponding to the message to be processed includes: The target processing model is used to reason about the prompt information to obtain the operation instructions corresponding to the message to be processed. Based on the operation instructions and the corresponding operation permissions of the security policy, the operation instructions are processed, and the processing result corresponding to the message to be processed is determined.

5. The method according to claim 4, wherein processing the operation instruction according to the operation permission corresponding to the operation instruction and the security policy, and determining the processing result corresponding to the message to be processed, includes: Based on the operation permissions corresponding to the security policy, a target tool that meets the operation permissions is determined from the candidate tools corresponding to the operation instructions; The operation instructions are executed based on the target tool to determine the processing result corresponding to the message to be processed.

6. The method according to claim 1, wherein processing the prompt information based on a security policy using a target processing model to determine the processing result corresponding to the message to be processed includes: The usage method corresponding to the message source of the message to be processed is determined according to the security policy; If the usage method indicates that the message to be processed cannot be used in the reasoning process, filter out the messages to be processed that are not used for reasoning from the prompt information to obtain the filtered prompt information; The filtered prompt information is processed based on the security policy using the target processing model to determine the processing result corresponding to the message to be processed.

7. The method according to claim 1, wherein determining the message source of the message to be processed includes at least one of the following: Parse at least one of the message header field, event type field, sender identifier field, and system event identifier field of the message to be processed to determine the message source of the message to be processed; The data structure of the message to be processed is detected, and the source of the message is determined based on the data structure.

8. The method according to claim 1, wherein generating prompt information based on the message to be processed and the corresponding message source includes: The message to be processed is subjected to security detection according to the detection rules corresponding to the message source of the message to be processed; In response to the pending message passing the security check, a prompt message is generated based on the pending message and the corresponding message source.

9. The method according to claim 8, further comprising: In response to the pending message failing the security check, a detection failure message is output, which includes at least one of the following: the pending message, the corresponding message source, and the reason for failing the security check.

10. An electronic device, comprising a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program to perform: Receive pending messages; Determine the message source of the message to be processed; Generate prompt information based on the message to be processed and its corresponding message source; The prompt information is processed by the target processing model based on the security policy to determine the processing result corresponding to the message to be processed. The message to be processed from different message sources corresponds to different security policies.

Images