Interaction method, apparatus, electronic device, storage medium, and program product

Abstract

The present disclosure provides an interaction method, which can be applied to the technical fields of artificial intelligence, information security, computer software and the like, and the method comprises: in response to a first application program being started, calling a second application program, wherein the second application program is in a trusted execution environment; based on interaction information received by the second application program, generating a target interaction request and a disturbance request; sending the target interaction request and the disturbance request to a server to execute an interaction task by using the server, so as to obtain a target interaction result generated in response to the target interaction request and a disturbance result generated in response to the disturbance request; and feeding back the target interaction result to a target object. The present disclosure also provides an interaction device, an electronic device, a storage medium and a program product.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence, information security, and computer software, and more specifically, to an interaction method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With breakthroughs in artificial intelligence technologies such as deep learning and natural language processing, the intelligence level of Human-Computer Interaction (HCI) is constantly improving, enabling machines to better understand human language and behavior and achieve more precise interactions. With the popularization of technologies such as the Internet of Things and cloud computing, the application scenarios of HCI are constantly expanding, covering multiple fields such as smart homes, autonomous driving, and telemedicine, bringing great convenience to human life.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technology: user data is at risk during human-computer interaction. Summary of the Invention

[0004] In view of the above, this disclosure provides an interaction method, apparatus, electronic device, storage medium, and program product.

[0005] One aspect of this disclosure provides an interaction method, comprising: invoking a second application in response to the launch of a first application, wherein the second application is in a trusted execution environment; generating a target interaction request and a interference request based on interaction information received by the second application, wherein the interaction information is input by a target object, the target interaction request indicating the interaction intent of the interaction information, and the interference request indicating an interference intent whose semantic similarity to the interaction intent is greater than a predetermined threshold; sending the target interaction request and the interference request to a server to perform an interaction task using the server, obtaining a target interaction result generated in response to the target interaction request and an interference interaction result generated in response to the interference request; and feeding back the target interaction result to the target object.

[0006] According to an embodiment of this disclosure, generating an interference request based on interaction information received by a second application includes: rewriting the interaction information to obtain an interference request when there is a mismatch between the model performance of a large model configured locally and the interaction task indicated by the interaction information.

[0007] According to embodiments of this disclosure, the method further includes: when the model performance of the large model matches the interaction task indicated by the interaction information, invoking the large model to determine whether to perform a retrieval based on the interaction information; when the retrieval identification result determines to perform a retrieval, obtaining the retrieval result matching the interaction information from a locally configured database; and obtaining the target interaction result based on the retrieval result and the interaction information.

[0008] According to embodiments of this disclosure, rewriting interactive information to obtain an interference request includes: when the interactive information includes text, performing field analysis on the interactive information to obtain field analysis results, wherein the field analysis results include multiple fields in the interactive information and field attribute information of the fields; and semantically rewriting a target field among the multiple fields to obtain an interference request, wherein the target field is determined based on the interactive intent of the interactive information and the field attribute information of each field.

[0009] According to embodiments of this disclosure, rewriting interactive information to obtain an interference request includes: when the interactive information includes an image, recognizing the image to obtain an image recognition result; and generating an image based on the image recognition result to obtain the interference request.

[0010] According to embodiments of this disclosure, after invoking the second application, the method further includes: closing the communication between the first application and the interactive hardware; and opening the communication between the second application and the interactive hardware so that the second application can receive interactive information input by the target object through the interactive hardware.

[0011] According to embodiments of this disclosure, feeding back a target interaction result to a target object includes: determining a target interaction result from multiple interaction results sent by a server based on result-request mapping information, wherein the result-request mapping information characterizes the correspondence between the interaction result and the target interaction request or the interference request, and the multiple interaction results include the target interaction result and the interference interaction result; and feeding back the target interaction result to the target object by displaying it on a display.

[0012] According to embodiments of this disclosure, sending a target interaction request and an interference request to a server includes: encrypting the target interaction request and the interference request respectively to obtain an encrypted target interaction request and an encrypted interference request; and sending the encrypted target interaction request and the encrypted interference request to the server through an interaction interface of a first application for connecting to the server.

[0013] Another aspect of this disclosure provides an interaction device, comprising: a program invocation module for invoking a second application in response to the launch of a first application, wherein the second application is in a trusted execution environment; an interference generation module for generating a target interaction request and an interference request based on interaction information received by the second application, wherein the interaction information is input by a target object, the target interaction request indicating the interaction intent of the interaction information, and the interference request indicating an interference intent whose semantic similarity to the interaction intent is greater than a predetermined threshold; a task execution module for sending the target interaction request and the interference request to a server to execute an interaction task using the server, thereby obtaining a target interaction result generated in response to the target interaction request and an interference interaction result generated in response to the interference request; and an interaction feedback module for feeding back the target interaction result to the target object.

[0014] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0015] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0016] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the method described above.

[0017] According to embodiments of this disclosure, interaction information of a target object can be received in a trusted execution environment, thereby improving data security from the perspective of the operating environment. In addition, interference requests with noise are mixed into the target interaction requests of the interaction information, reducing the probability of real data being leaked or tampered with during communication between the terminal device and the server. Thus, user data security is improved from multiple angles and aspects during human-computer interaction. Attached Figure Description

[0018] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustration schematically depicts an exemplary system architecture for training methods, recommendation methods, and apparatus for large models that can be applied according to embodiments of the present disclosure;

[0020] Figure 2 A flowchart illustrating an interaction method according to an embodiment of the present disclosure is shown schematically;

[0021] Figure 3 A schematic flowchart of an interaction method according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A block diagram of a terminal device according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 5 A timing diagram illustrating an interaction method according to an embodiment of the present disclosure is shown schematically;

[0024] Figure 6 A block diagram schematically illustrates an interactive device according to embodiments of the present disclosure; and

[0025] Figure 7 A block diagram of an electronic device suitable for implementing an interaction method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0031] Ensuring user data security during human-computer interaction presents numerous challenges. Balancing convenience and security in interaction requires further exploration.

[0032] In view of this, embodiments of the present disclosure provide an interaction method, comprising: invoking a second application in response to the launch of a first application, wherein the second application is in a trusted execution environment; generating a target interaction request and an interference request based on interaction information received by the second application, wherein the interaction information is input by a target object, the target interaction request indicating the interaction intent of the interaction information, and the interference request indicating an interference intent whose semantic similarity to the interaction intent is greater than a predetermined threshold; sending the target interaction request and the interference request to a server to perform an interaction task using the server, obtaining a target interaction result generated in response to the target interaction request and an interference interaction result generated in response to the interference request; and feeding back the target interaction result to the target object.

[0033] According to embodiments of this disclosure, interaction information of a target object can be received in a trusted execution environment, thereby improving data security from the perspective of the operating environment. In addition, interference requests with noise are mixed into the target interaction requests of the interaction information, reducing the probability of real data being leaked or tampered with during communication between the terminal device and the server. Thus, user data security is improved from multiple angles and aspects during human-computer interaction.

[0034] Figure 1 The illustration schematically depicts an exemplary system architecture to which interactive methods and apparatus can be applied according to embodiments of the present disclosure.

[0035] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0036] like Figure 1As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0038] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the interaction method provided in this embodiment can generally be executed by server 105. Correspondingly, the interaction device provided in this embodiment can generally be located in server 105. The interaction method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the interaction device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Alternatively, the interaction method provided in this embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Correspondingly, the interaction device provided in this embodiment can also be located in terminal devices 101, 102, or 103, or in other terminal devices different from terminal devices 101, 102, or 103.

[0041] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Figure 2 A flowchart illustrating an interaction method according to an embodiment of this disclosure is shown schematically.

[0043] like Figure 2 As shown, the method includes operations S210~S240.

[0044] In operation S210, in response to the first application being launched, the second application is invoked.

[0045] In operation S220, a target interaction request and an interference request are generated based on the interaction information received by the second application.

[0046] In operation S230, the target interaction request and the interference request are sent to the server so that the server can perform the interaction task and obtain the target interaction result generated in response to the target interaction request and the interference interaction result generated in response to the interference request.

[0047] In operation S240, the target interaction result is fed back to the target object.

[0048] The target user can refer to a user who interacts with a terminal device. The target user can use the terminal device to open a first application, which is configured with functional modules for human-computer interaction. Optionally, the first application can run in a rich execution environment (REE), such as a runtime environment developed by a third party, which may be subject to data tampering during the operation of the first application.

[0049] The terminal device may be configured with a second application, which may optionally reside in a trusted execution environment. For example, the second application may run in a trusted execution environment (TEE), such as a data-secure and reliable execution environment isolated from the rich execution environment by hardware mechanisms. The second application running in this environment is protected, and no operating system can tamper with or steal the content in the trusted execution environment.

[0050] Based on the configuration operations of the target object, when the first application is launched, the second application can be called so that when interacting with the target object, the second application can receive the interaction information input by the target object, thereby ensuring that the interaction information is in a safe and reliable execution environment and preventing user data from being tampered with or leaked.

[0051] Interactive information can be control commands for the target object, such as generating an image, playing a song, checking the weather, or translating text, but it is not limited to these. It can also be chat messages intended for conversation. Any information input by the target object with an interactive intent is acceptable.

[0052] When a terminal device needs to perform an interactive task through a remote server to obtain the interactive result and then return it to the target object, the interactive information can be directly sent to the server as the target interactive request so that the server can perform the interactive task and obtain the interactive result.

[0053] However, during the interaction between the terminal device and the server, there is still the risk of data leakage and tampering. To avoid this data security issue, a distraction request can be generated simultaneously based on the interaction information to create a target interaction request. Both the target interaction request and the distraction request are then sent to the server, allowing the server to execute the interaction tasks separately and obtain the target interaction result generated in response to the target interaction request and the distraction interaction result generated in response to the distraction request.

[0054] Optionally, the interactive information can be directly used as the target interactive request, but it is not limited to this. The interactive information can also be processed by adding authentication information, encryption, data format conversion, encoding, or encapsulation to generate the target interactive request. As long as it is a target interactive request that indicates the interactive intent of the interactive information, it is acceptable.

[0055] Optionally, the interference request can be interference interaction information determined after transforming the interaction information by its intent. For example, if the interaction information may include "Recommend a three-day tour itinerary for city AA", then the interference interaction information may include "Recommend a two-day tour itinerary for city BB". Similar to the method of generating the target interaction request based on the interaction information, interference requests can be generated based on the interference interaction information. There are no limitations on the interference request with the indicated interference intent, as long as the semantic similarity between the interference intent and the interaction intent is greater than a predetermined threshold. The predetermined threshold is not specifically limited and can be adaptively adjusted according to actual needs.

[0056] The terminal device responds to the target interaction result and interference interaction result received from the server by receiving the target interaction result and then sends the target interaction result back to the target object.

[0057] Therefore, by interfering with the request, the importance of the information in the target interaction request is reduced, and the probability of real data leakage is decreased by adding noise to the interfering request. For example, by interfering with the target interaction request, the difficulty of identifying the mapping relationship between the target object and the real interaction information is increased, thus reducing the probability of the target object's true interaction intent being leaked.

[0058] Optionally, there is no limit to the number of interference requests. The more interference requests, the greater the interference capability and the lower the probability of real data leakage. However, this also increases resource consumption. Adjustments can be made according to actual needs.

[0059] According to embodiments of this disclosure, interaction information of a target object can be received in a trusted execution environment, thereby improving data security from the perspective of the operating environment. In addition, interference requests with noise are mixed into the target interaction requests of the interaction information, reducing the probability of real data being leaked or tampered with during communication between the terminal device and the server. Thus, user data security is improved from multiple angles and aspects during human-computer interaction.

[0060] In a relevant example, a raw data encoding scheme can be used. For instance, the interactive information input by the target object is encoded locally to obtain the encoded result, which is then transmitted to the cloud for processing. Finally, the result is decoded and output locally.

[0061] By utilizing the original data encoding scheme, since the vector itself still carries the original input information of the target object, the interactive information input by the target object can still be restored to a high degree through attack methods such as embedding restoration attacks.

[0062] In another relevant example, a differential privacy scheme can also be employed. For instance, the data is divided into a secret domain and a plaintext domain. User privacy data is encrypted using a cloud public key and transmitted to the cloud. In the cloud, it is decrypted using a cloud private key, and then inferred using a large model in the secret domain and a parallel random confusion low-rank matrix. The result is then transmitted to the plaintext domain model for further inference to obtain the interactive result.

[0063] Using differential privacy schemes requires additional encryption and decryption operations on privacy data, which brings additional performance overhead. Furthermore, low-rank matrices mean that they can only work in a small subspace. When the privacy data input by the user happens to be outside this space, the user's privacy is not protected.

[0064] In another related example, a fully homomorphic encryption scheme can also be used. Specifically, the pre-trained model undergoes an adaptive transformation to make it compatible with homomorphic operations. During the inference phase, the server receives the user's encrypted ciphertext input and uses fully homomorphic encryption combined with model weights to complete the ciphertext computation. Finally, the generated ciphertext result is decrypted by the user using their private key to restore the plaintext.

[0065] Using a fully homomorphic encryption scheme, homomorphic computation only natively supports algebraic operations. Therefore, nonlinear functions in the model must first undergo polynomial approximation, which introduces additional computational complexity and precision loss. Secondly, noise introduced during fully homomorphic computation must be refreshed through bootstrapping, which incurs significant performance overhead. Furthermore, the ciphertext size increases dramatically after encrypting plaintext data, posing a severe challenge to limited GPU memory and bandwidth.

[0066] The advantages of the embodiments of this disclosure have been further illustrated above with comparative examples. The following will further illustrate how to improve user data security while simultaneously increasing interaction efficiency using a large model configured locally.

[0067] According to embodiments of this disclosure, when performing such Figure 2 Prior to the operation S220 shown, the interaction method may further include: determining whether the model performance of the large model configured locally matches the interaction task indicated by the interaction information.

[0068] Large models can include large language models (LLMs), but are not limited to them. They can also include large vision models (LVMs) or multimodal large models (MLMs), as long as they are large models that can be used to perform interactive tasks.

[0069] For example, a large model configured locally can be lightweight and suitable for running on terminal devices, thereby improving interaction efficiency while reducing the requirements for hardware resources.

[0070] Model performance can refer to the types of tasks a large model can perform, but it is not limited to this; it can also refer to the processing time for performing tasks.

[0071] For example, it can be determined whether the task types that the large model can perform are the same as the task types of the interactive tasks indicated by the interactive information, and whether the model performance of the large model configured locally matches the interactive tasks indicated by the interactive information.

[0072] For example, it can be determined whether the processing time of the interactive task based on the interaction information indication based on the large model is shorter than the predetermined processing time threshold, and whether the model performance of the large model configured locally matches the interactive task indicated by the interaction information.

[0073] Figure 3 A schematic flowchart of an interaction method according to an embodiment of the present disclosure is shown.

[0074] like Figure 3 As shown, the interaction method may include operations S310 to S330.

[0075] In operation S310, determine whether the performance of the large model configured locally matches the interaction task indicated by the interaction information. If they do not match, execute operation S320; otherwise, execute operation S330.

[0076] When operating S320, if there is a mismatch between the model performance of the large model configured locally and the interaction task indicated by the interaction information, an interference request is generated based on the interaction information of the target object received by the second application.

[0077] When operating S330, if the performance of the large model configured locally matches the interaction task indicated by the interaction information, the large model is invoked to execute the interaction task and obtain the target interaction result.

[0078] According to embodiments of this disclosure, generating an interference request based on interaction information of a target object received by a second application may include: rewriting the interaction information to obtain the interference request.

[0079] The large model can be used as a fully automated mirror-level intent obfuscator. It automatically switches obfuscation strategies based on the type of interaction information, generating obfuscation information that is perfectly aligned in terms of grammatical, logical, and visual features. Obfuscation requests are then generated based on this obfuscation information.

[0080] According to embodiments of this disclosure, the rewriting capability of a large model is utilized to increase the similarity between the interference request and the target interaction request, thereby improving the interference effect of the interference request.

[0081] According to embodiments of this disclosure, the type of interactive information differs in different interaction scenarios. For example, in a graph processing interaction scenario, the interactive information may include images. As another example, in a translation scenario, the interactive information may include text. Different methods for generating interference requests are employed for different types of interactive information.

[0082] Taking interactive information, including text, as an example, for example... Figure 3 The operation S320 shown, which rewrites the interactive information to obtain an interference request, may include: if the interactive information includes text, performing field analysis on the interactive information to obtain field analysis results. The field analysis results include multiple fields in the interactive information and their field attribute information. Semantic rewriting is then performed on the target field among the multiple fields to obtain the interference request. The target field is determined based on the intent recognition results of the interactive information and the field attribute information of each field.

[0083] The interaction intent of a target object can be determined using interaction information through intent recognition (internet). The interaction information is segmented to obtain a field sequence containing multiple fields. Entity information is extracted from the field sequence to obtain the part-of-speech tag for each field, which serves as the field attribute information. The field sequence and the field attribute information are combined as the field analysis result. The target field (slot) indicating the interaction intent can then be determined from the field sequence.

[0084] The target field is rewritten, for example, through semantic substitution, generating multiple sets of pseudo-intent requests within the same semantic space as distractor requests. These distractor requests maintain a syntactic structure highly similar to the target interaction request generated based on the interaction information, performing equivalent substitutions only on key entities. For example, for the interaction information "Recommend a three-day travel route in city AA," only the target keywords such as city and date need to be replaced to generate the corresponding distractor request.

[0085] According to embodiments of this disclosure, by leveraging the rewriting capabilities of a large model, when the interaction information is text, through intent recognition and field analysis, the target field indicating the interaction intent can be accurately determined from each field based on the field analysis results and the interaction intent. This achieves semantically invariant rewriting, ensuring that the interference intent indicated by the interference request is similar to the interaction intent indicated by the target interaction request. Consequently, in continuous dialogue human-computer interaction scenarios, the diversity of intents will not prevent the continuation of dialogue or render the preceding context unusable as reference information for subsequent dialogue interactions, thus maintaining the accuracy of the large model's inference. Furthermore, by combining interference requests with target interaction requests, multiple sets of privacy-protected requests can be formed, making it impossible for the cloud to determine which request represents the user's actual input data, thereby preventing the leakage or tampering of user data.

[0086] Taking interactive information including images as an example, for example... Figure 3 The operation S320 shown, which rewrites the interaction information to obtain an interference request, may include: if the interaction information includes an image, recognizing the image to obtain an image recognition result; and generating an image based on the image recognition result to obtain the interference request.

[0087] When a video uploaded by the target object is used as interactive information, the video can be split into frames to obtain multiple image frames. The images or image frames are then identified to determine their features, yielding an image recognition result. This result can include semantic information. For example, if an image shows a boy playing soccer, the recognition result could be "A boy is playing soccer on the lawn." Based on the recognition result, text-based image generation is performed to obtain a distractor image. Based on this distractor image, a distractor request is generated. This can be done using the distractor image as the distractor request, but it's not limited to this; encoding, adding authentication information, and other methods can be used to generate the request in conjunction with the target interaction request.

[0088] According to embodiments of this disclosure, the rewriting method for large models can be extended from text rewriting to text-to-image rewriting, thus broadening the application scope. Furthermore, performing text-to-image rewriting based on image recognition results enhances the interference effect of interfering requests, preventing the server from determining the target interaction request from multiple requests through multi-turn dialogue information during task execution, thereby improving data security.

[0089] According to embodiments of this disclosure, for example, Figure 3 Operation S330, as shown, invokes a large model to execute an interaction task and obtains the target interaction result. This may include: invoking the large model to determine whether to perform a search based on the interaction information; if the search result determines to perform a search, retrieving search results that match the interaction information from a locally configured database; and obtaining the target interaction result based on the search results and the interaction information.

[0090] The locally configured large model can be a lightweight end-device model based on a general large model, after distillation or pruning, thereby improving the compatibility between the large model and the terminal device. If the model performance of the large model matches the interactive task indicated by the interactive information, it is determined that the locally configured large model can be used to execute the interactive task.

[0091] Optionally, interactive information can be input into a large model, along with prompts indicating the execution of interactive tasks, to obtain the target interactive result.

[0092] Optionally, to further optimize the accuracy and effectiveness of the target interaction results output by the large model, a vector database can be built within the TEE environment to realize large model inference technology based on retrieval augmented generation (RAG).

[0093] For example, the large model can enhance retrieval when the interaction intent indicated by the interaction information determines that a retrieval is needed. Furthermore, a locally configured database can be used for retrieval to obtain results. For instance, the interaction information can be vectorized, and vector retrieval can be performed based on the vectors obtained from the interaction information to obtain retrieval results. The retrieval results and interaction information are then input into the large model to obtain the target interaction result.

[0094] According to embodiments of this disclosure, deploying a large model for performing interactive tasks locally on the terminal device enables user data to remain within the local domain, improving data security. Furthermore, configuring a local database that remains within the local domain allows the large model to perform enhanced reasoning modes for retrieval, improving the accuracy and effectiveness of the target interactive results output by the large model, thereby enhancing interaction efficiency, security, and the effectiveness of feedback.

[0095] Figure 4 A block diagram of a terminal device according to an embodiment of the present disclosure is shown schematically.

[0096] like Figure 4 As shown, the terminal device can be configured with REE and TEE environments, a first application, and a second application. The first application runs in the REE, and the second application runs in the TEE. Large models and databases can be configured in the TEE environment to improve user data security when calling large models for retrieval and enhanced intelligent interaction. The terminal device's TEE environment can also be configured with security driver components. For example, a collection of drivers deployed on the terminal device. This covers secure access interfaces for hardware resources such as sound cards, touchscreens, and displays.

[0097] like Figure 4 As shown, the terminal device may also include interactive hardware such as a touchscreen, display, and keyboard. The interactive hardware is used to sense the interactive information input by the target object and display the target interaction results.

[0098] The second application can control the communication path between the first application and the driver set via the interrupt controller. This ensures that interactive information acquired through the interactive hardware is in a data-secure operating environment. See the following embodiment description for details.

[0099] According to embodiments of this disclosure, when performing such Figure 2 The operation S210 shown, after invoking the second application, may further include: disabling communication between the first application and the interactive hardware; and enabling communication between the second application and the interactive hardware so that the second application can receive interactive information input by the target object through the interactive hardware.

[0100] The first and second applications can share the interactive hardware configured on the terminal device through a set of drivers. For data security, a secure interrupt controller can control who has exclusive access. For example, when the second application is invoked, the secure interrupt controller can be enabled, disabling communication between the first application and the interactive hardware while enabling communication between the second application and the interactive hardware. This ensures that the interactive hardware is exclusively accessed by the TEE, and the REE cannot intercept or respond to it. It is responsible for providing a controlled peripheral interaction channel for trusted applications such as the second application in an isolated state.

[0101] According to embodiments of this disclosure, user-input interactive information can be directly transmitted to the TEE via a secure interrupt controller, interactive hardware, and secure driver components. The REE runtime environment cannot intercept or respond to this, thereby achieving end-to-end user privacy data protection on the terminal device. Furthermore, using this terminal device eliminates the need for additional encryption operations and cryptographic calculations, resulting in minimal performance overhead.

[0102] like Figure 4 As shown, the REE runtime environment can also be configured with an interactive interface and software. The interactive interface is used for communication between the first application and the server. The software is used to load the second application. A secure runtime control can be configured on the first application, allowing users to operate the secure runtime control and use the software to install the second application and the TEE runtime environment.

[0103] After the second application generates the target interaction request and the interference request, it can send the target interaction request and the interference request to the server through the interaction interface.

[0104] The above text refers to... Figure 4 The hardware and operating program of the terminal device shown have been described. The following section will explain in detail how to use the hardware and operating program of the terminal device to execute the interaction method.

[0105] Figure 5 A timing diagram of an interaction method according to an embodiment of the present disclosure is illustrated schematically.

[0106] like Figure 5 As shown, after the user launches the first application, the second application is invoked. The second application controls the security interrupt controller to close the communication between the first application and the interactive hardware, and open the communication between the second application and the interactive hardware to establish a secure human-computer interaction session.

[0107] like Figure 5 As shown, the second application writes the data required to render the trusted interactive interface into the cache, and the second application obtains the corresponding data and displays the screen of the interactive device to complete the interface rendering.

[0108] like Figure 5 As shown, the interface seen by the target object is created by the trusted application; the REE is intercepted and becomes unresponsive. The target object completes multimodal information input within this interface, waking up the second application and passing in interactive information.

[0109] like Figure 5 As shown, the second application calls the large model to determine whether to execute the interaction task locally based on the interaction information. If it determines to execute the interaction task using the local large model, the interaction task is executed, the target interaction result is generated, and sent to the display to show the target object.

[0110] like Figure 5 As shown, when it is determined that the request will be sent to the server to perform a task, a target interaction request and a interference request can be generated based on the interaction information.

[0111] like Figure 5 As shown, the target interaction request and interference request are sent to the server through the interaction interface of the first application for connecting to the server.

[0112] like Figure 5 As shown, the server responds to the target interaction request and the interference request respectively, executes the interaction task of the target interaction request and the interaction task of the interference request, and determines the target interaction result and the interference interaction result.

[0113] like Figure 5 As shown, the server sends multiple interaction results, which are a combination of the target interaction result and the interference interaction result, to the second application through the interaction interface of the first application.

[0114] like Figure 5 As shown, the second application determines the target interaction result from multiple interaction results based on the result-request mapping information.

[0115] The result-request mapping information represents the correspondence between the interaction result and the target interaction request, and the correspondence between the interaction result and the interference request. Multiple interaction results include the target interaction result corresponding to the target interaction request and the interference interaction result corresponding to the interference request.

[0116] like Figure 5 As shown, the target interaction result is fed back to the target object by being displayed on the screen.

[0117] According to embodiments of this disclosure, establishing a result-request mapping information enables precise filtering of target interaction results from multiple interaction results to obtain the target interaction request, thereby improving interaction efficiency and accuracy.

[0118] According to an optional embodiment of this disclosure, sending the target interaction request and the interference request to the server may include: encrypting the target interaction request and the interference request respectively to obtain an encrypted target interaction request and an encrypted interference request. The encrypted target interaction request and the encrypted interference request are then sent to the server through an interaction interface of the first application for connecting to the server.

[0119] The second application can establish encrypted communication. It adds request header parameters to the interaction information to obtain the target interaction request, and adds request header parameters to the interfering interaction information to obtain the interfering request. Both the target interaction request and the interfering request are then encrypted.

[0120] After receiving the encrypted target interaction request and the encrypted interference request, the server decrypts them according to the corresponding encryption algorithm to obtain the decrypted target interaction request and interference request. In response to receiving the target interaction request and interference request, the server calls the large model configured on the server to execute the interaction task and obtains the target interaction result generated in response to the target interaction request and the interference interaction result generated in response to the interference request.

[0121] The target interaction result corresponding to the target interaction request and the interference interaction result corresponding to the interference request are encrypted separately to obtain multiple encrypted interaction results. These encrypted interaction results are then fed back to the second application through an interaction interface, so that the second application can decrypt them and display the target interaction result on the screen, completing one request and response cycle.

[0122] According to embodiments of this disclosure, secure communication between terminal devices and servers is achieved by utilizing encrypted communication. This combines different mechanisms such as communication security, noise addition to interference requests, and operational environment security, employing different security protection methods from different perspectives to jointly improve interaction security and thereby enhance the user interaction experience.

[0123] Figure 6 A block diagram of an interactive device according to an embodiment of the present disclosure is shown schematically.

[0124] like Figure 6 As shown, the interactive device 600 of the embodiments of this disclosure includes a program calling module 610, an interference generation module 620, a task execution module 630, and an interactive feedback module 640.

[0125] The program invocation module is used to invoke a second application in response to the startup of the first application, wherein the second application is in a trusted execution environment. In one embodiment, the program invocation module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0126] The interference generation module 620 is used to generate a target interaction request and an interference request based on the interaction information received by the second application. The interaction information is input by the target object, the target interaction request indicates the interaction intent of the interaction information, and the interference request indicates an interference intent whose semantic similarity to the interaction intent is greater than a predetermined threshold. In one embodiment, the interference generation module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0127] The task execution module 630 is used to send the target interaction request and the interference request to the server, so that the server can execute the interaction task to obtain the target interaction result generated in response to the target interaction request and the interference interaction result generated in response to the interference request. In one embodiment, the task execution module 630 can be used to execute the operation S230 described above, which will not be repeated here.

[0128] The interactive feedback module 640 is used to provide feedback on the target interaction result to the target object. In one embodiment, the interactive feedback module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0129] According to embodiments of this disclosure, the interference generation module includes a rewriting submodule.

[0130] The rewrite submodule is used to rewrite the interaction information to obtain interference requests when there is a mismatch between the model performance of the large model configured locally and the interaction task indicated by the interaction information.

[0131] According to embodiments of this disclosure, the interactive device further includes: a retrieval determination module, a retrieval module, and a result generation module.

[0132] The retrieval determination module is used to determine whether to perform a retrieval based on the interaction information when the model performance of the large model matches the interaction task indicated by the interaction information.

[0133] The retrieval module is used to retrieve search results that match the interactive information from a locally configured database when the search identification results determine that a search will be performed.

[0134] The results generation module is used to obtain the target interactive results based on the search results and interactive information.

[0135] According to embodiments of this disclosure, the rewriting submodule includes a field analysis unit and a semantic rewriting unit.

[0136] The field analysis unit is used to perform field analysis on interactive information, including text, and obtain field analysis results. The field analysis results include multiple fields in the interactive information and field attribute information of the fields.

[0137] The semantic rewriting unit is used to semantically rewrite the target field among multiple fields to obtain the interference request. The target field is determined based on the interaction intent of the interaction information and the field attribute information of each field.

[0138] According to embodiments of this disclosure, the rewriting submodule includes an image recognition unit and an image generation unit.

[0139] The image recognition unit is used to recognize images when the interactive information includes images, and to obtain image recognition results.

[0140] The image generation unit is used to generate an image based on the image recognition results to obtain the interference request.

[0141] According to embodiments of this disclosure, after invoking the second application, the interactive device further includes a closing module and a turning-on module.

[0142] The shutdown module is used to shut down communication between the first application and the interactive hardware.

[0143] The enabling module is used to enable communication between the second application and the interactive hardware, so that the second application can receive interactive information input by the target object through the interactive hardware.

[0144] According to embodiments of this disclosure, the interactive feedback module includes a result filtering submodule and a result display submodule.

[0145] The result filtering submodule is used to determine the target interaction result corresponding to the target interaction request from multiple interaction results sent by the server based on the result-request mapping information. The result-request mapping information represents the correspondence between the interaction result and the target interaction request or the interference request. The multiple interaction results include the target interaction result and the interference interaction result.

[0146] The results display submodule is used to provide feedback to the target object by displaying the target interaction results on the screen.

[0147] According to embodiments of this disclosure, the task execution module includes an encryption submodule and a sending submodule.

[0148] The encryption submodule is used to encrypt the target interaction request and the interference request respectively, resulting in encrypted target interaction requests and encrypted interference requests.

[0149] The sending submodule is used to send encrypted target interaction requests and encrypted interference requests to the server through the interaction interface of the first application for connecting to the server.

[0150] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0151] For example, any plurality of the program invocation module 610, interference generation module 620, task execution module 630, and interactive feedback module 640 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the program invocation module 610, interference generation module 620, task execution module 630, and interactive feedback module 640 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the program calling module 610, interference generation module 620, task execution module 630, and interactive feedback module 640 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0152] It should be noted that the data processing device part in the embodiments of this disclosure corresponds to the data processing method part in the embodiments of this disclosure. The specific description of the data processing device part is referred to in the data processing method part, and will not be repeated here.

[0153] Figure 7 A block diagram of an electronic device suitable for implementing an interaction method according to an embodiment of the present disclosure is shown schematically.

[0154] Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0155] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0156] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0157] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The system 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0158] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0159] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0160] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0161] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0162] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0163] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0164] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0165] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0166] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0167] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An interaction method, characterized in that, The method includes: In response to the launch of the first application, the second application is invoked, wherein the second application is in a trusted execution environment; Based on the interaction information received by the second application, a target interaction request and an interference request are generated, wherein the interaction information is input by the target object, the target interaction request indicates the interaction intent of the interaction information, and the interference request indicates an interference intent whose semantic similarity to the interaction intent is greater than a predetermined threshold. The target interaction request and the interference request are sent to the server to execute an interaction task, obtaining a target interaction result generated in response to the target interaction request and an interference interaction result generated in response to the interference request; and The target interaction result is fed back to the target object.

2. The method according to claim 1, characterized in that, Based on the interaction information received by the second application, an interference request is generated, including: If there is a mismatch between the performance of the large model configured locally and the interaction task indicated by the interaction information, the interaction information is rewritten to obtain the interference request.

3. The method according to claim 2, characterized in that, The method further includes: If the performance of the large model matches the interactive task indicated by the interactive information, the large model is invoked to determine whether to perform a retrieval and identification result based on the interactive information. If the search results determine that a search should be performed, search results matching the interaction information are retrieved from a locally configured database; and Based on the search results and the interaction information, the target interaction result is obtained.

4. The method according to claim 2 or 3, characterized in that, The step of rewriting the interaction information to obtain the interference request includes: When the interactive information includes text, field analysis is performed on the interactive information to obtain field analysis results, wherein the field analysis results include multiple fields in the interactive information and field attribute information of the fields; and The target field among the multiple fields is semantically rewritten to obtain the interference request, wherein the target field is determined based on the interaction intent of the interaction information and the field attribute information of each field.

5. The method according to claim 2 or 3, characterized in that, The step of rewriting the interaction information to obtain the interference request includes: When the interactive information includes an image, the image is recognized to obtain an image recognition result; and Based on the image recognition results, an image is generated to obtain the interference request.

6. The method according to claim 1, characterized in that, After invoking the second application, the method further includes: Close the communication between the first application and the interactive hardware; and Communication between the second application and the interactive hardware is initiated so that the second application can receive the interactive information input by the target object through the interactive hardware.

7. The method according to claim 1, characterized in that, The step of feeding back the target interaction result to the target object includes: Based on result-request mapping information, the target interaction result is determined from multiple interaction results sent by the server, wherein the result-request mapping information characterizes the correspondence between the interaction result and the target interaction request or the interference request, and the multiple interaction results include the target interaction result and the interference interaction result; and The target interaction result is fed back to the target object by being displayed on the screen.

8. The method according to claim 1, characterized in that, Sending the target interaction request and the interference request to the server includes: The target interaction request and the interference request are encrypted respectively to obtain an encrypted target interaction request and an encrypted interference request; and The encrypted target interaction request and the encrypted interference request are sent to the server through the interaction interface of the first application for connecting to the server.

9. An interactive device, comprising: The program invocation module is used to invoke the second application in response to the startup of the first application, wherein the second application is in a trusted execution environment; An interference generation module is used to generate a target interaction request and an interference request based on interaction information received by the second application, wherein the interaction information is input by a target object, the target interaction request indicates the interaction intent of the interaction information, and the interference request indicates an interference intent whose semantic similarity to the interaction intent is greater than a predetermined threshold. The task execution module is configured to send the target interaction request and the interference request to the server, so as to utilize the server to execute the interaction task and obtain the target interaction result generated in response to the target interaction request and the interference interaction result generated in response to the interference request; and The interactive feedback module is used to feed back the target interaction result to the target object.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images