Human-computer interaction response method and device and electronic equipment

Abstract

This application relates to a human-computer interaction response method that addresses the contradiction between achieving high content quality in large-scale models and low-latency response required for real-time interaction in intelligent assistants. This solution employs an active pre-simulation approach, analyzing user context through a probabilistic event pruning mechanism to predict a set of high-probability user events. This priority event set is then sent to a second model for processing, asynchronously pre-generating information assets corresponding to the priority event set and storing them in a multimodal database. When an interaction request is received, the first model only needs to quickly retrieve the relevant information assets matching the interaction request from the pre-built multimodal database, and then orchestrate and synthesize them with the interaction request and context data to quickly and accurately output the response information. This solution decouples the costly deep generation process from the real-time interaction link, reducing user-perceived interaction latency while ensuring both high-quality and in-depth response content.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a human-computer interaction response method, device, and electronic device. Background Technology

[0002] Currently, the widespread adoption of intelligent assistants based on large language models on mobile devices (such as smartphones and smart glasses) faces a core technological bottleneck: the contradiction between model capabilities and response speed. On the one hand, large models with powerful reasoning and content generation capabilities, such as models with hundreds of billions or even trillions of parameters, are typically massive and computationally intensive, resulting in high response latency and making it difficult to meet the demands for instant interaction in mobile scenarios. After a user makes a request, they often have to wait several seconds to receive a high-quality response, reducing the smoothness of the user experience. On the other hand, small models designed for rapid response, while fast and with low latency, lack cognitive depth, information integration capabilities, and the quality of generated content, making them unable to complete complex tasks and limiting the "intelligence" level of the assistant.

[0003] Related technologies alleviate this contradiction in the following ways: Model compression techniques (such as knowledge distillation, pruning, and quantization): These techniques aim to create statically optimized, more efficient smaller models by "solidifying" the knowledge of the large model into the smaller model during the training phase. This approach is an offline process; once deployed, the large model no longer participates in actual inference operations, which limits the possibility of dynamically utilizing the powerful generation and inference capabilities of the large model at runtime.

[0004] Hierarchical reasoning models: These models employ large and small models working together, but their design goal is to solve single, complex, closed-ended tasks requiring multi-step reasoning (such as problem-solving). Their collaboration is synchronous, designed to deepen single computations, and not for managing dynamic, proactive information asset generation for open-ended dialogue.

[0005] Speculative decoding: This technique uses a small model to quickly generate draft tokens, which are then validated once by a large model, thus accelerating the generation process of a single response. This is a synchronous acceleration technique triggered after a user request, and its operation target is the underlying language tokens. It aims to reduce the computational latency of a single generation, but it does not solve the macroscopic interaction latency that is perceptible to the user caused by the complex content generation task itself.

[0006] Some existing proactive assistant technologies describe the ability to proactively provide "hints" based on context. However, the "proactivity" of these technologies is usually based on simple rules or pattern matching, triggering predefined simple operations, without involving the pre-generation of complex, high-quality information assets by large generative AI models.

[0007] In conclusion, no effective solution has yet been proposed to address the issue of the inability to simultaneously achieve both response speed and content quality in intelligent assistants within related technologies. Summary of the Invention

[0008] This application provides a human-computer interaction response method, apparatus, and computer device to at least solve the problem in the related art that the response speed and content quality of intelligent assistants cannot be simultaneously achieved.

[0009] In a first aspect, embodiments of this application provide a human-computer interaction response method, the method comprising: Upon receiving an interaction request from a user, a first model is used to determine relevant information assets that match the interaction request from multiple information assets. These multiple information assets are pre-generated using a second model based on the user's context data in an active rehearsal manner. Some evaluation parameters of the first model are smaller than those of the second model. The first model processes the relevant information assets, the interaction request, and the context data to generate response information. The response information is output as a signal that can be perceived by the user.

[0010] In some embodiments, the multiple information assets are generated in the following ways: Determine the context data; The context data is processed using a probabilistic event pruning mechanism to obtain a priority event set; The set of priority events is sent to the second model for active pre-rehearsal processing to obtain the multiple information assets. The second model is an expert group, which includes multiple large language models.

[0011] In some embodiments, the context data is processed using a probabilistic event pruning mechanism to obtain a priority event set including: By adding, subtracting, or replacing the constituent elements of the context data, multiple perturbation context sets are generated; By using the pre-simulation model, predictions are made based on each of the aforementioned perturbation context sets, and multiple candidate events for the user in future time periods are output. Based on a probabilistic event pruning mechanism, the priority event set is determined from multiple candidate events.

[0012] In some embodiments, based on a probabilistic event pruning mechanism, determining the priority event set among a plurality of candidate events includes: Configure a corresponding indicator function for each of the candidate events; The perturbation detection frequency of each candidate event is statistically analyzed using the indicated function. By using a preset computational model, the probability score of each candidate event is estimated based on the disturbance detection frequency, and candidate events with probability scores greater than a preset judgment threshold are combined into the priority event set.

[0013] In some embodiments, the constituent elements of the context data are added, removed, or replaced to generate multiple sets of perturbation contexts, specifically including: The Shapley value algorithm is used to calculate the degree of influence of each component of the context data on the potential candidate events. The semantic influence of each component element is calculated based on the degree of influence of each component element. Based on the semantic influence, the target constituent elements to be processed are determined, and multiple sets of the perturbation contexts are generated by adding, subtracting or replacing the target constituent elements.

[0014] In some embodiments, the set of priority events is sent to the second model for active pre-simulation processing to obtain the multiple information assets, including: The events in the priority event set are converted into generation tasks and assigned to multiple large language models corresponding to the expert team; The expert team comprises multiple large language models that perform the generation task to generate the multiple information assets.

[0015] In some embodiments, the plurality of information assets includes at least one of the following: a first information asset, a second information asset, and a third information asset, wherein, The methods for generating the first information asset include: Obtain geographic information associated with the above and below data, and generate a first information asset in advance using an active pre-simulation method based on the geographic information. The first information asset includes: route planning information and location information. The methods for generating the second information asset include: Based on the document data in the context data, a second information asset is generated in advance in an active rehearsal manner. The second information asset includes: text summary, text filtering results, and text contribution information. The methods for generating the third information asset include: Based on user preferences and language styles in the context data, a third information asset is generated in advance in an active rehearsal manner. The third information asset includes: communication drafts and content templates.

[0016] In some embodiments, after pre-generating multiple information assets from the user's context data using a second model in a proactive rehearsal manner, the method further includes: The multiple information assets are acquired, and an index is configured for each information asset based on the event type, event keywords, event generation time, and event feature vector associated with each information asset. Store the information assets with configured indexes in a multimodal database; Using the first model, the relevant information assets are retrieved from the multimodal database based on the index, the interaction request, and the semantic information of the information assets.

[0017] Secondly, embodiments of this application provide a human-computer interaction response device, the device comprising: The retrieval module is used to determine relevant information assets matching the interaction request from multiple information assets through a first model when receiving an interaction request input by a user. The multiple information assets are pre-generated by a second model using the user's context data in an active pre-rehearsal manner. Some evaluation parameters of the first model are smaller than those of the second model. The output module is used to process the relevant information assets, the interaction request, and the context data to generate response information; and to output the response information as a user-perceptible signal.

[0018] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0019] Compared to related technologies, the human-computer interaction response method and apparatus provided in this application are implemented based on a first model with a smaller number of parameters and a second model with a larger number of parameters and computational reasoning capabilities. By asynchronously combining the complex reasoning capabilities of the second model with the rapid response capabilities of the first model, the problem of simultaneously achieving high content quality and fast response speed in intelligent assistants is solved. Time-consuming and computationally intensive content generation is pre-processed by the second model before the user request, thereby ensuring the depth, accuracy, and richness of the response content. When a user initiates an interaction request, the first model is used to quickly retrieve pre-generated relevant information assets based on the interaction request and synthesize them with context data and the interaction request. This allows for rapid and accurate matching of high-quality response information to the user's interaction request, while significantly reducing the perceived interaction latency. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram illustrating an application scenario of a human-computer interaction response method according to an embodiment of this application; Figure 2 This is a flowchart of a human-computer interaction response method according to an embodiment of this application; Figure 3 This is a structural block diagram of a human-computer interaction response device according to an embodiment of this application; Figure 4 This is an architectural diagram of a human-computer interaction response device according to an embodiment of this application; Figure 5 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0022] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0023] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0024] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0025] For ease of understanding, some of the terms used in this application are explained below: LLM (Large Language Model): refers to a deep learning model trained on massive amounts of text data that has the ability to understand and generate natural language.

[0026] Expert Group: refers to a collection of large, highly intelligent, multimodal language models that are responsible for performing computationally intensive, high-quality content generation tasks in the background.

[0027] Low-latency model: refers to a small language model that has been optimized for latency. It is characterized by fast response speed and low computational resource consumption, and is responsible for real-time interaction with the user on the front end.

[0028] Proactive pre-enactment refers to the process by which the system, before receiving explicit instructions from the user, calls upon a team of experts to generate relevant content based on predictions of the user's future needs.

[0029] Information Assets: These are structured, high-information-density content units pre-generated by a team of experts. Examples include a complete weather report, a drafted email, or a data block containing map links and contact information.

[0030] Probabilistic event pruning refers to an algorithmic mechanism used to evaluate and filter potential future user needs (events), aiming to prioritize the allocation of computing resources to the most likely events.

[0031] Context refers to various dynamic or static data related to the user that can be used to infer their intentions and needs, including but not limited to geographical location, time, calendar schedule, conversation history, device status, ambient audio, etc.

[0032] In related technologies, intelligent assistants based on large language models face a contradiction between model capabilities and response speed in applications. Large models with powerful reasoning and content generation capabilities are usually computationally intensive, resulting in high response latency and making it difficult to meet the demands for real-time interaction in mobile scenarios. Smaller models designed for rapid response, while having low latency, lack cognitive depth and the quality of generated content, limiting the assistant's intelligence level. Existing technologies such as model compression and speculative decoding have not provided a complete solution that allows intelligent assistants to dynamically and on-demand utilize the deep intelligence of large models while maintaining low interaction latency.

[0033] This application provides a method for accelerating the response of intelligent assistants by utilizing proactive, probabilistic event pre-visualization. The core of this method lies in establishing a dynamic multi-model collaborative system, where the time-consuming content generation process is completed by a second model before the user's request. The user's real-time request only needs to be quickly retrieved and synthesized through the first model, thereby achieving an immediate and high-quality response.

[0034] Figure 1 This is a schematic diagram illustrating an application scenario of a human-computer interaction response method according to an embodiment of this application, such as... Figure 1 As shown, the solution in this application can be applied to portable smart devices such as head-mounted displays (AR glasses, VR glasses, XR glasses).

[0035] In this application environment, the user wears AR glasses as the terminal device. This device continuously collects dynamic information about the user's context, such as location, time, calendar events, and ambient audio.

[0036] Before the user initiates an interaction request, the backend predicts a set of priority events that the user is likely to experience in the future based on the context data. Then, the terminal device establishes communication with the cloud server. Based on the set of priority events, the cloud server asynchronously pre-generates high-quality information assets (such as a complete weather report document, a traffic guide, or a drafted email) through a second model with a high number of parameters and computational reasoning capabilities, and stores these information assets in a multimodal database.

[0037] When the user communicates via voice (which can be accessed through...) Figure 1 When a request is initiated via microphone capture or gesture, the device activates a lightweight first model, retrieves information from pre-built information assets, synthesizes the retrieved information into response information, and finally presents multimodal perceptible signals to the user through the AR glasses' screen or speaker based on the response information.

[0038] This process significantly reduces the perceived latency by decoupling the time-consuming and costly content generation from real-time front-end interaction, while also ensuring the depth and quality of the response content, achieving the beneficial effect of combining complex reasoning capabilities with generation speed.

[0039] Figure 2 This is a flowchart of a human-computer interaction response method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: Step S201: Determine the user's context data; In one embodiment, a context monitoring module deployed on a user device (e.g., a smartphone, smart glasses, etc.) continuously and with low power collects multiple context data sources. Context data refers to various dynamic or static data related to the user that can be used to infer their intentions and needs. These data sources may include, but are not limited to: Time and geographic information: such as the current system time, date, and the user's real-time geographic location coordinates (obtained via GPS or network positioning).

[0040] User schedules and reminders: For example, accessing the calendar app on a user's device to get upcoming calendar events, meeting reminders, flight information, etc.

[0041] Communication and Documents: For example, keywords in a user's recent emails, text messages, or instant messages, or the content of documents recently accessed or downloaded.

[0042] Device and environmental status: such as device battery level, network connectivity (Wi-Fi / 5G), device orientation, motion sensor data (to determine whether the user is stationary, walking, or driving), and ambient audio characteristics acquired through the microphone (to determine whether the user is in a quiet office or a noisy street).

[0043] Interaction history: The user's past conversations with the intelligent assistant, query preferences, etc.

[0044] The context monitoring module continuously collects multimodal context data. Specifically, at any time t, the user's context can be uniformly represented as a structured or semi-structured set. :

[0045] in, Represents a piece of context data, for example =(Type: Geographic Location, Value: Coordinates X, Y), =(Type: Calendar, Value: "3 PM, Flight MU5188"). This representation provides standardized input for subsequent probabilistic event pruning.

[0046] Through step S201, the user equipment continuously and with low power collects various dynamic or static contextual data related to the user that can be used to infer their intentions and needs, providing a data foundation for the pre-rehearsal phase.

[0047] Step S202: Process the context data using a probabilistic event pruning mechanism to obtain a priority event set; This step is used to obtain the context data from step S201. The system predicts and filters out user events that are likely to occur in the future, thus obtaining a priority event set.

[0048] In one exemplary embodiment, a probabilistic time pruning mechanism can be activated periodically (e.g., every 5 minutes) or when the context changes significantly (e.g., the user's geographic location moves rapidly or a new calendar event is about to begin). Its goal is to calculate the probability score for a set of predefined (e.g., common built-in system needs such as "check the weather" or "navigation") or dynamically generated (e.g., specific needs inferred from context, such as "check in for flight MU5188") candidate events, and to select target events with probability scores greater than a preset threshold to form a priority event set.

[0049] In one specific implementation, the specific process of this step is as follows: S202.1: Add, remove, or replace the constituent elements of the context data to generate multiple sets of perturbation contexts; To explore the impact of different contextual elements on future events, this step examines the current context. Apply a perturbation, which can be a... The constituent elements can be added, removed, or replaced.

[0050] For example, if Includes {Location: Company, Time: 12:00, Calendar: 3 PM Flight}.

[0051] Disturbance 1 (replacement): ={Location: Airport, Time: 12:00, Calendar: 3 PM Flight}.

[0052] Disturbance 2 (increased): ={Location: Company, Time: 12:00, Calendar: 3 PM Flight, Search History: "Airport Bus"}.

[0053] Disturbance 3 (removed): ={Location: Company, Time: 12:00}.

[0054] In this way, a subset of its elements are selected to form multiple perturbation context sets. (Where j = 1 - N, and N is the total number of perturbations).

[0055] In an alternative implementation, the Shapley Value algorithm can be introduced to improve the effectiveness of the perturbation.

[0056] First, based on the influence of each component of the context data on the potential candidate events, the semantic influence of each component is calculated. The potential candidate events are output events generated during the Shapley value calculation process for provisional analysis.

[0057] For example, historical data analysis revealed that the influence of "calendar" events on predicting "traffic" demand (i.e., semantic influence) is far greater than that of "device battery level." Based on semantic influence, priority is given to selecting target components to be processed (i.e., elements with high influence, such as "calendar" and "location"), and these selected target components are added, removed, or replaced to generate multiple sets of perturbation contexts.

[0058] S202.2: Using the pre-model, predictions are made based on each set of perturbation contexts, and multiple candidate events for the user in the future are output.

[0059] Perform N probes. In the j-th probe, the perturbed context is... As input, query a dedicated pre-model. This is used for prediction. The pre-simulation model... It is a medium-sized language model whose task is to predict candidate events for a user's possible needs based on perturbation context data.

[0060] In one exemplary embodiment, the probing question could be: "In this context, what is the user's most pressing need?" or "What might the user do next?".

[0061] Pre-model Output one or more candidate events. For example, based on (Location is at an airport), the model might output {"Query flight status", "Find gate"}; based on (After searching for airport buses), the model might output {"Plan a route to the airport", "Query bus timetable"}.

[0062] S202.3: Based on a probabilistic event pruning mechanism, determine the set of priority events from multiple candidate events; First, for each potential candidate event (For example ="Check flight status", Configure the corresponding instruction function for "planning the route to the airport".

[0063] Optionally, define an indicator function. When the model by The input is keyword matching or semantic similarity, and the output content contains the event. When the value is 1, its value is 1; otherwise, it is 0.

[0064] It is understandable that, through the indicator function, after N predictions, the perturbation detection frequency of each candidate event is statistically analyzed. Furthermore, the event... probability score It is estimated by using a pre-defined computational model (e.g., averaging), that is, calculating its frequency of occurrence in all N disturbance probes:

[0065] Finally, set the preset judgment threshold. (e.g., 0.6), all satisfying The candidate events are selected to form the priority event set (i.e., the priority event set). .

[0066] For example, {"Check flight status", "Plan route to the airport"} were selected, while {"Book dinner"} was selected. If it is, it will be pruned and will not be rehearsed.

[0067] In step S202, the pre-simulation model is used to make predictions based on the perturbation context, providing a sufficient number of candidate events for probabilistic event pruning. Furthermore, through the probabilistic event pruning mechanism, the probability score of each candidate event is calculated and filtered using a judgment threshold to obtain a set of priority events in which users are likely to have interaction needs in the future. This ensures that in the process of generating information assets using the second model, computing resources are only used for high-value pre-simulation tasks.

[0068] Step S203: Send the priority event set to the second model for active pre-simulation processing to obtain multiple information assets; In this embodiment, the second model can be an expert group or a single general-purpose artificial intelligence model with a high number of parameters and high reasoning ability.

[0069] In one implementation, the second model is an expert group, which is a collection of multiple highly intelligent large language models (LLMs) responsible for performing computationally intensive, high-quality content generation tasks in the background. For example, the expert group may include models for text generation, image processing, and models for logical reasoning, with multiple models working together to generate multimodal information assets.

[0070] In another implementation, the second model can also be a general artificial intelligence model deployed on a cloud server, which has multiple functions such as text generation, image processing and logical reasoning.

[0071] It should be noted that, regardless of the specific form adopted, the second model in this embodiment consumes more computational resources, has a larger parameter scale, and produces higher quality content than the first model with a lightweight architecture in the subsequent real-time interaction stage. In this embodiment, the second model is configured to perform deep language reasoning and content generation based on a priority event set, thereby pre-producing structured, high-information-density information assets.

[0072] In one specific embodiment, the step of generating information assets is completed collaboratively by the task assignment module and the expert team.

[0073] First, the task assignment module will prioritize the event set. Each candidate event in This is transformed into a generative task that can be executed by a large language model. .

[0074] Among them, the generation task It typically contains structured instructions, including: descriptions of each candidate event in the priority event set (e.g., "task type: route planning"), and current context data (e.g., "starting point: {company coordinates}, ending point: {airport coordinates}, mode: driving").

[0075] Furthermore, the task assignment module will generate this task data. The data is then distributed to multiple large language models within the expert team.

[0076] The expert team consists of various large language models. Each model possesses different specializations (e.g., Model A excels at API calls and data summarization, Model B excels at route planning, and Model C excels at email composition). These large language models perform generation tasks in parallel, generating multiple information assets. .

[0077] In this embodiment, the information asset is a structured, high-information-density content unit, rather than a simple text reply; wherein, the information asset includes at least a first information asset, a second information asset, and a third information asset, and the specific generation method is as follows: 1. Generate first information assets based on geographic information: When a priority event involves spatial movement or location query, obtain context-related geographic data (such as current location, destination, real-time traffic conditions) and pre-generate first information assets containing route planning information or location information.

[0078] For example, for the task of "planning a route to the airport," the large language model A in the expert team calls the map API, combines it with the current real-time traffic conditions, plans the best route, estimates the cost, and generates a transportation guide document containing map links, travel time, and cost, as the primary information asset. .

[0079] 2. Generating Secondary Information Assets Based on Document Data: When priority events involve the acquisition or analysis of text information, the document data in the context (such as papers, poster lists) is processed in depth to pre-generate secondary information assets containing text summaries, text screening results, or text contribution information.

[0080] For example, the large language model B in the expert panel extracts the core arguments of the presentations and papers to generate a "one-page summary," and selects highly relevant conference posters based on the user's research field and summarizes their contributions, serving as a second information asset. .

[0081] 3. Generate third-party information assets based on user preferences: When priority events involve interpersonal communication, third-party information assets containing communication drafts or content templates are generated in advance based on the user's language style and preferences (such as formality and common phrases).

[0082] For example, in the expert team, the large language model C, for the event of "notifying colleagues," pre-drafts an email draft containing specific flight numbers and arrival times, serving as a third-party information asset. .

[0083] Through the above step S203, the second model with high computational reasoning ability is used to actively rehearse priority events in order to generate structured information assets in advance; the time-consuming calculation and real-time interaction are decoupled, and the potential needs of users are adapted in advance while ensuring the depth and quality of the content.

[0084] Step S204: Store the multiple information assets generated by the second model into a multimodal database.

[0085] In order to facilitate rapid retrieval of the first model in the subsequent process, an index needs to be configured for the information assets during storage.

[0086] Specifically, the index information may include: Event types: such as "transportation", "flights", "mail"; Event keywords: such as "MU5188", "airport", "Shanghai colleague"; Event creation time: the creation timestamp of the asset. Event feature vector: a semantic vector generated by encoding the content of the information asset using an embedding model, used to support semantic retrieval.

[0087] Store the information assets with configured indexes in a multimodal database.

[0088] In one embodiment, the multimodal database can be a cloud-native multimodal database (such as Lindorm) that supports vector retrieval, full-text retrieval, and structured data fusion retrieval to ensure that the first model can query efficiently.

[0089] Step S204 configures multidimensional indexes for pre-generated assets and stores them in a multimodal database to build an efficient searchable resource pool; this can reduce the query and retrieval time of the first model and further improve the speed and matching accuracy of calling high-quality content in real-time interaction.

[0090] Step S205: Upon receiving an interaction request from the user, the first model is used to determine the relevant information asset that matches the interaction request from multiple information assets. The relevant information asset, the interaction request, and the context data are processed to generate response information, which is then output as a user-perceptible signal.

[0091] In this embodiment, the first model is a low-latency model deployed on the user terminal side or edge computing nodes, configured to perform information asset retrieval tasks and response information synthesis tasks. Some evaluation parameters of the first model in this embodiment are smaller than those of the second model. Specifically, some evaluation parameters can be limited to one or two specific evaluation parameters. Due to the different evaluation parameters, the differences between the first and second models are specifically reflected in the parameter magnitude, emergence ability, generalization ability, and task boundaries.

[0092] Specifically, in one exemplary embodiment, when a user issues an interaction request through a user interface module (e.g., voice or text input), When (for example, when a user says, "Heading to the airport"), the response process is triggered, and the specific execution process includes: S205.1: Understand the interaction intent based on the interaction request using the first model.

[0093] In this embodiment, the first model is a small language model that has undergone latency optimization and has the characteristics of high response speed. Optionally, the latency optimization of the small language model can be carried out by means of model compression, inference thickening, and deployment environment adaptation processing.

[0094] The first model receives user requests. and current context Firstly, based on interactive requests and context This allows for quick understanding of the user's true intent in the current context. For example, understanding the intent of "going to the airport" as "needing transportation routes and flight information" is crucial.

[0095] S205.2: The first model retrieves relevant information assets that match the interaction request from a multimodal database.

[0096] This retrieval process is a comprehensive query that can simultaneously utilize keyword indexes, event type indexes, and semantic vector indexes. For example, it can retrieve information assets that have "event type = transportation OR event type = flight" and "generated within the past 1 hour" and "semantic vector similar to 'going to the airport'".

[0097] Using the first model, based on the index and semantic information of the interaction request and information assets, a retrieval is performed in the multimodal database to obtain relevant information assets matching the interaction request. Specifically, in one exemplary embodiment, related information assets Includes traffic guides generated during the rehearsal and flight status .

[0098] S205.3: Process relevant information assets, interaction requests, and context data to obtain response information.

[0099] The first model is based on relevant information assets. Interaction Request and context data Generate response information ;

[0100] Compared to related technologies, this step does not generate response information from scratch based on a localized small model in real time; instead, it uses a first model to "orchestrate and integrate" relevant information assets pre-generated from a second model to quickly generate high-quality response information.

[0101] In one alternative embodiment, an example of the natural language indicated by the response message is: "Flight MU5188 is currently in normal condition and is scheduled to depart on time. Based on current traffic conditions, we recommend that you depart immediately, which will take approximately 45 minutes to reach the airport. I have already planned a route for you; would you like me to call a taxi for you?" S205.4: Output the response information as a user-perceptible signal.

[0102] Optionally, the response information in this embodiment is a Domain Specific Language (DSL), which is configured to describe the specific syntax of multimodal content layout and elements such as mixed text and graphics. This specific syntax can be parsed and rendered by a visual interface (the mechanism can be analogous to HTML or Markdown), or parsed by audio output devices such as headphones and speakers to output user-perceptible signals.

[0103] In an exemplary embodiment, based on the response information, multimodal content such as mixed text and images can be presented on a specific visualization interface (such as a smart glasses lens using a full-color optical waveguide scheme). The mixed text and images content may include text, full-color images and videos, etc.

[0104] Furthermore, in this embodiment, the user-perceptible methods include, but are not limited to: Visual output: Presenting responses in the form of text, structured cards, or a mix of text and graphics through the display screen of a device (such as a smartphone or smart glasses); Auditory output: The text-to-speech (TTS) engine is invoked to broadcast the response information through the device's speaker or connected headphones (such as Bluetooth headphones), which is suitable for driving or multitasking scenarios. Haptic output: Providing key cues or confirmation information in specific vibration patterns via the device's vibration motor (such as a smartwatch or mobile phone).

[0105] In this embodiment, through multimodal output design, the method is not limited to a single screen interaction and can be flexibly adapted to various terminals such as smart speakers, in-vehicle systems, and wearable devices; the user can intelligently select the most suitable output method according to the user context, which significantly enhances the applicability and interactive experience of this solution.

[0106] In step S205, the low-latency first model is used to retrieve and orchestrate pre-set information assets to generate a response, effectively avoiding the computational time consumption of real-time deep generation. While achieving millisecond-level instant interactive response, it outputs response signals with high content richness and accuracy.

[0107] Step S206: Optimize the model based on user feedback signals regarding response information; To continuously optimize its predictive capabilities, embodiments of this application also provide a feedback loop, specifically including: After generating response information and outputting it as a user-perceptible signal, the feedback signal corresponding to the response information is obtained, wherein the feedback signal may include: Implicit feedback: Messages in the response indicating that an information asset has been adopted by the user. For example, recording that an information asset (travel guide) has been clicked by a user to view it or used to initiate navigation is considered positive feedback.

[0108] Explicit feedback: User satisfaction with the response, such as clicking the "satisfied" button or giving verbal praise. Secondly, these feedback signals are used to adjust the parameters of the probabilistic event pruning mechanism through reinforcement learning algorithms. For example, successfully predicted and used pre-exercises can be used as positive rewards to update the computational parameters of the preset computational model, or the preset judgment threshold can be dynamically adjusted.

[0109] Through step S206, optimization information can be learned from historical experience. This information is used to optimize the parameters involved in the probabilistic event pruning process, making the response information increasingly accurate in predicting the personalized needs of specific users over time, thus forming an adaptive closed-loop intelligent system.

[0110] Through steps S201 to S206 above, this solution addresses the contradiction in existing intelligent assistants where it's difficult to simultaneously achieve high intelligence and high content generation quality in large-scale models while meeting the low-latency response requirements of mobile interaction. This solution employs an active pre-simulation approach, analyzing user context through a probabilistic event pruning mechanism to predict a set of high-probability future priority events. Furthermore, it instructs a second model to asynchronously pre-generate multiple information assets corresponding to the priority event set and store them in a multimodal database. When a user interaction request is received, the first model deployed on the front end only needs to quickly retrieve the pre-generated relevant information assets from the multimodal database, orchestrate and synthesize them, and then output the response information quickly and accurately. This solution decouples the high-cost deep generation process from the real-time interaction link on the front end, reducing user-perceived interaction latency while ensuring both high quality and depth in the response content.

[0111] The methods of the embodiments of this application will be further described in conjunction with specific application scenarios.

[0112] Scenario Example 1: Business Arrangements Suppose the user is a business person whose smartphone calendar shows a flight to Shanghai at 3 PM, and the current time is 12 PM.

[0113] (1) Context monitoring: Monitored user context data: {Current time: 12:00 PM, Location: Company, Calendar event: 3 PM, Flight MU5188 to Shanghai, Recent emails: Meeting arrangements with Shanghai branch}.

[0114] (2) Probabilistic Event Pruning: The probabilistic pruning module is activated. It performs perturbation analysis on the context data. For example, the model simulates scenarios like "What would happen if a user searched for 'airport' now?" or "What would happen if a user opened a ride-hailing app?". High-frequency potential events output by the model include: "Checking flight status," "Planning a route to the airport," "Notifying colleagues in Shanghai of flight information," and "Drafting an automated reply email during a business trip." These events receive high probability scores, exceeding a preset threshold, and are therefore deemed to require rehearsal. Events such as "Booking dinner" and "Checking local weather" score lower and are pruned.

[0115] (3) Proactive rehearsal: The expert team receives the set of priority events and begins work asynchronously: Model A calls the travel API to obtain the real-time status (on time / delayed) of MU5188 and generates a summary document containing the boarding gate and departure time, which is saved as information asset 1.

[0116] Model B combines real-time traffic conditions to plan the best route from the company to the airport, estimates taxi fares, and generates a transportation guide that includes a map and travel time, which is stored as information asset 2.

[0117] Model C drafts an email based on the user's language habits: "Hello [colleague name], my flight MU5188 is scheduled to depart on time. I look forward to meeting you this afternoon." and saves it as information asset 3.

[0118] These information assets are stored in the cloud-native multi-model database Lindorm[1]. Lindorm provides one-stop data services such as low-cost storage, online query and offline analysis for massive amounts of general time-series, semi-structured and unstructured data. It supports forward sorting, full text, vector fusion retrieval and AI inference capabilities for AI scenarios; and is compatible with open source standards such as MySQL protocol, HBase, ES, Hive, Spark, HDFS.

[0119] (4) User interaction and rapid response: At 1 p.m., the user picks up his phone and says to the smart assistant, "Let's go to the airport." The first model is activated. After understanding the user's instruction, the smart assistant immediately retrieves information asset 1 and information asset 2 related to "airport" and "flight" from the multimodal database.

[0120] Since the response is not generated from scratch, the user response process is simply a rapid integration through the first model. Therefore, the response can be synthesized in a short time and displayed on the screen as: "Flight MU5188 is currently in normal status and is expected to depart on time. Based on current traffic conditions, we suggest you depart immediately, which will take approximately 45 minutes to reach the airport. I have already planned a route for you; would you like me to call a taxi for you?" If the user then says, "Send a message to my colleague in Shanghai," the first model immediately retrieves information asset 3 and asks: "Okay, would you like to send the following content to [colleague's name]?" In this example, all the time-consuming data retrieval, route planning, and email composition were completed in advance. The user's request received an informative, highly intelligent, and almost instantaneous response.

[0121] Scenario Example 2: Attending an Academic Conference (1) Context monitoring: Monitor the user's context data, including: {Calendar: Attended the AI ​​Frontier Summit all day, Location: Conference Center, Email / Document: Downloaded multiple conference papers, Contacts: Marked multiple attendees as "Want to get to know"}.

[0122] (2) Probabilistic event pruning: The module predicts priority events, such as: "Generate a summary for the next speech", "Query the current location of tagged contacts", "Find poster displays related to your research field", "Draft a meeting request email with an expert".

[0123] (3) Proactive rehearsal: The expert team receives the list of priority events and begins work asynchronously: Model A reads the paper for the next presentation in the user's calendar and generates a one-page summary containing the core arguments, experimental results, and potential problems, as information asset 1.

[0124] Model B (multimodal model) obtains the indoor map of the venue and the real-time location API of the attendees (if available), and generates a map that marks the location of the user and the contacts they want to meet, as information asset 2.

[0125] Model C scans all the poster displays and abstracts, selects the three most relevant to the topics of the papers downloaded by the user, and summarizes their contributions as information asset 3.

[0126] Model D drafts a meeting request email template based on the user's style, as information asset 4.

[0127] (4) User Interaction and Rapid Response: After a speech, the user says to the smart glasses, "Summarize the next speech." The first model immediately retrieves information asset 1 from the multimodal database and presents it in audio or text form. At lunch, the user asks, "Where are the people I marked?" Information asset 2 is immediately displayed. The user then says, "Help me schedule an appointment with Professor Li," and the system retrieves information asset 4 and asks if a specific time needs to be entered before sending.

[0128] (5) This scenario demonstrates the ability of the proposed solution to handle complex, multimodal, and specialized scenarios. By preparing deeply customized information assets in advance, it greatly improves the work efficiency of users as professionals.

[0129] On the other hand, embodiments of this application also provide a human-computer interaction response device. Figure 3 This is a structural block diagram of a human-computer interaction response device according to an embodiment of this application, such as... Figure 3 As shown, the device includes: a retrieval module 30 and an output 31, wherein: The retrieval module 30 is used to determine, in the case of receiving an interaction request input by the user, the relevant information asset matching the interaction request from multiple information assets through a first model; It should be noted that the information asset is generated asynchronously and concurrently in advance by analyzing the user context through a probabilistic event pruning mechanism before receiving the interaction request, predicting the set of priority events that are likely to occur in the future, and using a second model with a high number of parameters and high inference capabilities; optionally, multiple information assets are stored in a multimodal database. The second model in this embodiment can be an expert group, which includes multiple large language models; The output module 31 processes the relevant information assets, interaction requests and context data through the first model to generate response information; and outputs the response information as a user-perceptible signal.

[0130] In this embodiment, the first model is a low-latency model deployed on the user terminal side or edge computing nodes, configured to perform information asset retrieval tasks and response information synthesis tasks. The capabilities of the first model in this embodiment are less than those of the second model, and this difference is specifically reflected in the magnitude of parameters, emergence capability, generalization capability, and task boundaries.

[0131] When a user interaction request is received, the first model only needs to quickly orchestrate and synthesize these pre-built relevant information assets and context data to output response information.

[0132] This human-computer interaction response device can decouple the high-cost deep generation process from the user's real-time interaction link, thereby reducing the user's perceived interaction latency while ensuring that the response content has both the high quality and depth of a large model.

[0133] also, Figure 4 This is an architectural diagram of a human-computer interaction response device according to an embodiment of this application, such as... Figure 4 As shown, the device includes: The context monitoring module (101) is deployed on the user device and is responsible for continuously and with low power to collect multiple context data sources, including but not limited to geolocation, time, calendar schedule, conversation history, device status, ambient audio, etc.

[0134] The probabilistic event pruning module (102) is responsible for analyzing contextual data, predicting and filtering out high-probability future user events and needs.

[0135] The task assignment module (103) is responsible for packaging the priority events selected by the pruning module into specific task data and concurrently assigning them to the expert team.

[0136] The expert team (104) can be deployed in the cloud or public network data center. It consists of multiple highly intelligent, multimodal large models and is responsible for executing task data and producing information assets.

[0137] A multimodal database (105) is used to store information assets generated by the expert team and to create an index for fast retrieval.

[0138] The small language model (106), a lightweight language model optimized for latency, is used to handle real-time user requests and interact with multimodal databases.

[0139] The user interface module (107), deployed on the user device, is responsible for receiving user input (voice, text, etc.) and outputting the final synthesized response. This module stores the interaction information in the context monitoring module.

[0140] In one embodiment, when the above-mentioned device is working, the context monitoring module (101) collects data, the probabilistic event pruning module (102) filters events, the task assignment module (103) distributes tasks, and the expert team (104) generates information assets and stores them in the multimodal database (105). When a user initiates a request through the user interface module (107), the first model (106) retrieves the assets in (105) and synthesizes a response, which is then returned to the user through (107).

[0141] In one embodiment, Figure 5 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 5 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 5 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operating system, the computer programs are executed by the processor to implement a human-computer interaction response method, and the database stores data.

[0142] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0143] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A human-computer interaction response method, characterized in that, The method includes: Upon receiving an interaction request from a user, a first model is used to determine relevant information assets that match the interaction request from multiple information assets. These multiple information assets are pre-generated using a second model based on the user's context data in an active rehearsal manner. Some evaluation parameters of the first model are smaller than those of the second model. The first model processes the relevant information assets, the interaction request, and the context data to generate response information. The response information is output as a signal that can be perceived by the user.

2. The method according to claim 1, characterized in that, The methods for generating the aforementioned multiple information assets include: Determine the context data; The context data is processed using a probabilistic event pruning mechanism to obtain a priority event set; The set of priority events is sent to the second model for active pre-rehearsal processing to obtain the multiple information assets. The second model is an expert group, which includes multiple large language models.

3. The method according to claim 2, characterized in that, The context data is processed using a probabilistic event pruning mechanism to obtain a priority event set including: By adding, subtracting, or replacing the constituent elements of the context data, multiple perturbation context sets are generated; By using the pre-simulation model, predictions are made based on each of the aforementioned perturbation context sets, and multiple candidate events for the user in future time periods are output. Based on a probabilistic event pruning mechanism, the priority event set is determined from among the multiple candidate events.

4. The method according to claim 3, characterized in that, Based on a probabilistic event pruning mechanism, the set of priority events is determined from among the multiple candidate events, including: Configure a corresponding indicator function for each of the candidate events; The perturbation detection frequency of each candidate event is statistically analyzed using the indicated function. By using a preset computational model, the probability score of each candidate event is estimated based on the disturbance detection frequency, and candidate events with probability scores greater than a preset judgment threshold are combined into the priority event set.

5. The method according to claim 3, characterized in that, By adding, removing, or replacing elements of the context data, multiple perturbation context sets are generated, specifically including: The Shapley value algorithm is used to calculate the degree of influence of each component of the context data on the potential candidate events. The semantic influence of each component element is calculated based on the degree of influence of each component element. Based on the semantic influence, the target constituent elements to be processed are determined, and multiple sets of the perturbation contexts are generated by adding, subtracting or replacing the target constituent elements.

6. The method according to claim 2, characterized in that, The set of priority events is sent to the second model for active pre-simulation processing, resulting in the following multiple information assets: The events in the priority event set are converted into generation tasks and assigned to the corresponding large language models in the expert group; The expert team comprises multiple large language models that perform the generation task to generate the multiple information assets.

7. The method according to claim 6, characterized in that, The plurality of information assets includes at least one of the following: a first information asset, a second information asset, and a third information asset, wherein, The methods for generating the first information asset include: Obtain geographic information associated with the above and below data, and generate a first information asset in advance using an active pre-simulation method based on the geographic information. The first information asset includes: route planning information and location information. The methods for generating the second information asset include: Based on the document data in the context data, a second information asset is generated in advance in an active rehearsal manner. The second information asset includes: text summary, text filtering results, and text contribution information. The methods for generating the third information asset include: Based on user preferences and language styles in the context data, a third information asset is generated in advance in an active rehearsal manner. The third information asset includes: communication drafts and content templates.

8. The method according to claim 1, characterized in that, After generating multiple information assets in advance using the user's context data through a second model in a proactive rehearsal manner, the method further includes: The multiple information assets are acquired, and an index is configured for each information asset based on the event type, event keywords, event generation time, and event feature vector associated with each information asset. Store the information assets with configured indexes in a multimodal database; Using the first model, the relevant information assets are retrieved from the multimodal database based on the index, the interaction request, and the semantic information of the information assets.

9. A human-computer interaction response device, characterized in that, The device includes: The retrieval module is used to determine relevant information assets matching the interaction request from multiple information assets through a first model when receiving an interaction request input by a user. The multiple information assets are pre-generated by a second model using the user's context data in an active pre-rehearsal manner. Some evaluation parameters of the first model are smaller than those of the second model. The output module is configured to process the relevant information assets, the interaction request, and the context data through the first model to generate response information; and to output the response information as a user-perceptible signal.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images