Context-aware based interaction method, system, device, and medium

Abstract

The application relates to the technical field of artificial intelligence, and specifically provides an interaction method, system, device and medium based on context perception, which comprises the following steps: collecting multi-modal data, and performing pretreatment and feature extraction on the multi-modal data to obtain feature vectors of various modes; dynamically modeling and updating historical interaction features, environmental state features and user portrait features based on a graph neural network to generate a context state vector; performing cross-modal feature fusion on the feature vectors of the various modes based on an attention mechanism to generate a fusion feature vector, wherein weight calculation of the attention mechanism depends on the context state vector; generating an interaction strategy through a reinforcement learning algorithm according to the fusion feature vector and the context state vector, and executing a corresponding multi-modal interaction response. The application greatly enhances the overall robustness of an intelligent agent in a strange or unstable environment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a context-aware interaction method, system, device, and medium. Background Technology

[0002] In AI-driven multimodal interaction systems, fusing text, speech, image, and sensor data is crucial for achieving accurate semantic understanding and intelligent decision-making. Existing technologies widely employ attention mechanisms and other methods for cross-modal feature fusion, aiming to uncover the complementarity between different modal information. However, these methods typically treat contextual information (such as user state and environmental parameters) as static or independent auxiliary features, simply concatenating them with features from various modalities or using them as fixed inputs, failing to establish a real-time, tightly coupled feedback control mechanism.

[0003] Specifically, traditional methods fail to take the dynamically evolving context state as the core basis when calculating fusion weights, and fail to adjust the contribution of different modalities to the final decision in real time and explicitly. This results in the fusion process lacking sufficient adaptability and specificity when the system encounters sudden environmental changes, user intent shifts, or unfamiliar scenarios that it has not been trained on before. It is prone to semantic understanding bias or decision lag, which seriously restricts the interaction accuracy and overall robustness of the agent in open and dynamic real-world scenarios.

[0004] Therefore, how to enable the cross-modal fusion process to deeply perceive and respond in real time to dynamically changing contextual information has become a technical bottleneck that urgently needs to be solved to improve the performance of the next generation of intelligent interactive systems. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the prior art, the present invention provides a context-aware interaction method, system, device and medium to solve the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a context-aware interaction method, comprising: Collect multimodal data, and preprocess and extract features from the multimodal data to obtain feature vectors for each modality; Based on graph neural networks, historical interaction features, environmental state features, and user profile features are dynamically modeled and updated to generate context state vectors. Based on the attention mechanism, cross-modal feature fusion is performed on the feature vectors of each modality to generate a fused feature vector, wherein the weight calculation of the attention mechanism depends on the context state vector; Based on the fused feature vector and the context state vector, an interaction strategy is generated through a reinforcement learning algorithm, and the corresponding multimodal interaction response is executed.

[0007] In an optional implementation, the method further includes, when acquiring multimodal data: A high-precision hardware clock source is set as the global master clock. All sensors receive the global acquisition trigger signal from the master device through a unified hardware trigger line to achieve hardware-level synchronization of acquisition start-up. When each sensor collects each frame of data, in addition to generating its own local timestamp, it also synchronously records the precise moment when it receives the synchronization pulse of the global master clock. Based on the local timestamp sequence recorded by each sensor and the corresponding synchronization pulse time sequence, the drift rate and initial offset of the local clock of each sensor relative to the global master clock are periodically fitted. For each frame of data acquired by each sensor, the local timestamp is compensated using the estimated drift rate and initial offset of that sensor, and an accurate timestamp aligned with the global master clock is calculated.

[0008] In an optional implementation, the multimodal data is preprocessed and feature extracted to obtain feature vectors for each modality, including: Based on the current context state vector and real-time monitored system resource status indicators, a dynamic configuration strategy is generated for the preprocessing of each modality; wherein, the context state vector includes at least the interaction task type and environment state characteristics; the system resource status indicators include at least the processor computing load and available memory; and the dynamic configuration strategy includes at least the processing algorithm selected for each modality, the set processing parameters, and the assigned processing priority. According to the dynamic configuration strategy, each modality-specific preprocessing unit is controlled to perform corresponding adaptive preprocessing operations, including: for sensor data, processing is performed using a noise reduction algorithm combining time-series prediction and a real-time anomaly detection algorithm selected based on the strategy; for text data, the corresponding model is dynamically selected and called between a lightweight feature extraction model and a high-precision deep semantic model for processing according to the strategy; for image or video data, feature extraction is performed after dynamically adjusting the resolution or sampling frame rate according to the strategy. The quality of each modal data after preprocessing is evaluated, and optimization processing or re-acquisition is triggered when the quality is lower than a preset threshold.

[0009] In one optional implementation, a context state vector is generated by dynamically modeling and updating historical interaction features, environmental state features, and user profile features based on a graph neural network, including: Construct a dynamic multimodal context graph, wherein: the nodes of the graph include at least user nodes, device nodes, and environment parameter nodes, and each node is attached with a corresponding feature vector; the edges of the graph represent the entity relationships between nodes and have weights dynamically calculated based on interaction frequency and time; The dynamic multimodal context graph is input into a graph neural network for context representation learning; the graph neural network aggregates the neighbor information of each node to update the node features by introducing an attention mechanism with a time decay function, and generates a global context state vector by aggregating the updated full graph node features. Based on periodic time-series signals and event triggering signals driven by interactive events, environmental changes, or scene switching events, the features of corresponding nodes and the weights of associated edges in the dynamic multimodal context graph are dynamically updated; the updated graph is used for context representation learning in the next time step.

[0010] In an optional implementation, cross-modal feature fusion is performed on the feature vectors of each modality based on an attention mechanism to generate a fused feature vector, including: Based on the current context state vector and the feature vectors of each modality, a set of cross-modal attention weights is calculated; wherein, the attention weight of each modality reflects the importance of the modality feature to the overall semantic contribution in the current context; The calculated cross-modal attention weights are used to perform a weighted summation of the corresponding modal feature vectors to generate the fused feature vector.

[0011] In an optional implementation, a set of cross-modal attention weights is calculated based on the current context state vector and the feature vectors of each modality, including: The first vector, formed by concatenating the feature vector of the target modality with the context state vector, is then mapped to the query vector through the first linear transformation matrix. :

[0012] in, Let be the first linear transformation matrix. Let be the first vector. For the feature vector of the target mode, This is the context state vector; Combine the feature vectors of all modalities with the context state vector, and map them to key vectors respectively through the second linear transformation matrix:

[0013] in, Let j be the key vector corresponding to the j-th mode. , This is the second linear transformation matrix. Let j be the feature vector of the j-th mode; Calculate query vector With each key vector The dot product is used to obtain a preliminary attention score; The attention scores are normalized using the Softmax function to obtain the cross-modal attention weights for the target modality.

[0014] In an optional implementation, the method for concatenating the feature vector and the context state vector includes: Generate a pair of modulation parameters for the context vector C using a lightweight neural network:

[0015]

[0016] in, Scaling factor For bias terms, , Same dimension as the feature vector, and These are the trainable parameters of the neural network; The feature vector is adjusted using scaling factors and bias terms to selectively enhance or suppress the corresponding dimensions of the feature vector:

[0017] The adjusted feature vector The concatenated vector is obtained by juxtaposing and concatenating the original context vector C with the concatenated vector: .

[0018] Secondly, the present invention provides a context-aware interaction system, comprising: The feature acquisition module is used to acquire multimodal data and preprocess and extract features from the multimodal data to obtain feature vectors for each modality. The context learning module is used to dynamically model and update historical interaction features, environmental state features, and user profile features based on graph neural networks, and generate context state vectors. The modality fusion module is used to perform cross-modal feature fusion on the feature vectors of each modality based on an attention mechanism to generate a fused feature vector, wherein the weight calculation of the attention mechanism depends on the context state vector; The strategy generation module is used to generate an interaction strategy based on the fused feature vector and the context state vector using a reinforcement learning algorithm, and to execute the corresponding multimodal interaction response.

[0019] Thirdly, a device is provided, comprising: Memory, used to store context-aware interactive programs; A processor, used to implement the steps of the context-aware interaction method as provided in the first aspect when executing the context-aware interaction program.

[0020] Fourthly, a computer-readable medium is provided, on which a context-aware interactive program is stored, wherein when the context-aware interactive program is executed by a processor, the steps of the context-aware interactive method as provided in the first aspect are implemented.

[0021] The beneficial effects of this invention lie in the fact that the context-aware interaction method, system, device, and medium provided by this invention achieve precise adaptation of the fusion strategy to the current semantic environment by deeply embedding the real-time evolving dynamic context state vector into the core process of cross-modal attention weight calculation. The resulting benefits are mainly reflected in the following aspects: First, the system's interaction accuracy in complex dynamic scenarios (such as sudden environmental changes and user intent shifts) is significantly improved, as the fusion weights can respond to contextual changes in real time, effectively avoiding decision-making biases caused by information lag in traditional methods. Second, the timeliness and relevance of decision-making are enhanced, as modal contribution is adjusted in real time based on precise context, reducing redundant calculations and improving system response speed and resource utilization efficiency. Finally, this method significantly enhances the overall robustness of the agent in unfamiliar or unstable environments, providing a reliable multimodal interaction solution for high-risk, high-dynamic scenarios such as smart healthcare and emergency command. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0024] Figure 2 This is a context-aware interaction flowchart of an embodiment of the present invention.

[0025] Figure 3 This is a schematic flowchart illustrating the calculation of cross-modal attention weights according to an embodiment of the present invention. Figure 4 This is a schematic block diagram of a system according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0028] The context-aware interaction method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the context-aware interaction system runs on the computer device.

[0029] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a context-aware interactive system. Depending on different needs, the order of steps in this flowchart can be changed, and some can be omitted.

[0030] like Figure 1 As shown, the method includes: S1. Collect multimodal data, and preprocess and extract features from the multimodal data to obtain feature vectors for each modality; S2. Based on graph neural networks, historical interaction features, environmental state features, and user profile features are dynamically modeled and updated to generate context state vectors; S3. Based on the attention mechanism, cross-modal feature fusion is performed on the feature vectors of each modality to generate a fused feature vector, wherein the weight calculation of the attention mechanism depends on the context state vector; S4. Based on the fused feature vector and the context state vector, generate an interaction strategy using a reinforcement learning algorithm and execute the corresponding multimodal interaction response.

[0031] refer to Figure 2 The context-aware interaction method of the present invention includes the following detailed process: In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0032] S101. Multimodal data acquisition is achieved through a multi-level synchronization strategy combining hardware synchronization and software compensation, ensuring high-precision temporal alignment of data from different sensors (such as cameras, microphones, temperature and humidity sensors, motion sensors, etc.). The specific implementation steps are as follows: S101.1. Hardware Master-Slave Triggered Synchronization Configure a high-precision, low-drift hardware clock source as the global master clock (e.g., a GPS disciplined clock or a temperature-controlled crystal oscillator). All sensors act as slave devices, connected to the master device via a unified hardware trigger line (e.g., a PPS pulse signal line). When a data acquisition cycle needs to be initiated, the master device simultaneously sends a global acquisition trigger signal to all slave sensors. Upon receiving this hardware trigger signal, each sensor synchronously begins data acquisition, thus physically eliminating initial time deviations introduced by different startup sequences between devices.

[0033] S101.2. Synchronize timestamp associated records During the data acquisition process of each sensor, not only is a local timestamp based on its own internal clock generated for each frame of data, but the sensor also accurately records the precise moment when it receives periodic synchronization pulses (such as the rising edge of the synchronization pulse) from the global master clock. This means that each frame of local data is associated with a high-precision, unified time reference point.

[0034] S101.3. Dynamic Software Estimation and Modeling of Clock Drift Because of slight differences in the internal clock crystals of different sensors, their clocks will still drift over time and temperature even after hardware synchronization is initiated. To address this, the system runs a dynamic drift compensation algorithm in the background. This algorithm periodically (e.g., every 10 seconds) collects the local timestamp sequence reported by each sensor and its associated global synchronization pulse time sequence. For each sensor, the algorithm (e.g., using a linear regression model) uses this data to fit its local clock drift rate and initial offset relative to the global master clock, thereby establishing a dynamic clock error model.

[0035] S101.4. Real-time compensation and alignment of data timestamps During the real-time processing of the data stream, for each frame of data acquired by each sensor, the system calls the clock error model established for that sensor. Using the drift rate and initial offset parameters estimated in the model, the local timestamp attached to the frame of data is compensated in real time and corrected to a unified time coordinate system aligned with the global master clock, resulting in a high-precision globally aligned timestamp.

[0036] S102. Preprocess and extract features from the multimodal data to obtain feature vectors for each modality, including: S102.1. Dynamic Configuration Strategy Generation First, intelligent decision-making is performed to generate the optimal preprocessing scheme for the current moment. The decision engine receives input from two dimensions: one is the current context state vector obtained from the dynamic context model, which at least includes a clear interaction task type (e.g., "emergency medical diagnosis", "daily home control") and precise environmental state characteristics (e.g., "nighttime, low light, user in living room"); the other is real-time resource status indicators obtained from the system monitoring module, such as the current CPU computing load, available memory size, and network bandwidth. Based on this information, the decision engine generates an immediate dynamic configuration strategy for each type of modality data (sensor, text, image / video). This strategy explicitly defines three core elements: ① the processing algorithm selected for each modality (e.g., choosing "face recognition optimization pipeline" instead of "object detection pipeline" for image processing); ② the specific parameters set for the algorithm (e.g., target resolution for image scaling, intensity coefficient of the noise reduction algorithm); ③ the processing priority assigned to each modality task under the current system resource constraints to ensure that critical tasks are processed first.

[0037] S102.2. Policy-Driven Adaptive Preprocessing Execution The preprocessing pipeline controller receives the above dynamic configuration strategy and precisely controls the operation of each modally specialized processing unit accordingly: Sensor data processing unit: For time-series data generated by sensors such as temperature, humidity, and motion sensors, it will invoke a selected combination of algorithms according to a strategy. For example, when resources are abundant and high accuracy is required, wavelet transform and Kalman filtering will be applied simultaneously for noise reduction and prediction; when resources are limited, only basic filtering algorithms may be used. Simultaneously, this unit will integrate and run real-time anomaly detection algorithms (such as the statistical Z-score method) to automatically mark, repair, or remove outlier data points.

[0038] Text Data Processing Unit: This unit incorporates a variety of feature extraction models, ranging from lightweight to high-precision. Based on the semantic complexity indicated by the strategy, it dynamically loads and invokes the appropriate model. For example, for simple commands like "turn on the lights," it directly uses a fast bag-of-words model or FastText to extract keyword features; for long sentences expressing complex emotions or intentions, it dynamically loads large pre-trained models such as BERT for deep semantic encoding. Furthermore, this unit supports real-time cleaning of the text stream converted from speech recognition, such as removing interjections and correcting colloquial expressions.

[0039] Image / Video Data Processing Unit: The core of this unit is to adaptively adjust the "volume" of the input data. Based on the task requirements and system load defined in the strategy, it dynamically determines the downsampling resolution of the image (e.g., reducing from 1080P to 720P to save computation) or the sampling frame rate of the video (e.g., reducing from 30FPS to 5FPS when the scene is static). After adjustment, the corresponding convolutional neural network is invoked for visual feature extraction.

[0040] S102.3. Quality Assessment and Feedback Control After preprocessing, the data for each modality does not flow directly downstream but first enters a quality assessment loop. The assessment module uses lightweight metrics (such as confidence scores for text features, image sharpness metrics, and sensor data smoothness) to quickly score the output quality. If the assessment score is lower than the preset quality threshold for that type of data, feedback control is triggered: a more complex but more effective backup algorithm may be automatically activated to perform "re-processing" optimization on that batch of data; if the data source itself is determined to be of low quality (e.g., severely overexposed images), a request is sent to the data acquisition layer to attempt to re-acquire data. Only data that passes the quality assessment has its extracted feature vectors allowed for subsequent fusion and decision-making.

[0041] In a specific example: In the "Home Security Alert" scenario, the context indicates the task is "Moving Object Detection," and the environment is "Nighttime." Resource monitoring shows high CPU load. Based on this, the decision engine generates a strategy: for image data, a "Moving Target Detection Algorithm" is selected, with parameters set to "resolution reduced to 720p, infrared channel prioritized," and assigned high priority; for door and window sensor data, a "Simple Threshold Alarm Algorithm" is selected, assigned medium priority. The pipeline executes accordingly, effectively reducing the overall system load while ensuring the responsiveness of core security functions. The image quality assessment stage focuses on checking the clarity of motion contours in the processed image to ensure it meets detection requirements.

[0042] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0043] S201. Construction and Initialization of Dynamic Multimodal Context Graph First, construct a graph structure G. t =(V t E t It serves as a carrier of the global context.

[0044] Node (V) t Construction: The nodes in the graph represent different entities and are associated with multimodal feature vectors. This mainly includes: User node (v uIts eigenvector x u It is composed of static user profiles (such as age and preferences) and dynamic behavioral features obtained by encoding recent interaction sequences (clicks, voice commands) through LSTM / Transformer.

[0045] Device node (v d Its eigenvector x d It includes the device's real-time status (such as the brightness and on / off status of a smart light) and its inherent functional capability vectors.

[0046] Environment parameter node (v) e Its eigenvector x e It includes real-time sensor data (temperature, humidity) and trend characteristics derived from time-series analysis (such as "temperature is rising").

[0047] Edge (E) t Construction: Edges represent relationships between entities, with weights w ij (t) is dynamic. For example, the weight of the "control" relationship edge between the user and the device can be calculated by decaying based on the historical interaction frequency and the time of the most recent interaction: Other edge types include “located” relation edges representing spatial co-occurrence, and “triggered” relation edges generated by rule-based or causal discovery algorithms.

[0048] S202. Context Representation Learning Based on Dynamic Graph Attention Network (DGAT) The constructed dynamic graph is fed into a Dynamic Graph Attention Network (DGAT) for deep representation learning to generate a context state vector C(t) containing global semantics.

[0049] Dynamic attention message passing: For node i and its neighbor nodes j∈N(i), DGAT calculates a dynamic attention coefficient α. ij (t). Its innovation lies in considering not only node features (x transformed by the weight matrix W) when calculating correlation, but also... i ,x j It also incorporates a time decay function. The output of , where t ij This is the time of the last interaction. This gives more weight to recent interactions in attention allocation. Node features are then updated by weighting and aggregating neighbor information based on the attention coefficient: .

[0050] Global context vector generation: After multiple layers of DGAT propagation, high-level features of all nodes are obtained. The system uses a graph-level readout operation, such as multi-head attention pooling, to aggregate features from all nodes and ultimately output a compact and comprehensive global context state vector C(t) to guide subsequent fusion and decision-making.

[0051] S203. Dynamic Evolution of Context Graph and Triggered Update Mechanism The context graph is constantly evolving, and its updates are triggered by multiple mechanisms to ensure that the model is always in sync with the real world.

[0052] Periodic lightweight updates: Update the dynamic characteristics of all nodes at fixed time intervals (e.g., per second), such as refreshing the readings of environmental sensor nodes.

[0053] Event-driven forced update: Interaction event triggering: When a user interacts with the device, the characteristics of the relevant user nodes and device nodes are immediately updated, and the weights of the associated edges are recalculated.

[0054] Environmental mutation trigger: When sensor data fluctuations exceed a preset threshold δ, a forced recalculation and update of the environmental nodes and their associated edges is triggered.

[0055] Scene switching trigger: When the system infers a change in the dominant scene (such as switching from "leisure" to "work"), it will activate a complete graph reasoning process, updating or creating the corresponding abstract scene nodes.

[0056] Updated image G t+1 The context representation is immediately used for learning in the next time step, forming a closed loop.

[0057] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0058] S301. Based on the current context state vector and the feature vectors of each modality, a set of cross-modal attention weights is calculated; wherein, the attention weight of each modality reflects the importance of the modality's features to the overall semantic contribution in the current context. Please refer to... Figure 3 .

[0059] The first vector, formed by concatenating the feature vector of the target modality with the context state vector, is then mapped to the query vector through the first linear transformation matrix. :

[0060] in, Let be the first linear transformation matrix. Let be the first vector. For the feature vector of the target mode, This is the context state vector; Combine the feature vectors of all modalities with the context state vector, and map them to key vectors respectively through the second linear transformation matrix:

[0061] in, Let j be the key vector corresponding to the j-th mode. , This is the second linear transformation matrix. Let j be the feature vector of the j-th mode; and These are trainable parameters. Methods for concatenating feature vectors with context states include: Generate a pair of modulation parameters for the context vector C using a lightweight neural network:

[0062]

[0063] in, Scaling factor For bias terms, , Same dimension as the feature vector, and These are the trainable parameters of the neural network; The feature vector is adjusted using scaling factors and bias terms to selectively enhance or suppress the corresponding dimensions of the feature vector:

[0064] The adjusted feature vector The concatenated vector is obtained by juxtaposing and concatenating the original context vector C with the concatenated vector: .

[0065] Calculate query vector With each key vector The dot product yields the preliminary attention score:

[0066] if Features and a certain mode The features are highly relevant in the current context (i.e.) and If the directions are similar, the dot product value is larger. It will be very high.

[0067] The attention scores are normalized using the Softmax function to obtain the cross-modal attention weights for the target modality:

[0068] By switching the target modality, the cross-modal attention weights for all modalities are finally obtained.

[0069] S302. Using the calculated cross-modal attention weights, the corresponding modal feature vectors are weighted and summed to generate the fused feature vector.

[0070] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0071] S401. Definition of Reinforcement Learning Model Elements First, a deep reinforcement learning agent is instantiated, whose core elements are precisely defined based on the current task scenario: State Space (S): The state s observed by the agent. t It is a composite vector, composed of fused feature vectors F fusion The system comprises the global context state vector C(t), the recent interaction history, and the global context state vector C(t). This ensures that the agent's decisions are based on both the current multimodal semantic understanding and the dynamic environment and past experience.

[0072] Action Space (A): Defined as a series of multimodal interactive responses that an agent can execute. This is a composite action space, for example: Action = (Response Type, Target Device, Specific Parameters)

[0073] Reward Function (R): Design a multidimensional composite reward signal R. t This function guides the learning process. It comprehensively evaluates the immediate effects of each interaction. Task accuracy bonus (R1): Determined by subsequent user feedback or preset task completion verification (such as whether the device responds correctly to the instructions).

[0074] Response efficiency penalty (R2): Introduces a negative reward based on response latency to encourage rapid decision-making.

[0075] User experience reward (R3): can be estimated through implicit feedback (such as the user's motivation for subsequent interactions) or explicit feedback (such as satisfaction ratings).

[0076] The total reward is calculated as follows: , where ∑ω=1, and the weights can be adjusted according to the scenario.

[0077] S402. Strategy Generation and Decision Execution Policy Networks and Decisions: A deep neural network (such as an Actor network) is used as the policy function. At each decision time t, the policy network receives the state s. t As input, the system outputs the probability distribution or deterministic action value of all possible actions through forward propagation. Based on this strategy (exploratory noise may be introduced during the early stages of training), the system selects an optimal or near-optimal action 'a'. t .

[0078] Multimodal response execution: Selected action a t It is passed to the interactive execution module. This module is a multi-channel output executor. If the action includes a text / voice response, then the Natural Language Generation (NLG) and Text-to-Speech (TTS) services are invoked to generate an audible or visible response.

[0079] If the action involves device control, standardized control commands are sent to the target device via IoT protocols such as MQTT and CoAP.

[0080] If the action includes visual prompts, the corresponding interface will be rendered on the specified display terminal.

[0081] S403. Experience Replay and Continuous Strategy Optimization Experience storage: The system stores the experience tuples (s) of each interaction step. t ,a t ,R t ,s t+1 Stored in an experience playback buffer.

[0082] Model training and updates: Periodically sample a batch of historical experience from the buffer and use the loss function of reinforcement learning algorithms (such as Proximal Policy Optimization (PPO) and Deep Deterministic Policy Gradient (DDPG)) to update the policy network. And an optional value network is used for training and updates. The training objective is to minimize the loss function, which is equivalent to maximizing the expected long-term cumulative reward.

[0083] Online Adaptation: After deployment, the system can enable online learning mode, using continuous interaction data generated in real-world scenarios to fine-tune the strategy, enabling the agent to adapt to specific user habits or environmental changes, achieving personalized interaction that becomes smarter with use.

[0084] In one example: In a smart home scenario, state s tThe algorithm incorporates the semantic feature of "the user says 'it's a bit hot'" and the context vector of "the user is in the living room, the air conditioner is currently off, and the room temperature is 28°C". The policy network calculates a high-probability action that might output: (Device control, living room air conditioner, mode: cooling, temperature: 25°C) (Device control, living room air conditioner, mode: cooling, temperature: 25°C). The execution module then sends a control command to the living room air conditioner via Wi-Fi and simultaneously generates a voice reply, "The air conditioner temperature has been lowered for you." The reward for this interaction is R. t The system will then calculate based on whether the air conditioner was successfully turned on, the response speed, and whether the user subsequently gave positive feedback (such as saying "thank you"), in order to optimize future decision-making strategies.

[0085] In some embodiments, the context-aware interactive system may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the context-aware interactive system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Features based on context-aware interaction.

[0086] In this embodiment, the context-aware interaction system can be divided into multiple functional modules based on the functions it performs, such as... Figure 4 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0087] The feature acquisition module is used to acquire multimodal data and preprocess and extract features from the multimodal data to obtain feature vectors for each modality. The context learning module is used to dynamically model and update historical interaction features, environmental state features, and user profile features based on graph neural networks, and generate context state vectors. The modality fusion module is used to perform cross-modal feature fusion on the feature vectors of each modality based on an attention mechanism to generate a fused feature vector, wherein the weight calculation of the attention mechanism depends on the context state vector; The strategy generation module is used to generate an interaction strategy based on the fused feature vector and the context state vector using a reinforcement learning algorithm, and to execute the corresponding multimodal interaction response.

[0088] Figure 5The context-aware interaction method provided in the embodiments of this application can be applied to devices. Those skilled in the art will understand that the device structures involved in the embodiments of this invention do not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0089] The device 500 may include a processor 510, a memory 520, and a communication unit 530. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0090] The memory 520 can be used to store execution instructions of the processor 510. The memory 520 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 520 are executed by the processor 510, the device 500 is able to perform some or all of the steps in the above method embodiments.

[0091] The processor 510 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 520, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 510 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0092] The communication unit 530 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0093] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0094] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0095] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0096] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0097] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0099] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A context-aware interaction method, characterized in that, include: Collect multimodal data, and preprocess and extract features from the multimodal data to obtain feature vectors for each modality; Based on graph neural networks, historical interaction features, environmental state features, and user profile features are dynamically modeled and updated to generate context state vectors. Based on the attention mechanism, cross-modal feature fusion is performed on the feature vectors of each modality to generate a fused feature vector, wherein the weight calculation of the attention mechanism depends on the context state vector; Based on the fused feature vector and the context state vector, an interaction strategy is generated through a reinforcement learning algorithm, and the corresponding multimodal interaction response is executed.

2. The method according to claim 1, characterized in that, When collecting multimodal data, the method further includes: A high-precision hardware clock source is set as the global master clock. All sensors receive the global acquisition trigger signal from the master device through a unified hardware trigger line to achieve hardware-level synchronization of acquisition start-up. When each sensor collects each frame of data, in addition to generating its own local timestamp, it also synchronously records the precise moment when it receives the synchronization pulse of the global master clock. Based on the local timestamp sequence recorded by each sensor and the corresponding synchronization pulse time sequence, the drift rate and initial offset of the local clock of each sensor relative to the global master clock are periodically fitted. For each frame of data acquired by each sensor, the local timestamp is compensated using the estimated drift rate and initial offset of that sensor, and an accurate timestamp aligned with the global master clock is calculated.

3. The method according to claim 2, characterized in that, The multimodal data is preprocessed and feature extracted to obtain feature vectors for each modality, including: Based on the current context state vector and real-time monitored system resource status indicators, a dynamic configuration strategy is generated for the preprocessing of each modality; wherein, the context state vector includes at least the interaction task type and environment state characteristics; the system resource status indicators include at least the processor computing load and available memory; and the dynamic configuration strategy includes at least the processing algorithm selected for each modality, the set processing parameters, and the assigned processing priority. According to the dynamic configuration strategy, each modality-specific preprocessing unit is controlled to perform corresponding adaptive preprocessing operations, including: for sensor data, processing is performed using a noise reduction algorithm combining time-series prediction and a real-time anomaly detection algorithm selected based on the strategy; for text data, the corresponding model is dynamically selected and called between a lightweight feature extraction model and a high-precision deep semantic model for processing according to the strategy; for image or video data, feature extraction is performed after dynamically adjusting the resolution or sampling frame rate according to the strategy. The quality of each modal data after preprocessing is evaluated, and optimization processing or re-acquisition is triggered when the quality is lower than a preset threshold.

4. The method according to claim 1, characterized in that, Based on graph neural networks, historical interaction features, environmental state features, and user profile features are dynamically modeled and updated to generate a context state vector, including: Construct a dynamic multimodal context graph, wherein: the nodes of the graph include at least user nodes, device nodes, and environment parameter nodes, and each node is associated with a corresponding multimodal feature; the edges of the graph represent the entity relationships between nodes and have weights dynamically calculated based on interaction frequency and time; The dynamic multimodal context graph is input into a graph neural network for context representation learning; the graph neural network aggregates the neighbor information of each node to update the node features by introducing an attention mechanism with a time decay function, and generates a global context state vector by aggregating the updated full graph node features. Based on periodic time-series signals and event triggering signals driven by interactive events, environmental changes, or scene switching events, the features of corresponding nodes and the weights of associated edges in the dynamic multimodal context graph are dynamically updated; the updated graph is used for context representation learning in the next time step.

5. The method according to claim 1, characterized in that, Based on an attention mechanism, cross-modal feature fusion is performed on the feature vectors of the aforementioned modalities to generate a fused feature vector, including: Based on the current context state vector and the feature vectors of each modality, a set of cross-modal attention weights is calculated; wherein, the attention weight of each modality reflects the importance of the modality feature to the overall semantic contribution in the current context; The calculated cross-modal attention weights are used to perform a weighted summation of the corresponding modal feature vectors to generate the fused feature vector.

6. The method according to claim 5, characterized in that, Based on the current context state vector and the feature vectors of each modality, a set of cross-modal attention weights is calculated, including: The first vector, formed by concatenating the feature vector of the target modality with the context state vector, is then mapped to the query vector through the first linear transformation matrix. : in, Let be the first linear transformation matrix. Let be the first vector. For the feature vector of the target mode, This is the context state vector; Combine the feature vectors of all modalities with the context state vector, and map them to key vectors respectively through the second linear transformation matrix: in, Let j be the key vector corresponding to the j-th mode. , This is the second linear transformation matrix. Let j be the feature vector of the j-th mode; Calculate query vector With each key vector The dot product is used to obtain a preliminary attention score; The attention scores are normalized using the Softmax function to obtain the cross-modal attention weights for the target modality.

7. The method according to claim 6, characterized in that, Methods for concatenating feature vectors and context state vectors include: Generate a pair of modulation parameters for the context vector C using a lightweight neural network: in, Scaling factor For bias terms, , Same dimension as the feature vector, and These are the trainable parameters of the neural network; The feature vector is adjusted using scaling factors and bias terms to selectively enhance or suppress the corresponding dimensions of the feature vector: The adjusted feature vector The concatenated vector is obtained by juxtaposing and concatenating the original context vector C with the concatenated vector: .

8. A context-aware interactive system, characterized in that, include: The feature acquisition module is used to acquire multimodal data and preprocess and extract features from the multimodal data to obtain feature vectors for each modality. The context learning module is used to dynamically model and update historical interaction features, environmental state features, and user profile features based on graph neural networks, and generate context state vectors. The modality fusion module is used to perform cross-modal feature fusion on the feature vectors of each modality based on an attention mechanism to generate a fused feature vector, wherein the weight calculation of the attention mechanism depends on the context state vector; The strategy generation module is used to generate an interaction strategy based on the fused feature vector and the context state vector using a reinforcement learning algorithm, and to execute the corresponding multimodal interaction response.

9. A context-aware interactive device, characterized in that, include: Memory, used to store context-aware interactive programs; A processor, configured to implement the steps of the context-aware interaction method as described in any one of claims 1-7 when executing the context-aware interaction program.

10. A computer-readable medium storing a computer program, characterized in that, The readable medium stores a context-aware interactive program, which, when executed by a processor, implements the steps of the context-aware interactive method as described in any one of claims 1-7.

Images