Accompanying robot decision-making method, device and equipment based on cross-modal knowledge graph and medium

Abstract

The invention relates to an accompanying robot decision-making method and device based on a cross-modal knowledge graph, equipment and a medium. The method comprises the following steps: acquiring original multi-modal sensor data, and performing time alignment and perception feature extraction on the original multi-modal sensor data to obtain a perception unit set; fusing the perception unit set to obtain a time sequence semantic vector and a semantic candidate entity; updating the cross-modal knowledge graph, and performing conditional aggregation and read extraction on the updated cross-modal knowledge graph by adopting a graph neural network to obtain an emotional state vector with probability distribution, an environment semantic abstract and a trigger factor set; and in combination with the long-term user portrait and the preset task constraint, graph conditional decision is carried out to generate an accompanying robot execution instruction. By the adoption of the method, real-time decision making and long-term self-adaption of an accompanying scene can be achieved through multi-modal time sequence fusion and a time hyperedge-based cross-modal knowledge graph in combination with conditional graph reasoning and a personalized strategy.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and in particular relates to a decision-making method, device, equipment and medium for companion robots based on cross-modal knowledge graphs. Background Technology

[0002] With the development of robot control technology, multimodal perception and fusion analysis technologies have emerged to assist robot decision-making and control. By simultaneously utilizing data from multiple modalities such as voice, video, images, and sensors, more comprehensive environmental and user information can be obtained than from a single modality. Multimodal emotion recognition, environmental understanding, and behavior prediction methods have already been applied in scenarios such as intelligent interaction, smart healthcare, and emotion computing.

[0003] In traditional techniques, multimodal methods are typically handled through two approaches: feature-level fusion, which involves directly concatenating or weighting the embedded vectors of data such as speech, images, and sensor signals before inputting them into a classifier for prediction; and decision-level fusion, which involves independently identifying different modalities and then obtaining the final result through voting, weighting, or confidence rules during the decision-making stage.

[0004] However, the aforementioned methods focus on simple combinations of numerical features or decision results, lacking explicit representations of semantic entities, relationships, and temporal contexts between different modalities, making it difficult to support reasoning in complex scenarios. Feature-level or decision-level fusion often ignores the structural interactions between modalities, such as the association between images and speech describing the same event, leading to poor interpretability of the results. Insufficient characterization of the temporal dependencies and potential triggering factors in multimodal data makes it impossible to effectively extract key factors leading to changes in user state. When faced with differences in user profiles and changes in task environments, traditional methods often fail to dynamically adjust reasoning strategies, resulting in poor adaptability of robot decisions. Summary of the Invention

[0005] Therefore, it is necessary to provide a decision-making method, device, equipment, and medium for companion robots based on cross-modal knowledge graphs that can accurately model and extract user emotional states, environmental semantics, and triggering factors, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a decision-making method for companion robots based on cross-modal knowledge graphs, including:

[0007] The raw multimodal sensor data is acquired, and time alignment and sensor feature extraction are performed on the raw multimodal sensor data to obtain a set of time-stamped sensing units. The raw multimodal sensor data includes camera video frames, microphone audio streams and environmental sensor data distributed by timestamps. The set of sensing units includes the modal representation and confidence level of each modality.

[0008] An order-invariant ensemble multimodal fusion processor is used to fuse the set of perception units to obtain temporal semantic vectors and semantic candidate entities. The temporal semantic vectors include main semantic vectors, semantic assertions, and semantic confidence. The order-invariant ensemble multimodal fusion processor allocates fusion weights according to modality confidence.

[0009] The cross-modal knowledge graph is updated based on temporal semantic vectors and semantic candidate entities. A graph neural network is then used to perform conditional aggregation and readout extraction on the updated cross-modal knowledge graph to obtain a mood state vector with probability distribution, an environmental semantic summary, and a set of triggering factors. The cross-modal knowledge graph includes perceptual nodes, semantic entity nodes, user profile nodes, and a high-order cross-modal event subgraph connected by temporal hyperedges.

[0010] Based on the emotional state vector, environmental semantic summary, and trigger factor set, combined with long-term user profiles and preset task constraints, graph-conditional decision-making is used to generate instructions for the companion robot to execute; the instructions for the companion robot to execute include natural language text and voice parameters.

[0011] In one embodiment, the raw multimodal sensor data is time-aligned and sensor feature extracted to obtain a time-stamped set of sensing units, including:

[0012] Based on a sliding time window with a preset time step, the original multimodal sensor data is time-aligned according to timestamps to obtain a set of multimodal observations within the same time window;

[0013] Within the time window, each modal observation in the multimodal observation set is preprocessed to obtain the standard modal observation corresponding to each mode;

[0014] The perception quality is calculated for each mode of signal source, and the perception quality index corresponding to each mode is obtained. The perception quality index includes signal-to-noise ratio, frame drop rate, occlusion detection indication and delay estimation.

[0015] Based on standard modal observations, feature extraction is performed on each modality to obtain modal representations. The confidence of each modal representation is then calculated using perception quality indicators to obtain perception units. A perception unit includes modality type, modal representation, timestamp, and confidence.

[0016] All sensing units within the time window are grouped together to form a time-stamped sensing unit set.

[0017] In one embodiment, an order-invariant ensemble multimodal fusion processor is used to fuse the set of perception units to obtain a temporal semantic vector and semantic candidate entities, including:

[0018] The modal representations in the set of sensing units are mapped to the same dimension to obtain the embedding vectors corresponding to each sensing unit, and the initial weights of each embedding vector are determined according to the confidence level.

[0019] The embedding vectors are weighted according to the initial weights, and the interaction relationship between the weighted and modulated embedding vectors is captured based on the self-attention mechanism to obtain the intermediate interaction representation;

[0020] The intermediate interaction representation is mapped to a fixed-dimensional semantic vector by a preset aggregation function to obtain the main semantic vector; the aggregation weights corresponding to the aggregation function are determined by the intermediate interaction representation and the initial weights.

[0021] Based on the intermediate interaction representation and the main semantic vector, semantic category discrimination and attribute extraction are performed to obtain multiple semantic assertions and their corresponding probabilities.

[0022] Based on the semantic knowledge base, semantic retrieval is performed according to the main semantic vector and semantic assertions, and semantic candidate entities are determined according to the degree of matching between the semantics and the intermediate interaction representation.

[0023] The semantic confidence is obtained by aggregating the semantic assertions and calculating the confidence of each semantic candidate entity based on the probability.

[0024] In one embodiment, the cross-modal knowledge graph is constructed using the following method:

[0025] Based on the temporal semantic vectors and semantic candidate entities of historical raw multimodal sensor data, corresponding node candidates are identified and generated;

[0026] A temporal hyperedge is constructed based on the co-occurrence relationship of semantic candidate entities in the same time window; the temporal hyperedge connects the perceptual nodes and semantic entity nodes related to each modality and records the temporal attributes and semantic confidence.

[0027] A cross-modal knowledge graph is constructed based on node candidates and temporal hyperedges. The cross-modal knowledge graph consists of a two-layer structure: a short-term working graph and a long-term memory graph. The short-term working graph is used to store high-frequency event subgraphs within the most recent time window. The long-term memory graph is used to store summarized user historical preferences and long-term patterns.

[0028] In one embodiment, updating the cross-modal knowledge graph based on temporal semantic vectors and semantic candidate entities includes:

[0029] Candidate insertion nodes are obtained based on temporal semantic vectors and semantic candidate entities;

[0030] The embedding space similarity metric is obtained by comparing and learning the insertion node candidate with the node candidate based on node embedding similarity and contextual time information.

[0031] Based on a preset merging threshold, duplicate nodes with the same semantics are identified and merged according to the embedding space similarity metric, and the node confidence is merged according to the evidence fusion rule to obtain the initial updated cross-modal knowledge graph.

[0032] When the confidence of merged nodes in the initial updated cross-modal knowledge graph is lower than a preset threshold, based on the neighborhood structure of the initial updated cross-modal knowledge graph, the semantic information and confidence of neighboring nodes are propagated to the target nodes whose confidence is lower than the preset threshold, and the confidence of node candidates and time super-edges are normalized to obtain the updated cross-modal knowledge graph.

[0033] In one embodiment, a graph neural network is used to conditionally aggregate and extract readouts from the updated cross-modal knowledge graph, resulting in an emotion state vector with probability distribution, an environmental semantic summary, and a set of triggering factors, including:

[0034] A conditional graph neural network is used to aggregate message passing on the updated cross-modal knowledge graph, resulting in multiple high-order cross-modal event subgraphs connected by temporal hyperedges; conditionalization includes using short-term working graph segments and / or long-term memory graph context as conditional inputs to adjust message passing weights;

[0035] In the high-order cross-modal event subgraph, perceptual nodes and semantic entity nodes corresponding to user states are selected, and the perceptual nodes and semantic entity nodes are aggregated by emotion probability distribution to obtain an emotion state vector.

[0036] In the high-order cross-modal event subgraph, the perceptual nodes and semantic entity nodes corresponding to the environmental state are selected, and the perceptual nodes and semantic entity nodes are read out in a summary manner to obtain the environmental semantic summary.

[0037] Based on the message passing path, the causal chain of the emotional state vector is traced in the higher-order cross-modal event subgraph, and the set of triggering factors is obtained according to the node contribution corresponding to the semantic confidence.

[0038] In one embodiment, based on the emotional state vector, environmental semantic summary, and trigger factor set, combined with long-term user profiles and preset task constraints, graph-conditional decision-making is performed to generate instructions for the companion robot to execute, including:

[0039] A learning strategy based on graph conditionation is used to generate candidate action distributions based on emotion state vectors, environmental semantic summaries, trigger factor sets, and long-term user profiles. The candidate action distributions include the actions to be performed and their corresponding priorities.

[0040] The candidate action distribution is tested and modified for safety and ethics according to preset task constraints to obtain the execution action distribution;

[0041] Based on the functional action library of the companion robot, the distribution of the execution actions is mapped to the execution instructions of the companion robot; the execution instructions of the companion robot include natural language text, voice parameters and robotic arm action parameters; the voice parameters are obtained by mapping the set of trigger factors to the corresponding fine-grained prosodic parameter spectrum for emotion enhancement.

[0042] Secondly, this application also provides a decision-making device for a companion robot based on a cross-modal knowledge graph, comprising:

[0043] The data module is used to acquire raw multimodal sensor data, and to perform time alignment and sensor feature extraction on the raw multimodal sensor data to obtain a set of sensing units with time stamps;

[0044] The modality fusion module is used to fuse the set of perception units using an order-invariant aggregate multimodal fusion processor to obtain temporal semantic vectors and semantic candidate entities;

[0045] The knowledge graph module is used to update the cross-modal knowledge graph based on temporal semantic vectors and semantic candidate entities, and to use graph neural networks to perform conditional aggregation and readout extraction on the updated cross-modal knowledge graph to obtain a mood state vector with probability distribution, environmental semantic summary and trigger factor set;

[0046] The execution conversion module is used to generate execution instructions for the companion robot by performing graph-conditional decision-making based on the emotional state vector, environmental semantic summary, and trigger factor set, combined with long-term user profiles and preset task constraints.

[0047] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above-described companion robot decision-making methods based on cross-modal knowledge graphs.

[0048] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described companion robot decision-making methods based on cross-modal knowledge graphs.

[0049] The aforementioned decision-making methods, devices, equipment, and media for companion robots based on cross-modal knowledge graphs significantly reduce single-modal ambiguity through continuous multimodal monitoring and temporal semantic extraction, and the complementary fusion of visual, auditory, environmental, and physiological signals, thereby reducing false alarms and improving decision confidence. It can identify and generate executable instructions when companionship needs occur, integrating long-term user profiles and conditional strategies to make robot behavior consider user preferences, health constraints, and cultural differences, providing more personalized reassuring voices, inquiry strategies, and physical assistance, thus improving user satisfaction and compliance. Through confidence-driven backtracking, rule and constraint checks, and other multi-layered protection, it can reduce misoperation and ethical risks when facing high-risk decisions. The evidence chain, temporal hyperedges, and trigger factor sets of the knowledge graph provide a traceable source and reason for each decision, enhancing trust and accountability traceability. The long-term graph structure supports the abstraction of high-frequency events into long-term patterns, providing a solid historical basis for strategy optimization, personalized recommendations, and risk prediction. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the decision-making method for companion robots based on cross-modal knowledge graphs according to the present invention.

[0052] Figure 2 A step-by-step flowchart for updating the cross-modal knowledge graph in step S103;

[0053] Figure 3 This is a flowchart illustrating the steps of step S104.

[0054] Figure 4 This is a structural diagram of the companion robot decision-making device based on cross-modal knowledge graph of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] In one embodiment, such as Figure 1As shown, a decision-making method for companion robots based on cross-modal knowledge graphs is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0057] S101. Acquire raw multimodal sensor data, and perform time alignment and sensor feature extraction on the raw multimodal sensor data to obtain a set of time-stamped sensing units; the raw multimodal sensor data includes camera video frames, microphone audio streams and environmental sensor data distributed by timestamps; the set of sensing units includes the modal representation and confidence level of each modality.

[0058] Indicatively, raw multimodal sensor data is acquired from various sensing channels onboard or externally connected to the companion robot. This includes video frame streams from cameras, audio streams from microphones, signals from tactile or pressure sensors, environmental sensors monitoring temperature, light intensity, and noise levels, and physiological sensors monitoring heart rate and skin conductance. Optionally, emotion recognition results and their confidence distributions from independent emotion recognition modules or third-party sensing modules can be received as alternative inputs to interface with existing technologies without needing to rebuild the sensing link.

[0059] After acquiring the raw data, asynchronously arriving multimodal observations need to be aligned by time and divided into time windows. Specifically, a sliding time window mechanism based on a preset time step is used to buffer data from different sources according to their timestamps and map them to the same time window. Further, each modal observation within the same time window undergoes modality-specific preprocessing. Preprocessing for the visual modality includes frame denoising, face / human detection, keypoint / pose tracking, and local feature extraction; preprocessing for the auditory modality includes speech activity detection, framing and spectral transformation, and basic acoustic feature extraction; and preprocessing for the tactile and environmental modalities includes drift reduction, filtering, and event detection. After preprocessing, perceptual quality indices are calculated for each modality within the time window to quantify the reliability of that modality within the current time window. These indices include signal-to-noise ratio, frame drop rate, occlusion indication, and latency estimation.

[0060] Based on the preprocessing results and perceptual quality metrics, feature extraction and confidence scores are calculated for each modality to form a perceptual unit. Optionally, each perceptual unit is represented as a quadruple of modality type, modality representation, timestamp, and confidence score. All perceptual units within a time window are aggregated to obtain a time-stamped perceptual unit set. This set preserves the evidence chain of the original multimodal information and carries a measure of modality reliability, providing a unified and traceable data structure for subsequent aggregate fusion and knowledge graph construction. For example, modality representation can be achieved using lightweight convolutional / temporal encoders or pre-trained embedding network outputs, and confidence scores can be obtained directly from linear / nonlinear mappings of the perceptual quality metrics or estimated using a small confidence regressor.

[0061] S102. The set of perception units is fused using an order-invariant ensemble multimodal fusion processor to obtain temporal semantic vectors and semantic candidate entities. The temporal semantic vectors include main semantic vectors, semantic assertions, and semantic confidence. The order-invariant ensemble multimodal fusion processor allocates fusion weights according to modal confidence.

[0062] This paper illustrates an order-invariant ensemble multimodal fusion processor for fusing a set of perceptual units. Order invariance means the processor is insensitive to the order of the input set, facilitating stable behavior even when modalities are missing or dynamically added to the perceptual channel. The order-invariant ensemble multimodal fusion processor leverages the complementarity between modalities to enhance semantic signals, suppresses the negative impact of noisy modalities through confidence information, and explicitly generates intermediate interaction representations to support subsequent semantic assertions and knowledge graph insertion. For example, each modal representation in the perceptual unit set is mapped to a unified-dimensional embedding space through its respective modality encoder to obtain an embedding vector set. The confidence of each perceptual unit is normalized to obtain initial weights. Injecting the confidence as prior information into the attention calculation of the fusion processor allows for modulation of attention weights by multiplying or weighting the confidence in the attention scoring function, thereby giving high-confidence embeddings greater influence in interaction aggregation. Furthermore, a multi-head ensemble self-attention mechanism is used at the ensemble level to perform interactive computation on the weighted embeddings, obtaining the intermediate interactive representation of each perceptual unit to encode the information of the modality itself and the contextual influence from other modalities, while retaining the source modality index and time anchor point for easy subsequent tracing.

[0063] To map variable-length intermediate interaction representations to fixed-dimensional temporal semantic vectors, a set readout module is used after set self-attention. This can be implemented as a learnable aggregation function, for example, a weighted sum followed by mapping via a multilayer perceptron, or a pooling / attention readout mechanism based on Set Transformer. In this mechanism, the aggregation weights are jointly determined by the attention scores and initial confidence in the intermediate interaction representations, thereby passing the interaction strength and modal reliability to the formation process of the main semantic vector. The main semantic vector represents the overall semantic overview of the current time window, including semantic category clues, temporal features, and uncertainty estimates.

[0064] Optionally, after obtaining the main semantic vector, a semantic assertion generator based on a discriminant head is used to perform semantic category discrimination and attribute extraction on the intermediate interaction representation and the main semantic vector, generating several semantic assertions. Further, using the main semantic vector and semantic assertions as retrieval criteria, relevant entity candidates are retrieved from a semantic knowledge base or a pre-defined semantic template. Semantic candidate entities are selected by calculating the matching degree between the semantics and the intermediate interaction representation. The initial confidence of each semantic candidate entity is calculated by combining the discriminant probability with its support source confidence. The combination method can be a learnable fusion machine, a probabilistic Bayesian combination, or evidence theory, etc.

[0065] Furthermore, to enhance robustness in modality missing scenarios, the ensemble fusion processor includes a representation compensation module. This module is trained during the training phase using modality masks, randomly masking some modalities to learn cross-modal reconstruction tasks. It can employ a conditional variational autoencoder or a conditional regressor, enabling the prediction of the missing modality's embedding estimate based on the intermediate interaction representations of the remaining modalities and the main semantic vector in real-world missing scenarios. This estimate can then be used as virtual supporting evidence in confidence calculations or re-fusion.

[0066] S103. Update the cross-modal knowledge graph based on the temporal semantic vector and semantic candidate entities, and use a graph neural network to perform conditional aggregation and readout extraction on the updated cross-modal knowledge graph to obtain a mood state vector with probability distribution, environmental semantic summary and trigger factor set; the cross-modal knowledge graph includes perception nodes, semantic entity nodes, user profile nodes and high-order cross-modal event subgraphs connected by temporal hyperedges.

[0067] Cross-Modal Knowledge Graphs (CMKGs) are used to organize temporal semantic vectors, semantic candidate entities, and historical information in a structured manner, thereby enabling long-term memory, causal cue extraction, and higher-order reasoning. A CMKG includes perceptual nodes representing specific perceptual units or their abstractions, semantic entity nodes such as emotion categories, behavioral events, and environmental factors, as well as higher-order cross-modal event subgraphs including user profile nodes connected by temporal hyperedges. Temporal hyperedges are co-evidence from multiple modalities within the same time window; hyperedges can simultaneously connect multiple nodes and carry temporal and confidence attributes, thus providing richer structural information for causal reasoning and modality compensation.

[0068] Indicatively, the construction of the CMKG includes offline construction and online incremental updates. The online update process takes the temporal semantic vectors and semantic candidate entities output by the ensemble fusion engine as input. Specifically, it identifies and generates graph node candidates from the semantic information, then constructs temporal hyperedges based on the co-occurrence relationships of semantic candidate entities within the same time window, and uses an incremental writing strategy to insert new nodes, fused edges, or hyperedges into the existing graph. Optionally, to avoid full reconstruction, during insertion, the node embedding similarity obtained from contrastive learning and contextual temporal information are used for entity alignment and merging determination. Duplicate nodes exceeding the similarity threshold are merged, and their confidence scores are merged and normalized according to evidence fusion rules such as Bayesian updates.

[0069] Furthermore, a conditional graph neural network (GNN) is employed to perform message passing and aggregation computation on a cross-modal knowledge graph to extract hidden representations relevant to decision-making. Conditionalization involves injecting long-term user profiles, current task constraints, or contextual information as conditional inputs into the message passing process. This can be achieved by concatenating conditional vectors to node features, modulating node messages through a FiLM (feature-wise linear modulation) layer, or adjusting neighbor aggregation weights using learnable attention parameters. The specific implementation of message passing can involve setting independent message and aggregation functions for different node and edge types, and employing a special high-order aggregator for temporal hyperedges to fully integrate multimodal evidence within the time window. After several layers of conditional message passing, each node in the graph, as well as the event subgraph centered on the temporal hyperedge, will receive a conditional embedded representation. This representation integrates local structural information and carries biases from user profiles and task constraints, thus providing semantically and causally adapted representations for subsequent readout extraction.

[0070] The readout extraction phase aggregates and projects conditional embeddings at the subgraph level, resulting in an emotion state vector, an environmental semantic summary, and a set of triggering factors. The emotion state vector is obtained by applying classification or probability mapping to subgraph embeddings related to user emotions, outputting a probability distribution and an estimate of emotion intensity across a predefined set of emotion categories. The environmental semantic summary is obtained by aggregating environment-related nodes, such as noise level, illumination, and temperature, at the subgraph level, providing a compressed description of the current environmental state. The set of triggering factors identifies the nodes or hyperedges with the greatest impact on emotion judgment by analyzing attention weights in message passing, changes in node embeddings, and causal contributions, and outputs them in order of contribution.

[0071] To illustrate, a multi-stage training strategy is employed to train the graph neural network. The offline stage utilizes labeled historical data for supervised training, with objectives including node classification, edge type prediction, and subgraph-level sentiment prediction. Self-supervised tasks, such as node embedding comparison and neighborhood reconstruction, are also incorporated to improve generalization ability in sparse data scenarios. Optionally, the robustness of the model can be enhanced through training that simulates modality loss and adversarial noise.

[0072] S104. Based on the emotional state vector, environmental semantic summary, and trigger factor set, combined with long-term user profile and preset task constraints, perform graph conditional decision-making to generate instructions for the companion robot to execute; the instructions for the companion robot to execute include natural language text and voice parameters.

[0073] This illustration demonstrates how a graph-conditional decision-making framework transforms structured semantic information into specific execution instructions, while simultaneously considering personalization and security constraints. Long-term user profiles include basic user information, long-term preferences, past intervention response records, and potential medical / ethical constraints. These profiles can be generated based on statistical aggregation of long-term memory graphs in historical CMKG (Consumer Data Library), or on preference tags and care records entered by the user on the device. Preset task constraints are determined by scenario requirements and security policies, including but not limited to privacy rules, non-executable medical interventions, and threshold conditions for triggering alarms.

[0074] Specifically, emotional state vectors, environmental semantic summaries, and triggering factors are used as conditional inputs, and provided to the graph-conditionalized learning strategy module along with long-term user profiles and task constraints. This module can be implemented as a graph encoder plus a policy network. The graph encoder rapidly encodes the current short-term CMKG subgraph, and the policy network outputs a candidate action distribution on the conditionalized representation of the encoding. The action set covers multimodal behaviors such as voice soothing, guided breathing, prompting movement, adjusting the environment, alarming, or maintaining observation. Furthermore, to ensure safety and legal compliance, the candidate distribution is then fed into a rule constraint layer. The rule layer screens and corrects candidate actions using predefined hard constraints and priority strategies. When the highest confidence of a candidate action is lower than a preset threshold, alternative strategies can be triggered, such as taking a more neutral response, requesting user confirmation, or notifying caregivers.

[0075] Once the action is determined, the abstract action is mapped to executable instructions. When the action includes a natural language response, speech parameters need to be further generated to control the emotional speech synthesis. Optionally, trigger factors identified in the CMKG are mapped to fine-grained speech parameter spectra, such as mapping mechanisms for fundamental frequency contours, speech rate, pause patterns, and stress distribution. Specifically, trigger factors are matched with corresponding speaking strategies in historical corpora, and natural language text is generated through retrieval enhancement or template fine-tuning. Then, corresponding tone parameters are set based on the emotional preferences of the trigger factors and the output of the policy network. Finally, an emotion-controlled neural TTS (Text to Speech) engine is invoked to synthesize speech. For the non-speech parts of the action, the decision is mapped to the robot's motion and interaction commands, such as robotic arm movements, haptic feedback instructions, or smart home control commands, along with execution confidence and backoff strategies.

[0076] In the aforementioned decision-making method for companion robots based on cross-modal knowledge graphs, precise alignment of sliding time windows and modal confidence weighting enable the mapping of asynchronous and multi-frequency visual / audio / environmental / wearable signals to a unified semantic temporal sequence. This reduces false positives and false negatives caused by temporal mismatches, thereby improving the recall and precision performance of event detection. The order-invariant aggregate design and confidence modulation make the fusion unit insensitive to input order, tolerant of modal loss or short-term occlusion, and reduce the impact of single sensor failures on the overall system, thus improving availability and stable operating time. A high-order cross-modal event subgraph represented by temporal hyperedges, combined with short-term / long-term dual-layer graph storage, can capture composite events in real time and summarize user habits and risk patterns in long-term memory, supporting long-term personalization, trend prediction, and progressive adaptive strategy optimization. Conditional message passing and graph-conditional decision-making in graph neural networks can jointly consider short-term evidence and long-term profiles, achieving context-sensitive and user-preferred behaviors while preserving a traceable chain of evidence to generate interpretable decision explanations. The seamless mapping from semantics to natural language and prosodic parameters enables companion robots to output more natural and emotionally resonant voices and movements, improving user acceptance and the success rate of interventions.

[0077] In one embodiment, the raw multimodal sensor data is time-aligned and sensor feature extracted to obtain a time-stamped set of sensing units, including:

[0078] S11. Based on a sliding time window with a preset time step, the original multimodal sensor data is time-aligned according to the timestamp to obtain a multimodal observation set within the same time window.

[0079] This illustration demonstrates how a sliding time window mechanism based on a preset time step is used to segment and align timestamp data. Specifically, one or a set of configurable time steps are determined, and overlapping or non-overlapping sliding windows are constructed accordingly. Shorter step sizes and higher overlap ratios improve responsiveness to sudden events, while longer step sizes reduce computational burden and are more suitable for long-term pattern analysis. Fine-grained synchronization typically relies on hardware clock synchronization, such as cameras / microphones sharing hardware timestamps, and uses interpolation or resampling to upscale low-frequency data to higher-frequency axes when needed. Wide-window synchronization correlates events of different modalities using sliding time windows, with the window length adaptively set based on the scene and sensor latency. Optionally, in different deployment scenarios, an adaptive window length algorithm can be used, depending on resource and real-time requirements. In terms of time alignment, when the device supports hardware timestamps or network time synchronization, precise timestamps should be used first for fine alignment. In the absence of a unified clock, the time delay between modes can be estimated and corrected by delay estimation and mutual information / cross-correlation methods, or an event-driven alignment strategy can be adopted, that is, different mode signals triggered by the same event are grouped into the same time window, thereby forming a multimodal observation set for that time window.

[0080] S12. Within the time window, preprocess each modal observation in the multimodal observation set to obtain the standard modal observation corresponding to each mode.

[0081] For example, visual input requires denoising, color normalization, lens distortion correction and stabilization, and may include region-of-interest cropping and multi-scale resampling as needed; if there is motion blur or low-light conditions, preprocessing also includes image enhancement or learning-based deblurring methods. Audio stream preprocessing should include speech activity detection, noise suppression, echo cancellation, and beamforming of array microphones to enhance the target speaker's signal. Environmental sensors are filtered, have missing value interpolation and outlier removal performed, and are calibrated as necessary to eliminate sensor drift.

[0082] S13. Calculate the sensing quality for each mode of signal source to obtain the sensing quality index corresponding to each mode; the sensing quality index includes signal-to-noise ratio, frame drop rate, occlusion detection indication and delay estimation.

[0083] For example, the signal-to-noise ratio (SNR) can be calculated by estimating the power of the signal and noise; in visual scenes, the local feature response energy or keypoint detection confidence can be used as an approximate SNR; the frame drop rate is calculated and recorded as the missing frame metric by the ratio of the expected number of frames to the actual number of frames received; occlusion detection can be achieved through the stability of object detection, keypoint loss, and pixel occlusion rates of depth map anomalies or abrupt changes, and temporal consistency checks can also be used, such as a sharp decrease in the intersection-union ratio (IUU) of the predicted bounding box and the ground truth bounding box over time to determine temporary occlusion or long-term missing frames; latency estimation is based on the timestamp and the time difference of arrival, and, if necessary, combined with network layers and buffer time for comprehensive estimation. When mapping these raw metrics to standardized perceptual quality scores, normalization / standardization functions can be used, and adjustable weights can be defined for different modalities to synthesize a single quality score or retain multi-dimensional quality vectors for use by the fusion processor.

[0084] S14. Based on standard modal observation, feature extraction is performed on each modality to obtain modal representations, and the confidence of each modal representation is calculated in combination with the perception quality index to obtain a perception unit; the perception unit includes modality type, modal representation, timestamp and confidence.

[0085] For example, for visual modalities, pre-trained convolutional / transformer networks can be used to extract frame-level or region-level vectors, and object detection, pose estimation, facial expression analysis, and illumination / scene classifiers can be run in parallel to output structured semantics; for audio, short-time spectral features (MFCC) can be extracted and semantic embeddings can be obtained using a self-supervised model, while an emotion recognizer is run to obtain keywords and emotion distribution; for wearable or environmental sensors, statistics, spectral features, and event indicators can be extracted.

[0086] The confidence calculation of modal representation combines the perceived quality with the internal uncertainty of the model to obtain an overall confidence assessment of the representation. Specifically, the internal output of the model is used as the prior, and the external perceived quality is used as the calibration factor. The calibration can be multiplicative scaling or a probabilistic fusion method. For example, the modal representation is regarded as a Gaussian distribution with variance, and the sensor quality is mapped to the observation noise variance through Kalman filtering or Bayesian update to obtain the posterior confidence.

[0087] S15. Collect all the sensing units within the time window to form a set of sensing units with time stamps.

[0088] Each sensing unit not only includes modal representation and timestamp, but also needs to include a modal confidence score or uncertainty measure, characterizing the unit's credibility with downstream semantics. The confidence score can be obtained by combining the probability distribution output by the sensing feature extraction model, the calibrated confidence score, Bayesian estimation based on the model's confidence interval, or a combination of the sensor's own signal-to-noise ratio, transmission packet loss rate, and sensor health status.

[0089] In one embodiment, an order-invariant ensemble multimodal fusion processor is used to fuse the set of perception units to obtain a temporal semantic vector and semantic candidate entities, including:

[0090] S21. Map each modal representation in the set of sensing units to the same dimension to obtain the embedding vector corresponding to each sensing unit, and determine the initial weight of each embedding vector according to the confidence level.

[0091] Illustratively, a lightweight modality projector is designed or learned for each modality, mapping modal features to embedding vectors of the same dimension. This mapping can be either a linear transformation with a normalization layer or a small nonlinear encoder. Further, the confidence score of each perceptual unit is used as the initial weight for the embedding. Specifically, the initial weights can be directly applied as multipliers to the embedding vector, or the confidence score can be mapped to the attention bias term through a small normalization layer. Optionally, to avoid excessive confidence dominance that could suppress valuable low-confidence information, the confidence score is scaled using a temperature parameter before being combined with the weights learned by the model.

[0092] S22. The embedded vectors are weighted according to the initial weights, and the interaction relationship between the weighted and modulated embedded vectors is captured based on the self-attention mechanism to obtain the intermediate interaction representation.

[0093] To capture the interactions between elements in a set and model complex dependencies while preserving order invariance, a self-attention-based interaction module is employed. This involves first performing shared encoding on the weighted embeddings, then using a symmetric attention / pooling combination, such as using the multi-head attention module in Set Transformer with pooling readout, or performing pairwise interactions followed by element-level reduction. Alternatively, an equivalent strategy is to first multiply the initial weights into the embeddings, then use standard multi-head self-attention and perform unordered pooling on all elements during the readout phase, ultimately preserving set invariance in the final readout.

[0094] S23. Using a preset aggregation function, the intermediate interaction representation is mapped to a semantic vector of fixed dimension according to the timestamp to obtain the main semantic vector; the aggregation weight corresponding to the aggregation function is determined by the intermediate interaction representation and the initial weight.

[0095] The aggregation employs pooling with adaptive weights, specifically implemented as a learnable pooler that takes intermediate interaction representations and initial weights as input and outputs a fixed-dimensional vector. The pooling weights themselves can be computed by a small attention network that takes the intermediate representations and initial weights as input, predicts the aggregation weight for each element, and then performs a weighted sum or weighted average of the elements. If necessary, a projection layer and normalization are then applied to obtain the final main semantic vector.

[0096] S24. Based on the intermediate interaction representation and the main semantic vector, perform semantic category discrimination and attribute extraction to obtain multiple semantic assertions and their corresponding probabilities.

[0097] Based on intermediate interaction representations and the main semantic vector, semantic category discrimination and attribute extraction are performed. This involves identifying several semantic assertions that may be true within a given time window, such as "the user is eating" or "the room temperature is too high," and outputting the corresponding probability for each assertion. Illustratively, a hybrid approach combining a multi-label classifier and a regression branch is used. The multi-label classifier performs parallel binary classification on a set of predefined semantic labels, with confidence scores obtained through an activation function. Attribute extraction uses regression or classification outputs and provides confidence intervals. To accommodate open-domain semantics, sequence-to-sequence or retrieval-enhanced generative models can also be used to generate natural language semantic assertions from the main semantic vector, followed by confidence calibration of the generated results.

[0098] S25. Based on the semantic knowledge base, perform semantic retrieval according to the main semantic vector and semantic assertion, and determine the semantic candidate entities according to the degree of matching between the semantics and the intermediate interaction representation.

[0099] Abstract semantic assertions are mapped to potential entity candidates within a knowledge structure. Specifically, the main semantic vector and assertion vector are used as queries, and matching entity candidates are identified through vector similarity retrieval. A matching score is then calculated by combining intermediate interaction representations. The matching score relies not only on embedding similarity but also on a weighted fusion of cross-modal consistency features indicated in the interaction representations and rule-based semantic constraints, such as temporal and spatial consistency. The retrieval process typically returns the top K candidates for subsequent confidence calculations and multiple hypothesis evaluations.

[0100] S26. Aggregate the semantic assertions and calculate the confidence of each semantic candidate entity based on the probability to obtain the semantic confidence.

[0101] Indicatively, a Bayesian or weighted fusion strategy is used, where assertion probability is regarded as the likelihood of assertion, retrieval similarity is regarded as evidence of consistency with the knowledge base, perceptual confidence is used as the observation noise term, and finally the posterior probability is calculated as the entity confidence.

[0102] In one embodiment, the cross-modal knowledge graph is constructed using the following method:

[0103] S31. Identify and generate corresponding node candidates based on the temporal semantic vector and semantic candidate entities of the historical original multimodal sensor data.

[0104] This example illustrates how historical raw multimodal data, after front-end processing and aggregate fusion, is used to map temporal semantic vectors and semantic candidate entities to graph node candidates. Specifically, candidates are first normalized, deduplicated, and scored to avoid graph scaling and semantic redundancy. For instance, text normalization is employed, such as word segmentation, synonym normalization, named entity recognition, and disambiguation. Further, entity type annotations are performed on actions, objects, physiological signals, locations, and user features, and temporal semantic vectors are clustered to discover duplicate or semantically equivalent candidates.

[0105] S32. Construct a temporal hyperedge based on the co-occurrence relationship of semantic candidate entities in the same time window; the temporal hyperedge connects the perceptual nodes and semantic entity nodes related to each modality and records the temporal attributes and semantic confidence.

[0106] Temporal hyperedges bind multiple semantically related nodes that co-occur within the same time window through a higher-order connection structure. The hyperedge encodes both the temporal information of the simultaneous occurrence / association of these facts and carries confidence vectors and contextual attributes from different modalities. For example, a hyperedge can be explicitly represented as an entity in a graph, where the node is connected to participating entity nodes via ordinary edges and stores timestamps, durations, source modality sets, overall confidence, and other metadata; or a hypergraph or higher-order graph data structure can be used to directly support the hyperedge type. Specifically, within a configured time window, the co-occurrence matrix of each candidate entity is calculated. Statistical significance tests such as chi-square tests or mutual information, along with frequency thresholds, are used to filter out combinations with significant co-occurrence. Then, a temporal hyperedge is created for each significant combination, and the confidence of the hyperedge is set as a function of the confidence of the participating entities and the reliability of co-occurrence.

[0107] S33. Construct a cross-modal knowledge graph based on node candidates and time superedges. The cross-modal knowledge graph includes a two-layer structure: a short-term working graph and a long-term memory graph. The short-term working graph is used to store high-frequency event subgraphs within the most recent time window. The long-term memory graph is used to store summarized user historical preferences and long-term patterns.

[0108] Furthermore, to accommodate both agile responses to short-term events and long-term pattern induction, a short-term working graph and a long-term memory graph are employed. The short-term working graph primarily stores high-frequency event subgraphs, temporary entities, and event hyperedges within recent time windows, characterized by high write frequency, low persistence requirements, and rapid mutability. The long-term memory graph, on the other hand, stores long-term entities and relationships that have been statistically inductively derived, patterned, or manually confirmed, such as users' long-term preferences, habitual time patterns, and long-term risk markers.

[0109] Intuitively, the short-term working graph receives node candidates and superedge updates generated for each time window in a streaming manner, and performs real-time statistics on the occurrence frequency, average confidence, and temporal distribution of frequently occurring subgraphs. Periodic runs migrate short-term patterns that meet certain conditions to the long-term memory graph. Criteria for memory consolidation may include repetition exceeding a threshold, passing a statistical significance test, successful alignment with user profiles or external knowledge bases, or manual review approval. During migration, knowledge compression and merging are performed, such as entity alignment for equivalent entities, merging edge weights and updating temporal distribution information, and, if necessary, using abstraction rules to compress high-frequency, low-information-content events into patterned entries.

[0110] In one embodiment, such as Figure 2 As shown, updating the cross-modal knowledge graph based on temporal semantic vectors and semantic candidate entities includes:

[0111] S201. Obtain candidate insertion nodes based on temporal semantic vectors and semantic candidate entities.

[0112] After obtaining new temporal semantic vectors and semantic candidate entities, the primary task in updating the cross-modal knowledge graph is to transform these inputs into node candidates to be inserted, and to determine their similarity, complementarity, or contradiction with existing nodes in the graph, in order to decide whether to insert, merge, or label them as suspicious facts. Specifically, the temporal semantic vectors and semantic candidate entities are normalized and enriched to generate insertion node candidates. Normalization includes standardized naming, type labeling, and recording the original evidence index and time window attributes in a unified format; enrichment includes mapping the main semantic vector to vector embedding, retrieving external vocabularies or ontology from semantic candidate entities to supplement category / attribute information, and calculating the initial confidence based on modal evidence.

[0113] S202. Based on node embedding similarity and contextual time information, the embedding space similarity measure is obtained by comparing and learning the insertion node candidate with the node candidate.

[0114] Furthermore, determining whether a candidate insertion node already has semantically repetitive or similar nodes in the graph can be achieved through an embedding space similarity metric obtained by comparative learning based on node embedding similarity and contextual temporal information. Specifically, an embedding model pre-trained or updated online on a historical multimodal semantic alignment task is used to map all nodes and insertion candidates in the graph to the same vector space. The embedding model can be trained through comparative learning or triplet loss to ensure that semantically similar entities are spatially close, while unrelated entities are separated. Cosine similarity or inner product is commonly used for similarity measurement: when two embeddings are spatially close and their temporal attributes are within a tolerable time window, the similarity evidence is considered stronger; conversely, if the time offset is too large, it needs to be discounted or treated as weak similarity.

[0115] S203. Based on the preset merging threshold, duplicate nodes with the same semantics are judged and merged according to the embedding space similarity metric, and the node confidence is merged according to the evidence fusion rule to obtain the initial updated cross-modal knowledge graph.

[0116] Based on similarity metrics, duplicate nodes are identified and merged according to a preset merging strategy, thereby generating an initially updated knowledge graph. The merging decision is based on a configurable merging threshold or hierarchical threshold strategy: when similarity is above a strict threshold and the temporal context is consistent, nodes are merged directly; when similarity is between a lenient and strict threshold, nodes are marked as potentially duplicated and multiple hypotheses are retained, with subsequent decisions made based on accumulated evidence or manual review; when similarity is below the threshold, new nodes are added. Optionally, the merging operation involves not only merging entity identifiers but also performing evidence fusion on node attributes, edge weights, and confidence levels. Evidence fusion rules can employ Bayesian fusion, weighted average, or a learned fusion engine. For example, an interpretable fusion priority strategy is adopted, with conservative rules applied to key attributes, such as user physiological abnormality markers and medical contraindications (i.e., any high-confidence conflict requires manual verification), and probabilistic fusion applied to routine attributes.

[0117] S204. When the confidence of the merged nodes in the initial updated cross-modal knowledge graph is lower than a preset threshold, based on the neighborhood structure of the initial updated cross-modal knowledge graph, the semantic information and confidence of the neighboring nodes are propagated to the target nodes whose confidence is lower than the preset threshold, and the confidence of the node candidates and the time super-edge are normalized to obtain the updated cross-modal knowledge graph.

[0118] After initial similarity-based merging, a graph-based neighborhood confidence propagation and enhancement process is performed to form an updated cross-modal knowledge graph. Illustratively, neighborhood information of nodes, including the confidence of neighboring nodes, relation types, and temporal hyperedge participation, is used as additional evidence. A graph propagation algorithm transmits the semantic information and confidence of neighbors to the target node. Optionally, weighted average-based local propagation, label propagation, or Bayesian network propagation can be employed to perform confidence updates within a probabilistic graphical model framework. Alternatively, a graph neural network can be used for conditional aggregation, inputting the initial representation of the target node and neighbor representations into the network to output calibrated confidence and attribute distributions. Furthermore, the propagation process should consider time decay (i.e., the influence of older neighbor evidence decreases), modality source reliability (i.e., trusted sources such as medical devices have higher weights), and error amplification prevention mechanisms (i.e., propagation should be suppressed when the overall confidence of neighbors is low or contradictory). After propagation, the confidence of node candidates and temporal hyperedges is normalized to ensure that the confidence in the graph is represented on a consistent and interpretable scale.

[0119] In one embodiment, a graph neural network is used to conditionally aggregate and extract readouts from the updated cross-modal knowledge graph, resulting in an emotion state vector with probability distribution, an environmental semantic summary, and a set of triggering factors, including:

[0120] S41. On the updated cross-modal knowledge graph, a conditional graph neural network is used for message passing aggregation to obtain multiple high-order cross-modal event subgraphs connected by temporal hyperedges; conditionalization includes using short-term working graph segments and / or long-term memory graph context as conditional inputs to adjust message passing weights.

[0121] Conditional graph neural networks (GNNs) both propagate and converge information along the graph topology and adaptively adjust message passing and reading behavior based on external conditions, such as short-term working graph segments, long-term memory graph context, current task constraints, or user profiles. At the message passing level, conditional vectors obtained from the aggregation embedding of the short-term working graph or the user profile embedding of long-term memory can be concatenated with node / edge features and input into the message function, making each message explicitly dependent on conditional information. Alternatively, F iLM or conditional attention bias can be used, where scaling and offset parameters are generated from the conditional vectors and injected into the normalization and attention calculations of each layer, adjusting the information flow without modifying the basic message passing function. After several message passing iterations, the GNN generates conditional node / hyperedge representations and identifies multiple high-order cross-modal event subgraphs connected by temporal hyperedges based on these representations. Higher-order event subgraphs can be formed by selecting a group of nodes that share a common temporal hyperedge or have a high co-occurrence score. Event clustering algorithms can be run on the graph first to aggregate related nodes, and then each set can be used as a subgraph of interest in the GNN for fine-grained readout. Alternatively, local readouts can be performed on all temporal hyperedges in parallel on the entire graph to obtain a series of parallel event subgraph representations.

[0122] S42. In the higher-order cross-modal event subgraph, filter the perceptual nodes and semantic entity nodes corresponding to the user state, and aggregate the perceptual nodes and semantic entity nodes by emotion probability distribution to obtain the emotion state vector.

[0123] Schematic, in each higher-order event subgraph, nodes most relevant to the user's state are selected, i.e., nodes labeled with types such as human body / physiology / expression / voice emotion, or nodes marked as semantically matching the user's state by a learned relevance scorer. This selection can be achieved by combining node type, semantic label, temporal distance, and normalized weights of conditional attention. Furthermore, probabilistic aggregation is performed on the representations of perceptual nodes and semantic entity nodes; the output emotion state vector should contain both the probability for each emotion category and an uncertainty estimate.

[0124] S43. In the high-order cross-modal event subgraph, filter the perception nodes and semantic entity nodes corresponding to the environmental state, and perform summary reading on the perception nodes and semantic entity nodes to obtain the environmental semantic summary.

[0125] Nodes related to environmental conditions are identified through a filtering mechanism, such as temperature / light / gas sensor nodes, room scene recognition nodes, and object and location nodes. The conditional representations of these nodes are then read out in a summary format. Specifically, this can be achieved using structured summaries, such as key-value pairs or attribute tables containing information like current temperature, noise level, door and window status, and present personnel, or using natural language summaries. Structured summaries can be obtained directly from node vector mapping using an MLP; natural language summaries can be generated from a graph-to-text model, where the high-dimensional representation obtained from the readout is used as encoder output, fed into a controlled text generator, and filtered by confidence or template backoff after generation to ensure safety and accuracy. To ensure the compactness and usability of the summaries, the readout module can prioritize retaining attributes with high confidence and high relevance to the current task, and compress or truncate redundant information according to priority.

[0126] S44. Based on the message passing path, trace the causal chain of the emotional state vector in the high-order cross-modal event subgraph, and obtain the set of triggering factors according to the node contribution corresponding to the semantic confidence.

[0127] The extraction of the trigger factor set aims to list the key evidence items that lead to the current emotional or environmental state in an interpretable form, which is to conduct a traceable analysis of the message transmission path and node contribution. For example, a direct contribution measure based on attention / message weights can be used, that is, in the readout layer of a conditional GNN, the attention weights of nodes / edges are themselves a natural contribution score; a path score-based method can also be used, that is, calculating the message product or weighted sum on the readout path from the evidence node to the emotional or environmental vector target, the path score measures which relationships the node influences the final output; a counterfactual method based on causality or adversarial testing can also be used, that is, locally deleting or replacing candidate evidence, observing the change in the output probability distribution, and quantifying its causal contribution accordingly.

[0128] In one embodiment, such as Figure 3 As shown, based on the emotional state vector, environmental semantic summary, and trigger factor set, combined with long-term user profiles and preset task constraints, graph-conditional decision-making is used to generate instructions for the companion robot to execute, including:

[0129] S301. Using a graph-conditional learning strategy, a candidate action distribution is generated based on the emotion state vector, environmental semantic summary, trigger factor set, and long-term user profile. The candidate action distribution includes the action to be performed and its corresponding priority.

[0130] This example illustrates a graph-conditional learning strategy, where the readout of emotions, environment, triggering factors, user profiles, and constraint encodings from a knowledge graph serve as conditional inputs, driving a policy network or policy generator to output a distribution of candidate actions. Exemplarily, this strategy can be designed as one or a hybrid of various architectures, such as a combination of graph-conditional graph encoders, conditional policy networks, and value functions, trained within a reinforcement learning framework; or it can be based on imitation learning / behavior cloning using a conditional generative model, learning the distribution from large-scale human-robot interaction examples. Specifically, the input vector should include an emotion state vector, a structured environment summary, a list of triggering factors, user profile embeddings, task constraint encodings, and current resource / state information. The policy module outputs an action probability distribution or a set of scored candidates, where each candidate includes a high-level action label, estimated priority, estimated risk / reward value, execution prerequisites, and expected side effects.

[0131] When training the policy, offline expert demonstrations are first used to clone behavior and quickly converge the initial behavior of the policy. Then, constrained reinforcement learning is used to further optimize it in simulated environments and hierarchical real-world scenarios to balance utility and safety. The training loss should include task rewards, such as user comfort, accident avoidance, and task completion rate, as well as penalties, such as behaviors that violate hard constraints or cause complaints, and incorporate confidence calibration terms and interpretability regularization. For high-risk events with scarce samples, such as real-world fall scenarios, simulations and physics / statistical synthetic data are prioritized, and careful online fine-tuning is performed in small-scale real-world scenarios.

[0132] S302. The candidate action distribution is tested and modified for safety and ethics according to the preset task constraints to obtain the action distribution.

[0133] Candidate actions are subjected to safety and ethical checks and corrections based on preset task constraints to obtain an executable action distribution. Task constraints include both hard constraints (prohibited behaviors, such as unauthorized remote recording, prohibition of contact intervention, and prohibition of disease diagnosis) and soft constraints (preferences, comfort thresholds, family rules, etc.). The checks and corrections module consists of three layers: a rule engine layer, a constraint solving / verification layer, and a strategy correction layer. The rule engine first blocks obviously unexecutable or illegal candidates using priority rules; for example, any action involving the forced movement of a user must not pass without clear authorization or evidence of a life-threatening situation. The constraint solving layer performs formal verification, using constraint solving / satisfaction checks or planner simulations to determine the feasibility of actions within the current resources and environment. For example, it checks whether the robotic arm's movements will collide with the current environment, whether the voice broadcast will be within the prohibited decibel range, and assesses the candidate's time window, concurrency conflicts, and resource consumption. The strategy correction layer then performs soft corrections to the candidate distribution while maintaining maximum utility, such as lowering the priority of high-risk actions or replacing them with conservative strategies. Quadratic optimization or probabilistic renormalization can be used to produce the final distribution of executed actions.

[0134] S303. Based on the functional action library of the companion robot, the distribution of the execution actions is mapped to the execution instructions of the companion robot; the execution instructions of the companion robot include natural language text, voice parameters and robotic arm action parameters; the voice parameters are obtained by mapping the set of trigger factors to the corresponding fine-grained prosodic parameter spectrum for emotion enhancement.

[0135] The action library is organized hierarchically: high-level semantic actions, such as asking if there is pain; mid-level behavioral templates, such as walking next to the user and maintaining a distance of 0.5 meters while raising a hand to indicate pain; and low-level physical instructions / parameters, such as navigation target position, robotic arm end effector pose trajectory, gripping torque limits, and TTS text and prosodic parameters. The mapping layer is responsible for converting high-level actions into a set of executable low-level instructions, along with final safety checks and pre-execution simulations. The mapping process typically employs hierarchical planning: high-level actions are converted into a sequence of action subtasks by the task interpreter, the mid-level planner generates motion / interaction parameters for each subtask, and the low-level controller takes over precise execution.

[0136] Speech parameters include at least speech rate, fundamental frequency / pitch shift, volume, voice / gender selection, intonation profile, pause distribution, and sentiment labels. Specifically, the overall sentiment tone is first determined by the emotional state vector, and then the prosody is refined by trigger factors. For example, if the trigger factor indicates pain and has a high contribution, the speech rate is slightly slowed down, the fundamental frequency is lowered, and short pauses are added to express empathy. If the trigger factor is loneliness and the long-term profile indicates that the user prefers a relaxed tone, a warm voice is selected and empathetic statements are appropriately added. The generated natural language text is jointly generated by templates and a controlled generator. That is, the template first covers regular / high-risk speech, and then the controlled generation model enriches the details. All generated text must undergo constraint checks and be accompanied by confidence scores.

[0137] The mapping between the robotic arm and motion parameters must meet physical and safety constraints. For each action, the mapping layer should output the robotic arm's task-level parameters, such as target pose, trajectory type, maximum permissible speed / acceleration, force / torque limits, and safety distance constraints, and call the motion planning module, i.e., inverse kinematics, collision detection, and dynamic constraint solving, to generate the execution trajectory. If the required action is a light support action, the action parameters should include the upper limit of contact force, tactile / force feedback loop, and abnormal shutdown strategy.

[0138] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0139] Based on the same inventive concept, this application also provides a cross-modal knowledge graph-based companion robot decision-making device for implementing the aforementioned cross-modal knowledge graph-based companion robot decision-making method. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the cross-modal knowledge graph-based companion robot decision-making device provided below can be found in the limitations of the cross-modal knowledge graph-based companion robot decision-making method described above, and will not be repeated here.

[0140] In one exemplary embodiment, such as Figure 4 As shown, a decision-making device for a companion robot based on a cross-modal knowledge graph is provided, comprising:

[0141] Data module 401 is used to acquire raw multimodal sensor data, and to perform time alignment and sensor feature extraction on the raw multimodal sensor data to obtain a set of sensing units with time stamps;

[0142] The modal fusion module 402 is used to fuse the set of perception units using an order-invariant set-based multimodal fusion processor to obtain temporal semantic vectors and semantic candidate entities;

[0143] The knowledge graph module 403 is used to update the cross-modal knowledge graph based on the temporal semantic vector and semantic candidate entities, and to use a graph neural network to perform conditional aggregation and readout extraction on the updated cross-modal knowledge graph to obtain a mood state vector with probability distribution, an environmental semantic summary and a set of triggering factors.

[0144] The execution conversion module 404 is used to generate execution instructions for the companion robot by performing graph-conditional decision-making based on the emotional state vector, environmental semantic summary and trigger factor set, combined with long-term user profile and preset task constraints.

[0145] In one embodiment, the data module 401 is further configured to:

[0146] Based on a sliding time window with a preset time step, the original multimodal sensor data is time-aligned according to timestamps to obtain a set of multimodal observations within the same time window;

[0147] Within the time window, each modal observation in the multimodal observation set is preprocessed to obtain the standard modal observation corresponding to each mode;

[0148] The perception quality is calculated for each mode of signal source, and the perception quality index corresponding to each mode is obtained. The perception quality index includes signal-to-noise ratio, frame drop rate, occlusion detection indication and delay estimation.

[0149] Based on standard modal observations, feature extraction is performed on each modality to obtain modal representations. The confidence of each modal representation is then calculated using perception quality indicators to obtain perception units. A perception unit includes modality type, modal representation, timestamp, and confidence.

[0150] All sensing units within the time window are grouped together to form a time-stamped sensing unit set.

[0151] In one embodiment, the modal fusion module 402 is further configured to:

[0152] The modal representations in the set of sensing units are mapped to the same dimension to obtain the embedding vectors corresponding to each sensing unit, and the initial weights of each embedding vector are determined according to the confidence level.

[0153] The embedding vectors are weighted according to the initial weights, and the interaction relationship between the weighted and modulated embedding vectors is captured based on the self-attention mechanism to obtain the intermediate interaction representation;

[0154] The intermediate interaction representation is mapped to a fixed-dimensional semantic vector by a preset aggregation function to obtain the main semantic vector; the aggregation weights corresponding to the aggregation function are determined by the intermediate interaction representation and the initial weights.

[0155] Based on the intermediate interaction representation and the main semantic vector, semantic category discrimination and attribute extraction are performed to obtain multiple semantic assertions and their corresponding probabilities.

[0156] Based on a semantic knowledge base, semantic retrieval is performed according to the main semantic vector and semantic assertions, and semantic candidate entities are determined according to the degree of matching between the semantics and the intermediate interaction representation.

[0157] The semantic confidence is obtained by aggregating the semantic assertions and calculating the confidence of each semantic candidate entity based on the probability.

[0158] In one embodiment, a knowledge graph construction module is also included, for:

[0159] Based on the temporal semantic vectors and semantic candidate entities of historical raw multimodal sensor data, corresponding node candidates are identified and generated;

[0160] A temporal hyperedge is constructed based on the co-occurrence relationship of semantic candidate entities in the same time window; the temporal hyperedge connects the perceptual nodes and semantic entity nodes related to each modality and records the temporal attributes and semantic confidence.

[0161] A cross-modal knowledge graph is constructed based on node candidates and temporal hyperedges. The cross-modal knowledge graph consists of a two-layer structure: a short-term working graph and a long-term memory graph. The short-term working graph is used to store high-frequency event subgraphs within the most recent time window. The long-term memory graph is used to store summarized user historical preferences and long-term patterns.

[0162] In one embodiment, the knowledge graph module 403 is further configured to:

[0163] Candidate insertion nodes are obtained based on temporal semantic vectors and semantic candidate entities;

[0164] The embedding space similarity metric is obtained by comparing and learning the insertion node candidate with the node candidate based on node embedding similarity and contextual time information.

[0165] Based on a preset merging threshold, duplicate nodes with the same semantics are identified and merged according to the embedding space similarity metric, and the node confidence is merged according to the evidence fusion rule to obtain the initial updated cross-modal knowledge graph.

[0166] When the confidence of merged nodes in the initial updated cross-modal knowledge graph is lower than a preset threshold, based on the neighborhood structure of the initial updated cross-modal knowledge graph, the semantic information and confidence of neighboring nodes are propagated to the target nodes whose confidence is lower than the preset threshold, and the confidence of node candidates and time super-edges are normalized to obtain the updated cross-modal knowledge graph.

[0167] In one embodiment, a knowledge graph reading module is also included, for:

[0168] A conditional graph neural network is used to aggregate message passing on the updated cross-modal knowledge graph, resulting in multiple high-order cross-modal event subgraphs connected by temporal hyperedges; conditionalization includes using short-term working graph segments and / or long-term memory graph context as conditional inputs to adjust message passing weights;

[0169] In the high-order cross-modal event subgraph, perceptual nodes and semantic entity nodes corresponding to user states are selected, and the perceptual nodes and semantic entity nodes are aggregated by emotion probability distribution to obtain an emotion state vector.

[0170] In the high-order cross-modal event subgraph, the perceptual nodes and semantic entity nodes corresponding to the environmental state are selected, and the perceptual nodes and semantic entity nodes are read out in a summary manner to obtain the environmental semantic summary.

[0171] Based on the message passing path, the causal chain of the emotional state vector is traced in the higher-order cross-modal event subgraph, and the set of triggering factors is obtained according to the node contribution corresponding to the semantic confidence.

[0172] In one embodiment, the conversion module 404 is further configured to:

[0173] A learning strategy based on graph conditionation is used to generate candidate action distributions based on emotion state vectors, environmental semantic summaries, trigger factor sets, and long-term user profiles. The candidate action distributions include the actions to be performed and their corresponding priorities.

[0174] The candidate action distribution is tested and modified for safety and ethics according to preset task constraints to obtain the execution action distribution;

[0175] Based on the functional action library of the companion robot, the distribution of the execution actions is mapped to the execution instructions of the companion robot; the execution instructions of the companion robot include natural language text, voice parameters and robotic arm action parameters; the voice parameters are obtained by mapping the set of trigger factors to the corresponding fine-grained prosodic parameter spectrum for emotion enhancement.

[0176] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0177] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0178] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0179] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A decision-making method for companion robots based on cross-modal knowledge graphs, characterized in that, The method includes: Raw multimodal sensor data is acquired, and time alignment and sensor feature extraction are performed on the raw multimodal sensor data to obtain a set of time-stamped sensing units; the raw multimodal sensor data includes camera video frames, microphone audio streams, and environmental sensor data distributed by timestamps; the set of sensing units includes the modal representation and confidence level of each modality; The set of perception units is fused using an order-invariant ensemble multimodal fusion processor to obtain a temporal semantic vector and semantic candidate entities; the temporal semantic vector includes a main semantic vector, semantic assertions, and semantic confidence; the order-invariant ensemble multimodal fusion processor allocates fusion weights according to modal confidence. The cross-modal knowledge graph is updated based on the temporal semantic vector and the semantic candidate entities. A graph neural network is then used to perform conditional aggregation and readout extraction on the updated cross-modal knowledge graph to obtain an emotion state vector with probability distribution, an environmental semantic summary, and a set of triggering factors. The cross-modal knowledge graph includes perception nodes, semantic entity nodes, user profile nodes, and a high-order cross-modal event subgraph connected by temporal hyperedges. Based on the emotional state vector, the environmental semantic summary, and the set of triggering factors, combined with long-term user profiles and preset task constraints, graph-conditional decision-making is used to generate instructions for the companion robot to execute; the instructions for the companion robot to execute include natural language text and voice parameters.

2. The method according to claim 1, characterized in that, The process of performing time alignment and sensing feature extraction on the original multimodal sensor data to obtain a time-stamped set of sensing units includes: Based on a sliding time window with a preset time step, the original multimodal sensor data is time-aligned according to the timestamp to obtain a multimodal observation set within the same time window; Within the time window, each modal observation in the multimodal observation set is preprocessed to obtain the standard modal observation corresponding to each mode; The perception quality is calculated for the signal source of each mode to obtain the perception quality index corresponding to each mode; the perception quality index includes signal-to-noise ratio, frame drop rate, occlusion detection indication and delay estimation. Based on the standard modal observations, feature extraction is performed on each modality to obtain modal representations, and the confidence level of each modal representation is calculated in conjunction with the perception quality index to obtain a perception unit; the perception unit includes modality type, modal representation, timestamp, and confidence level; All the sensing units within the time window are grouped together to form the time-stamped sensing unit set.

3. The method according to claim 2, characterized in that, The process employs an order-invariant ensemble multimodal fusion processor to fuse the set of perception units, obtaining temporal semantic vectors and semantic candidate entities, including: The modal representations in the set of sensing units are mapped to the same dimension to obtain the embedding vectors corresponding to each sensing unit, and the initial weights of each embedding vector are determined according to the confidence level. The embedding vectors are weighted according to the initial weights, and the interaction relationship between the weighted and modulated embedding vectors is captured based on the self-attention mechanism to obtain the intermediate interaction representation; The intermediate interaction representation is mapped to a fixed-dimensional semantic vector according to the timestamp using a preset aggregation function to obtain the main semantic vector; the aggregation weight corresponding to the aggregation function is determined by the intermediate interaction representation and the initial weight. Based on the intermediate interaction representation and the main semantic vector, semantic category discrimination and attribute extraction are performed to obtain multiple semantic assertions and their corresponding probabilities. Based on the semantic knowledge base, semantic retrieval is performed according to the main semantic vector and the semantic assertion, and the semantic candidate entity is determined according to the degree of matching between the semantics and the intermediate interaction representation. The semantic confidence is obtained by aggregating the semantic assertions and calculating the confidence of each semantic candidate entity based on the probability.

4. The method according to claim 1, characterized in that, The cross-modal knowledge graph is constructed using the following methods: Based on the temporal semantic vector and semantic candidate entities from the historical raw multimodal sensor data, corresponding node candidates are identified and generated; Construct a temporal hyperedge based on the co-occurrence relationship of the semantic candidate entities within the same time window; The temporal hyperedge connects the modality-related perception nodes and semantic entity nodes and records the temporal attributes and semantic confidence. The cross-modal knowledge graph is constructed based on the node candidates and the time hyperedges; the cross-modal knowledge graph includes a two-layer structure of short-term working graph and long-term memory graph; the short-term working graph is used to store high-frequency event subgraphs within the most recent time window; the long-term memory graph is used to store summarized user historical preferences and long-term patterns.

5. The method according to claim 4, characterized in that, The step of updating the cross-modal knowledge graph based on the temporal semantic vector and the semantic candidate entities includes: Candidate insertion nodes are obtained based on the temporal semantic vector and the semantic candidate entities; An embedding space similarity metric is obtained by comparing and learning the insertion node candidate with the node candidate based on node embedding similarity and contextual time information. Based on a preset merging threshold, duplicate nodes with the same semantics are determined and merged according to the embedding space similarity metric, and the node confidence is merged according to the evidence fusion rule to obtain an initial updated cross-modal knowledge graph. When the confidence of the merged nodes in the initial updated cross-modal knowledge graph is lower than a preset threshold, based on the neighborhood structure of the initial updated cross-modal knowledge graph, the semantic information and confidence of the neighboring nodes are propagated to the target nodes whose confidence is lower than the preset threshold, and the confidence of the node candidates and the time superedge are normalized to obtain the updated cross-modal knowledge graph.

6. The method according to claim 5, characterized in that, The updated cross-modal knowledge graph is conditionally aggregated and readout extracted using a graph neural network to obtain a probability-distributed emotion state vector, an environmental semantic summary, and a set of triggering factors, including: A conditional graph neural network is used to perform message passing aggregation on the updated cross-modal knowledge graph to obtain multiple high-order cross-modal event subgraphs connected by temporal hyperedges; the conditionalization includes using the short-term working image segment and / or the long-term memory graph context as conditional inputs to adjust the message passing weights; In the higher-order cross-modal event subgraph, the perceptual nodes and semantic entity nodes corresponding to the user state are selected, and the perceptual nodes and semantic entity nodes are aggregated by emotion probability distribution to obtain the emotion state vector; In the higher-order cross-modal event subgraph, the perception nodes and semantic entity nodes corresponding to the environmental state are selected, and the perception nodes and semantic entity nodes are read out in a summary manner to obtain the environmental semantic summary. Based on the message passing path, the causal chain of the emotional state vector is traced in the higher-order cross-modal event subgraph, and the set of triggering factors is obtained according to the node contribution corresponding to the semantic confidence.

7. The method according to claim 1, characterized in that, The step of generating execution instructions for the companion robot by performing graph-conditional decision-making based on the emotional state vector, the environmental semantic summary, and the set of triggering factors, combined with long-term user profiles and preset task constraints, includes: A learning strategy based on graph conditionalization is used to generate a candidate action distribution according to the emotional state vector, the environmental semantic summary, the set of triggering factors, and the long-term user profile; the candidate action distribution includes the action to be performed and its corresponding priority. The candidate action distribution is then tested and modified for safety and ethical reasons according to the preset task constraints to obtain the execution action distribution; Based on the functional action library of the companion robot, the distribution of the execution actions is mapped to the execution instructions of the companion robot; the execution instructions of the companion robot include natural language text, voice parameters and robotic arm action parameters; the voice parameters are obtained by mapping the set of trigger factors to the corresponding fine-grained prosodic parameter spectrum for emotion enhancement.

8. A decision-making device for a companion robot based on cross-modal knowledge graphs, characterized in that, The device includes: The data module is used to acquire raw multimodal sensor data and perform time alignment and sensor feature extraction on the raw multimodal sensor data to obtain a set of sensing units with time stamps. The modality fusion module is used to fuse the set of perception units using an order-invariant aggregate multimodal fusion processor to obtain temporal semantic vectors and semantic candidate entities; The knowledge graph module is used to update the cross-modal knowledge graph based on the temporal semantic vector and the semantic candidate entities, and to use a graph neural network to perform conditional aggregation and readout extraction on the updated cross-modal knowledge graph to obtain an emotion state vector with probability distribution, an environmental semantic summary and a set of triggering factors. The execution conversion module is used to generate execution instructions for the companion robot by performing graph conditional decision-making based on the emotional state vector, the environmental semantic summary, and the set of triggering factors, combined with the long-term user profile and preset task constraints.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images