An AI intelligent power supply service accepting system based on multi-modal interaction

Abstract

The application discloses an AI intelligent power supply service accepting system based on multi-modal interaction, comprising: a multi-modal interaction sensing module, an emotion and intention intelligent analysis module, an intelligent service decision and generation module and a multi-modal response execution and feedback module connected in turn. Through the integration of heterogeneous sensors and intelligent fusion processing based on the "body-off body" cognitive mapping theory, the system can synchronously analyze multi-modal signals such as voice, vision and touch, and map low-level physical behaviors into structured primary semantic fragments, breaking through the information limitation of single mode, capturing complex intentions such as "directional inquiry accompanied by puzzled expression", providing high-quality input data rich in context evidence for subsequent deep analysis, and improving the comprehensiveness and accuracy of intention perception.

Description

Technical Field

[0001] This application belongs to the field of intelligent human-computer interaction technology, specifically relating to an AI intelligent power supply service acceptance system based on multimodal interaction. Background Technology

[0002] With the rapid development of smart grids and digital services, power supply companies are transforming their business processing models from traditional manual counters and hotlines to online and intelligent systems. Currently, most mainstream intelligent customer service or business processing systems adopt text-based or single-voice modal interaction methods, and their technical implementation usually relies on pre-set dialogue processes and keyword matching. These systems are efficient in standardized business query scenarios, but they show significant shortcomings when facing complex, non-standardized user needs, especially in interaction scenarios accompanied by strong emotional factors (such as complaints, anxiety, and confusion).

[0003] First, existing systems have limitations at the interaction perception level. Most systems can only process structured text or single-path voice commands, failing to effectively integrate and understand the information-rich multimodal signals naturally generated by users during interaction, such as facial expressions, gestures, body language, and vocal rhythm. For example, when a user says "the electricity bill is fine," it may be accompanied by a confused expression and hesitant tone, conveying a potential doubt that is not genuine. Single text analysis will completely miss this crucial information, leading to a misjudgment of the user's true intention.

[0004] Secondly, the depth of understanding of user intent and emotion is insufficient. Traditional methods rely heavily on superficial semantic analysis, lacking the ability to deeply analyze colloquial expressions, implicit emotions (such as sarcasm and euphemism), and the contextual relationships of multi-turn dialogues. For professional terms and complex business process descriptions commonly found in power supply services, the system struggles to perform accurate semantic disambiguation and structured understanding, and is even less able to dynamically integrate user emotional states into the service decision-making logic.

[0005] Furthermore, the level of intelligence in business decision-making and response generation is limited. Current system decision-making logic is mostly based on static "IF-THEN" rule trees, resulting in rigid and templated response content that lacks dynamic reasoning and personalized adaptation capabilities based on real-time context (such as user profiles, power grid operating status, and interaction history). This leads to mechanical and unemotional response strategies, particularly ineffective when addressing emotional distress, explaining complex policies, or providing personalized advice, resulting in a poor user experience.

[0006] Therefore, the industry urgently needs a power supply service acceptance system that can truly "understand" users (including their emotions and deep intentions) and "intelligently" provide personalized and humanized services, in order to break through the bottlenecks of existing technologies in natural interaction, deep understanding, intelligent decision-making and robust service. Summary of the Invention

[0007] This application provides an AI-powered intelligent power supply service acceptance system based on multimodal interaction, aiming to solve the problems of limitations in the interaction perception level and insufficient depth in understanding user intent and emotion in existing technologies.

[0008] An AI-powered intelligent power supply service acceptance system based on multimodal interaction includes: a multimodal interaction perception module, an emotion and intent intelligent analysis module, an intelligent service decision and generation module, and a multimodal response execution and feedback module connected in sequence.

[0009] The multimodal interaction perception module collects the user's voice, visual and touch signals, and maps the original behavioral data into primary fused semantic fragments based on interaction syntax rules, and outputs them to the emotion and intention intelligent analysis module.

[0010] The emotional and intention intelligent analysis module receives the primary fused semantic fragment, parses it through a hierarchical multi-feature fused emotional analysis model, and outputs the user's emotional state, refined intention, and confidence level to the intelligent business decision and generation module.

[0011] The intelligent business decision-making and generation module retrieves and reasons in the power grid service business knowledge graph based on the emotional state, refined intent and multi-source context information, generates a structured personalized response strategy tuple, and outputs it to the multimodal response execution and feedback module.

[0012] The multimodal response execution and feedback module transforms the response strategy tuple into specific multimodal output content and outputs it to the user through a physical device, while recording interaction data to form a feedback learning loop.

[0013] Optionally, the multimodal interaction sensing module includes a physical input layer and an intelligent fusion processing layer;

[0014] The physical input layer integrates an audio acquisition unit, a visual acquisition unit, a touch acquisition unit, and an optional auxiliary sensing unit, used to capture native multimodal physical signals generated by user interaction;

[0015] The intelligent fusion processing layer is used to perform data preprocessing, feature extraction, spatiotemporal alignment and association on the physical signal, and to perform preliminary fusion based on predefined interaction syntax rules to generate the primary fusion semantic fragment.

[0016] Optionally, the emotion and intent intelligent analysis module includes a feature pre-training submodule, a hierarchical dynamic fusion submodule, and a contextual understanding submodule;

[0017] The feature pre-training submodule contains multiple pre-trained models, which are used to extract spoken text features, multimodal implicit expression features, and long contextual structured features, respectively.

[0018] The hierarchical dynamic fusion submodule fuses the features extracted by the pre-trained model with the main network through a dynamic routing control mechanism, and outputs an abstract feature sequence.

[0019] The context understanding submodule integrates the abstract feature sequence through a sequence model and outputs sentiment state, refined intent, and confidence level using a multi-task learning framework.

[0020] Optionally, the intelligent business decision-making and generation module includes a business knowledge graph, a dynamic decision engine, and a response strategy generator;

[0021] The business knowledge graph stores entities, attributes, and relationships within the power grid service domain;

[0022] The dynamic decision engine is used to receive the emotional state and refined intent, combine multi-source contextual information, perform retrieval, reasoning and candidate strategy ranking in the knowledge graph, and select the optimal response strategy.

[0023] The response strategy generator is used to transform the selected strategy into a structured response strategy tuple containing core objectives, content component sequences, channel allocation, and personalized fill fields.

[0024] Optionally, the dynamic decision engine adopts a hybrid reasoning architecture that combines rule-based reasoning, case-based reasoning, and graph pathfinding, and is equipped with a rule conflict resolution mechanism.

[0025] Optionally, the dynamic decision engine also includes a multi-factor scoring model for ranking candidate strategies, wherein the factor weights of the model are adjusted using a hybrid mechanism that combines static initialization with dynamic online optimization based on reinforcement learning.

[0026] Optionally, the multimodal response execution and feedback module includes a content generator and scheduler, a multi-channel collaborative output controller, and a feedback learning data pipeline;

[0027] The content generator and scheduler is used to instantiate the response strategy tuple into specific text, charts, interfaces, and behavioral instructions.

[0028] The multi-channel collaborative output controller is used to distribute instantiated content to the corresponding physical output devices according to the timeline and manage multi-channel synchronization;

[0029] The feedback learning data pipeline is used to collect interaction data across the entire process, providing a data source for training and optimization of the upstream model.

[0030] Optionally, the multi-channel collaborative output controller also includes a service quality monitoring and graded degradation fault tolerance mechanism, which is used to automatically adjust the response mode to ensure service continuity when an output failure or performance timeout is detected.

[0031] Optionally, the multimodal interaction perception module, the emotion and intent intelligent analysis module, the intelligent business decision and generation module, and the multimodal response execution and feedback module are connected sequentially through a data interface to form a closed-loop processing flow from user signal input to multimodal response output.

[0032] Optionally, the feedback learning data pipeline feeds back the user behavior data and interaction logs collected by the multimodal response execution and feedback module to the emotion and intent intelligent analysis module and the intelligent business decision and generation module for incremental training of the model and online optimization of decision strategies, forming a continuous self-optimizing learning loop.

[0033] Compared with the prior art, this application has at least the following beneficial effects:

[0034] This application integrates heterogeneous sensors with intelligent fusion processing based on the "embodied-disembodied" cognitive mapping theory. The system can simultaneously analyze multimodal signals such as voice, vision, and touch, and map low-level physical behaviors into structured primary semantic fragments. This breaks through the information limitations of a single modality and can capture complex intentions such as "directive inquiries accompanied by confused expressions." It provides high-quality input data rich in contextual evidence for subsequent in-depth analysis, improving the comprehensiveness and accuracy of intention perception.

[0035] This application employs a hierarchical multi-feature fusion analysis model and dynamically integrates pre-trained expert features for spoken text, multimodal contradictory signals, and long dialogue contexts. The system can effectively identify users' explicit and implicit emotions (such as sarcasm and anxiety) and perform refined analysis of complex business intentions involving multi-turn dialogues and professional terminology. Combined with an attention-based dynamic routing mechanism, the system can flexibly adapt to the analysis focus of different interaction scenarios, thereby achieving a more nuanced and humanized understanding of user states.

[0036] This application constructs a knowledge graph of power grid services and adopts a hybrid reasoning engine that combines rule-based reasoning, case-based reasoning, and graph path lookup. The system can perform dynamic retrieval and reasoning based on real-time sentiment, intent, user profiles, and power grid status. Combined with a multi-factor scoring model based on online optimization of reinforcement learning, the response strategy generated by the system not only highly matches the user's immediate needs but also adaptively balances multiple objectives such as service efficiency, emotional reassurance, and business value under the premise of business compliance. Attached Figure Description

[0037] Figure 1This is a schematic diagram of the module connection of an AI intelligent power supply service acceptance system based on multimodal interaction, provided as an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0039] This application provides an AI-powered intelligent power supply service acceptance system based on multimodal interaction, comprising:

[0040] The multimodal interaction perception module is used to collect users' voice, vision, and touch signals, and based on the interaction grammar rules of embodied-disembodied cognition mapping, it maps the raw behavioral data into primary fused semantic fragments.

[0041] The multimodal interaction perception module consists of a physical input layer and an intelligent fusion processing layer. Through the collaboration of heterogeneous sensors and fusion computing based on cognitive mapping, it achieves accurate perception and structured analysis of the user's complex interaction intentions.

[0042] The physical input layer integrates multiple sensor units to capture native multimodal physical signals generated during user interaction. Specifically:

[0043] Audio acquisition unit: Employs a circular microphone array, which not only acquires the user's voice audio stream but also uses beamforming technology to locate the sound source and separate ambient noise from direct sound. The raw audio signal is converted from analog to digital and output as a time-series data stream in Pulse Code Modulation (PCM) format.

[0044] Visual acquisition unit: Includes at least one high frame rate color camera and one depth sensor. The color camera acquires video streams of the user's face and upper body at a rate of no less than 30 frames per second for extracting facial feature points, micro-expressions, and body contours. The depth sensor simultaneously acquires corresponding 3D point cloud data for accurately calculating the spatial coordinates and motion trajectories of gestures and body postures;

[0045] Touch acquisition unit: a capacitive or infrared touch screen, which records the user's touch events on the display interface, including but not limited to click coordinates, swipe vectors, press duration and pressure level, and generates corresponding touch event sequences;

[0046] Auxiliary sensing unit: Optionally integrates surface electromyography (sEMG) sensors or millimeter-wave radar. The sEMG sensor is attached to the surface of the interactive terminal or wearable device, detecting weak electrical signals from the user's forearm or hand muscle groups to predict gesture intentions before physical actions are performed. Millimeter-wave radar is used for non-contact sensing, capable of penetrating slight obstructions such as clothing to accurately capture subtle physiological and motor signals such as chest rise and fall, and finger movements, especially serving as a visual supplement in privacy-sensitive or low-light scenarios.

[0047] Each sensor unit has a built-in hardware timestamp module, driven by a central synchronous clock source, to ensure that all acquired multimodal data streams have a unified time reference with microsecond-level precision, laying the foundation for subsequent spatiotemporal alignment.

[0048] The intelligent fusion processing layer receives multiple raw data streams from the physical input layer. Its core task is to perform calculations based on the "embodied-disembodied" cognitive mapping theory, converting low-level physical signals into high-level, semantically understandable interactive events. This processing specifically includes the following sub-steps:

[0049] Data preprocessing and feature extraction sub-steps: Preprocessing of each modality's data is performed in parallel. For the audio stream, endpoint detection, frame segmentation, and windowing are performed to extract acoustic features such as Mel-frequency cepstral coefficients (MFCC) and fundamental frequency. For the video stream, a convolutional neural network (CNN) model is used for real-time face detection, 68-point facial keypoint localization, and calculation of motion unit intensities related to the Facial Action Coding System (FACS); simultaneously, OpenPose-like algorithms are used to extract the 3D coordinate sequence of human skeletal joints from RGB-D data. For touch data, it is transformed into a temporal feature vector containing coordinates, pressure, and contact area. Electromyography (EMG) and radar signals are filtered and feature values ​​(such as root mean square and waveform length) are extracted respectively.

[0050] Spatiotemporal alignment and association sub-steps: Using a unified timestamp, heterogeneous features extracted within the same time window are aligned. For example, the start time of the user saying "here" is matched with the peak time of the gesture trajectory of "the user's index finger pointing to a specific area of ​​the screen" detected by the depth sensor. By establishing a unified spatial model with the user's human coordinate system and screen coordinate system as references, the screen coordinates of the gesture, the gaze area estimated by the gaze, and the touch coordinates are correlated and calculated to determine whether they point to the same interactive target;

[0051] The initial intent fusion sub-step based on interaction grammar: This step embeds a lightweight rule engine and pattern matcher, which predefines an interaction grammar rule library derived from the "embodied-disembodied" mapping theory. Rules are triggered by combinations of multimodal features extracted and associated in the previous step. For example:

[0052] Example rule R1: IF (detection of "frowning" facial expression action unit intensity > threshold T1) AND (speech pitch standard deviation > threshold T2) AND (keywords contain "how" or "again") THEN generate a temporary semantic label "user emotional state: anxiety / dissatisfaction" and increase the priority weight of this interaction event;

[0053] Example rule R2: IF (detects "index finger extension" gesture) AND (gesture vector points to screen coordinate system region (X1, Y1, X2, Y2)) AND (voice keyword "fee" or "here" exists within time tolerance Δt) THEN Generate temporary semantic label "suspected directional query: the directional region is related to fee-related information";

[0054] The output of this sub-step is not the final semantics, but a set of preliminary fused semantic fragments with confidence levels, time intervals, and spatial anchors. These fragments retain multi-channel evidence and, as structured intermediate representations, are sent to the downstream "sentiment and intent intelligent analysis module" for deeper context integration and final intent determination.

[0055] The emotion and intention intelligent analysis module is connected to the multimodal interaction perception module. It receives the primary fused semantic fragments and parses them using a hierarchical multi-feature fusion emotion analysis model, outputting the user's emotional state, refined intention, and confidence level.

[0056] The emotion and intent intelligent analysis module, serving as the system's cognitive center, receives preliminary fused semantic fragments from the perception module. It employs a hierarchical, multi-feature fusion deep analysis model to achieve accurate and refined identification of user emotional states and business intentions. This model consists of three sub-modules working collaboratively: feature pre-training, hierarchical dynamic fusion, and contextual understanding.

[0057] Specifically, the feature extraction and pre-training submodule constructs a transferable feature knowledge base, which contains three independently trained but complementary pre-trained models. , , ), respectively optimizing the typical and challenging corpus features in the power customer service scenario;

[0058] Model (Conversational Short Text Feature Model) This model focuses on processing conversational and informal short texts converted from Automatic Speech Recognition (ASR). Its training data primarily consists of a large amount of emotionally tagged conversational inquiries, exclamations, and complaints from electricity users (e.g., "Why is the electricity bill so expensive?", "The power is out again!"). The model architecture employs a one-dimensional convolutional neural network (1D-CNN) combined with an attention mechanism. The convolutional kernel size is typically set to (2, 3, 4) to capture common word order features of bigrams and triplets in spoken language, as well as the effect of modal particles. The features learned by this model are lexical and phrase-level representations with strong emotional associations.

[0059] Specifically, the model The specific architecture of the (colloquial short text feature model) employs a one-dimensional convolutional neural network (1D-CNN) as the core feature extractor. Its input is a sequence of word vectors processed by the embedding layer, with a word vector dimension of 100. The 1D-CNN contains one convolutional layer, which uses three sizes of one-dimensional convolutional kernels in parallel: 2, 3, and 4, corresponding to capturing local features of bigrams, triples, and quadruples in the text. Each size of convolutional kernel has 128 kernels, and the ReLU activation function is used. Following the convolutional layer is a global max-pooling layer to extract the most salient signal from each feature map. The output of the pooling layer is flattened and fed into a fully connected layer for dimensionality reduction and feature integration, ultimately outputting a 128-dimensional feature vector as an abstract representation of the colloquial short text.

[0060] Model (Multimodal Implicit Expression Feature Model): This model is specifically designed to identify implicit emotions, such as irony and metaphor, resulting from inconsistencies between text and paralinguistic information. Its input consists of multimodal aligned feature pairs, such as: {text word vector sequence, corresponding prosodic features of the speech segment (e.g., fundamental frequency contour, energy variation), and corresponding facial expression encoding for the time window (e.g., specific action unit intensity)}. The model employs a dual-tower structure: one tower processes the text, and the other tower processes the fused vector of acoustic and visual features. Finally, a cross-attention layer calculates the correlation or inconsistency between modalities, and the output layer predicts the implicit emotion category. Training data requires manual annotation of multimodal dialogue segments containing obvious or subtle irony or implied meaning.

[0061] Model The specific architecture of the (multimodal latent expression feature model) adopts a multimodal feature interaction structure based on Factorization Machine (FM). Specifically, the model first contains three independent feature encoding sub-networks:

[0062] The text encoder consists of a bidirectional GRU network with 64 hidden units, used to encode a sequence of text word vectors into a fixed-length context vector.

[0063] Acoustic feature encoder: It consists of a two-layer fully connected neural network with 64 and 32 neurons in each layer, and is used to process prosodic feature vectors such as fundamental frequency, energy, and spectral centroid extracted from speech signals;

[0064] Visual feature encoder: Similar in structure to acoustic encoder, it is used to process visual feature vectors such as the intensity values ​​of facial expression action units extracted from facial images.

[0065] The output vectors of the three encoders are concatenated and fed into a factorization machine layer. This factorization machine explicitly models the second-order interactions between the modal feature vectors, with the following formula: ,in Features The latent vector is set to 8 dimensions. The output of the factorization machine then passes through an attention layer to amplify contradictory signals related to implicit sentiment, ultimately outputting a 64-dimensional multimodal fusion feature vector.

[0066] Model (Long Context Structured Feature Model): This model is responsible for understanding the context involving multi-turn dialogues and complex business descriptions. Its input is a relatively long text sequence with dialogue history markers. The model typically employs a hierarchical attention network, with the first layer operating at the lexical level and the second layer operating at the statement / turn level, thereby constructing a document-level representation. This model focuses on learning the semantic relationships between electricity business entities (such as "peak-valley electricity pricing" and "business expansion application"), as well as the evolution logic of user intent within the dialogue.

[0067] Model The specific architecture of the (long context structured feature model) adopts a hierarchical attention network (HAN) architecture. The model first segments the input long text (containing multiple dialogue turns) into a sequence of sentences according to the dialogue turn;

[0068] Word-level attention layer: The word vector sequence in each sentence is first input into a one-way GRU encoding, which has 128 hidden layer units. Then, an attention layer generates weights for each word, resulting in a weighted sentence vector representation.

[0069] Turn-based attention layer: The sequence of all sentence vectors (i.e., turn vectors) output by the word-level attention layer is input into a second unidirectional GRU for encoding. This GRU also has 128 hidden layer units. Then, it passes through another turn-based attention layer to generate weights for each turn vector. Finally, they are aggregated to obtain a document-level vector representation that reflects the core content and structure of the long dialogue, with a dimension of 128.

[0070] The above architectural parameters (such as 128 convolutional kernels, 128 GRU hidden units, and 8 dimensions of FM hidden vectors) are for illustrative purposes only. In actual implementation, they can be adjusted accordingly based on computing resources and data scale.

[0071] The hierarchical fusion analysis submodule contains a multi-layer convolutional neural network (CNN) as the main analysis network, which introduces a dynamic feature routing mechanism based on attention weights.

[0072] The input to the main network is the structured data output by the perception module, which is usually organized into a feature tensor containing word embedding sequences, primary sentiment labels, primary intent labels, and associated multimodal feature vectors.

[0073] Before each convolutional operation (such as the Lth layer), a routing controller is set up. This controller computes a set of ternary attention weights based on the current input feature tensor. , , ), respectively corresponding to the pre-trained model , , The current relevance is determined. Weights are calculated using a small feedforward neural network or gating mechanism. Subsequently, the controller dynamically adjusts the weights based on the current relevance. , , Extract or activate the corresponding feature maps, and then weight and concatenate or add these external features with the output of the first layer of the main network, and use them together as the input of the Lth convolutional layer;

[0074] Different network layers emphasize different fusion methods. For example, shallower layers (closer to the input) may rely more on... The provided colloquial lexical features; in the intermediate layer, when the network begins to capture complex patterns... Provided multimodal contradictory features and The weights of the provided long-range dependent features may be increased. This mechanism enables the network to flexibly and selectively draw on external expert knowledge, achieving fine-grained capture of hybrid features;

[0075] Furthermore, the dynamic routing control mechanism in the hierarchical fusion analysis submodule functions to dynamically calculate and allocate three pre-trained feature models based on the current input features. , , The fusion weights are calculated using a weight calculation network and a weight optimization mechanism.

[0076] The weight calculation network structure is a small feedforward neural network. Its input is the feature tensor output by the previous network layer, which is compressed into a fixed-dimensional feature vector after global average pooling. This vector is then input into a fully connected network with a two-layer structure:

[0077] The first layer is a hidden layer, with the number of neurons set to half the dimension of the input vector, and a rectified linear unit (ReLU) is used as the activation function;

[0078] The second layer is the output layer, which has a fixed number of 3 neurons, corresponding to... , , The weights of the three models;

[0079] The raw scores of the output layer are processed by a differentiable normalization function to ensure that the sum of the three weights is 1 and to preserve gradient propagation capability;

[0080] To prevent the routing controller from prematurely converging to a fixed weight distribution pattern (i.e., "weight collapse") during training, a temperature coefficient-based Gumbel-Softmax technique is introduced after the output layer.

[0081] Specifically, during the training phase, for the three raw scores (logits) generated by the output layer, noise sampled from the Gumbel distribution is first added to them, and then a temperature parameter is used. The Softmax operation is performed on the noise score, and the formula is:

[0082]

[0083] in, This is Gumbel noise. High temperature. The initial value is set to 1.0 to make the weight distribution more uniform and encourage the model to fully explore combinations of different feature models. During training, The weights are gradually decayed to a small value (e.g., 0.1) using a linear or exponential scheduling method, causing the weight distribution to gradually tend towards a sharp deterministic choice. This process effectively prevents the controller from getting stuck in local optima in the early stages of training, thus learning a more robust dynamic routing strategy. During the inference phase, Gumbel noise is removed, and the deterministic weights are directly calculated using the Softmax function.

[0084] In addition, to enhance the interpretability and stability of routing decisions, L1 norm sparsity constraints can be added to the weight vector in the loss function to encourage the model to primarily activate one or two of the most relevant feature models in most cases.

[0085] Furthermore, the final output layer of the emotion and intention intelligent analysis module adopts a multi-task learning framework, and its overall loss function consists of emotion classification loss. Intent recognition loss and confidence regression loss The weighted summation is expressed as:

[0086]

[0087] in, , , For positive scalar weight coefficients, the following two optional implementation methods can be used to determine the optimal weight combination:

[0088] Static weight setting based on hyperparameter search predetermines a fixed set of optimal weights before model training. The specific steps include: First, defining a search space for the weight coefficients. For example, each weight takes a value from the set {0.1, 0.5, 1.0, 2.0, 5.0}, and traversing all possible combinations using grid search or random search;

[0089] Secondly, the model is initially trained for a fixed number of epochs on the training set using each set of weights, and its overall performance is evaluated on a separate validation set. The evaluation metric is either the weighted harmonic mean of each task metric or a custom scoring function set according to business importance. Finally, the weight combination with the highest overall performance score on the validation set is selected. These are fixed weights used during the final model training. This method is simple and stable, and suitable for scenarios where the relative importance between tasks is clear and the data distribution is relatively stable.

[0090] Dynamic weight adjustment based on task uncertainty: This method automatically and dynamically adjusts the loss weights for each task during training. Its core idea is to assign trainable parameters to the loss weights, associating them with the inherent uncertainty (noise) of each task. The weights are calculated using the following formula:

[0091]

[0092] in, , This is a trainable scalar parameter representing the homoscedasticity uncertainty of the task. At this point, the total loss function becomes:

[0093]

[0094] In the formula, This term serves as a regularization term to prevent the uncertain parameters from increasing indefinitely. During training, the model optimizes both the network's main parameters and these uncertain parameters simultaneously through backpropagation. Tasks with high uncertainty (and their losses) (High volatility) will automatically receive a smaller weight. Conversely, the same applies. This method enables dynamic adaptation of weights to the training process, reducing the burden of manual parameter tuning;

[0095] The two methods described above can be used individually or in combination. For example, an initial range can be determined first through hyperparameter search, followed by fine-tuning using dynamic adjustment. The loss function... Using cross-entropy loss, Using binary cross-entropy loss, Mean squared error loss is used. The final values ​​of the specific weights typically range from 0.1 to 5.0, and are objectively determined using the methods described above.

[0096] The context understanding and output submodule is responsible for integrating serialization information and generating the final decision, specifically including:

[0097] The feature sequence, which has undergone multiple layers of abstraction and is output from the hierarchical fusion analysis submodule, is input into a bidirectional gated recurrent unit (Bi-GRU) network. The Bi-GRU scans the feature sequence in both forward and backward directions to fully capture the dependency between the current user's statement and the content stated in the preceding dialogue, forming a final state vector rich in contextual information.

[0098] Based on the final state vector of Bi-GRU, three parallel output heads are connected to form a multi-task learning framework, as follows:

[0099] Emotion output head: A Softmax classifier is used to output discrete emotion polarity (such as positive, neutral, negative), and a regression layer (Sigmoid activation) is connected to output a scalar value of emotion intensity (0 to 1).

[0100] Intent Output Header: Employs a multi-label classifier (e.g., using multiple Sigmoid units) to output a refined set of intent labels. The label system is built upon a power business knowledge graph and can simultaneously contain multiple intents such as "query electricity bill details," "report a fault," and "inquire about peak-valley electricity pricing procedures."

[0101] Confidence output head: Through a small neural network, it integrates the clarity of the current input features, the distribution of activation values ​​within the model, etc., and outputs an overall confidence score between 0 and 1 to evaluate the reliability of the analysis results;

[0102] The entire module is trained end-to-end, with the loss function being a weighted sum of the losses of the three output heads. The Adam optimizer with gradient clipping is used, and the training is conducted on a large-scale labeled multimodal dialogue dataset for electricity customer service.

[0103] The intelligent business decision-making and generation module is connected to the emotion and intent intelligent analysis module. It has a built-in power grid service business knowledge graph and dynamic decision engine. Based on the emotional state, refined intent and multi-source context information, it performs retrieval and reasoning in the knowledge graph to generate structured personalized response strategy tuples.

[0104] The intelligent business decision-making and generation module analyzes user sentiment and intent from the module and transforms them into specific, actionable personalized service strategies. Internally, the module consists of three parts: a business knowledge graph, a dynamic decision engine, and a response strategy generator.

[0105] Specifically, the construction and structure of the business knowledge graph includes a built-in business knowledge graph specifically designed for the smart electricity service domain. This graph uses the Resource Description Framework (RDF) for data modeling, and its ontology defines the core entity classes, attributes, and relationships.

[0106] The core entities include: "users" (subcategorized into residential, commercial, and industrial categories), "service items" (such as "electricity bill inquiry", "fault reporting", "electric vehicle charging pile application", and "energy efficiency diagnosis"), "service channels" (such as "online APP", "voice customer service", and "physical business hall"), "business process nodes", "business rules", and "value-added service products".

[0107] Relationships and Attributes: Entities are connected through predefined relationship attributes. For example, the "Service Item" entity is associated with the required "User Information" through the hasPrerequisite attribute; with the corresponding "Business Process" through the triggers attribute; and with a specific "User Profile" pattern or "Emotional State" through the isRecommendedFor attribute. Each "Business Rule" is stored in the knowledge base in the logical form of "IF-THEN" and bound to the relevant entity.

[0108] Knowledge Sources and Updates: The initial knowledge of the knowledge graph originates from a structured analysis of the power grid company's "four-dimensional model" service framework (user needs, service content, channel methods, and model import). The knowledge graph is periodically synchronized with external business systems (such as marketing and production management systems) through pre-defined data interfaces to update dynamic information such as business processes and electricity pricing policies.

[0109] The dynamic decision engine's workflow involves receiving structured output from the analysis module, namely user sentiment tags (including intensity), a detailed list of intents, and confidence levels, and then performing a decision according to the following steps:

[0110] Context building: The engine first integrates multi-source contextual information, including: 1) user static profile (obtained from the customer profile system); 2) interaction history (recent service records, preferences); 3) real-time power grid status (obtained via API, current load, power outage information in the area, and electricity price periods); 4) current dialogue status (the preceding text in a multi-turn dialogue).

[0111] Graph Retrieval and Reasoning: Using "refined intent" and "sentiment tags" as key search terms, subgraph matching and retrieval are performed within the business knowledge graph. For example, for the intent "to query the reasons for excessively high electricity bills" and the sentiment "negative-confused," the engine will retrieve service nodes and rules related to "electricity bill composition," "peak and off-peak electricity analysis," and "energy-saving suggestions."

[0112] Candidate Strategy Generation and Ranking: Based on the retrieved subgraph structure and bound business rules, multiple feasible service response paths are inferred, forming a preliminary set of candidate strategies. Subsequently, each candidate strategy is ranked according to the following multi-factor scoring model:

[0113] Intent matching score: The semantic matching score between the candidate strategy and the user's main intent;

[0114] Emotional fit: The degree to which the soothing and explanatory content of the strategy fits the user's current emotional intensity (e.g., high anxiety is matched with a highly soothing strategy).

[0115] Business feasibility: Determine the immediate executableness of the strategy based on real-time grid status and user conditions (such as whether smart meters have been installed);

[0116] User value and business objectives: The potential value of strategies in enhancing user experience or promoting the conversion of value-added services;

[0117] The candidate strategy with the highest score is selected as the current decision outcome.

[0118] The core reasoning mechanism of the dynamic decision engine adopts a hybrid reasoning architecture that combines case-based reasoning (CBR) and rule-based reasoning (RBR), and introduces graph embedding technology to assist in similarity calculation in order to cope with complex and ever-changing interaction scenarios.

[0119] The hybrid inference architecture is as follows:

[0120] Rule-Based Reasoning (RBR) Component: This component maintains an editable business rule base, with rules stored in the form of "IF <condition> THEN <action>". The condition part can contain logical judgments about user sentiment, intent, profile attributes, and real-time status; the action part typically points to a specific service node or strategy template ID in the knowledge graph. For example: IF Sentiment = Anxiety AND Intent includes "Power Outage Query" THEN activate service node "S_Reassurance and Information Provision". The RBR component is responsible for performing condition matching, quickly triggering clear and standard business logic;

[0121] Case-Based Reasoning (CBR) component: This component maintains a historical interaction case library. Each case records the complete context (such as sentiment, intent, user type, and power grid status) and the ultimately validated response strategy. CBR is activated when Relational Logic (RBR) cannot find a perfectly matching rule or when the context is complex. Its workflow follows the classic four-step approach of "Retrieve - Reuse - Revise - Retain". In the retrieval phase, graph embedding technology is used: the current context and the context descriptions of historical cases are mapped to low-dimensional vectors. By calculating cosine similarity, the K most similar historical cases are quickly retrieved for subsequent reuse and revision.

[0122] Graph-based pathfinding: For decisions involving multi-step business processes, the engine directly performs pathfinding on the business knowledge graph. Utilizing a graph query language (such as Cypher), starting with the current user state and intent, and ending with feasible service goals, it searches for the optimal or alternative service paths and transforms them into a sequence of candidate strategies.

[0123] Conflicting rules (such as "Rule A: Soothe emotions" and "Rule B: End the conversation efficiently" being mutually exclusive in a specific scenario) are handled through the following layered mechanism:

[0124] Meta-rule layer: This layer defines the highest-priority conflict arbitration meta-rules. Meta-rules are formulated based on higher-dimensional business principles and real-time status. For example, a meta-rule could be defined as: "When real-time power grid monitoring indicates a large-scale fault and agent resources are strained, the principle of 'ensuring communication efficiency' takes precedence over the principle of 'personalized reassurance'." Meta-rules are executed first, selecting or suppressing underlying rules that conflict.

[0125] Rule priority and confidence weighting: Each business rule is assigned a static base priority and a dynamic confidence score. The confidence score is dynamically updated based on factors such as the matching degree of the conditions that triggered the rule and the historical execution success rate. When multiple rules are triggered simultaneously and conflict, the system calculates the weighted score of each rule (base priority × dynamic confidence score) and executes the rule with the highest weighted score first.

[0126] Fuzzy reasoning and strategy fusion: For conflicting objectives that are not absolutely opposed, strategy fusion is used instead of an either-or choice. For example, by searching similar cases, a compromise strategy template can be found that can both partially soothe emotions (such as sending a short sympathetic message) and quickly guide users to a self-service query portal (such as sending a one-click progress query link), thus achieving a balance between conflicting objectives.

[0127] This hybrid reasoning mechanism combines the explicitness of RBR, the flexibility of CBR, and the relevance of graph computation, and ensures the rationality and business adaptability of the decision results through a hierarchical conflict resolution strategy.

[0128] Furthermore, the multi-factor scoring model used for candidate strategy ranking in the dynamic decision engine employs a hybrid mechanism combining static initialization and dynamic online learning to determine and optimize the weights of each factor, thereby achieving continuous self-optimization of the system's decision-making capabilities.

[0129] During the initial deployment of the model, each scoring factor (such as intent matching degree) Emotional compatibility Business feasibility User / Business Value The weights are statically configured. Two initialization methods can be chosen:

[0130] Assigning values ​​based on expert experience: Initial weight vectors are directly set by domain experts according to business priorities. For example, in service recovery scenarios, settings can be configured. (Emotional fit) has a high weight; in value-added service promotion scenarios, it can be set (User / business value) has a high weighting;

[0131] Based on offline supervised learning: Historical high-quality service interaction records are collected as the training set. Each record contains a context and a response strategy manually labeled as "optimal". A set of weights that best matches the model's scoring results with the manually selected optimal choice is obtained using backpropagation algorithms (such as linear regression or gradient descent). This method provides a data-driven basis for the initial weights.

[0132] After the system is deployed and running, the dynamic optimization of weights is achieved through a lightweight reinforcement learning module.

[0133] Modeled as a Markov Decision Process (MDP):

[0134] State(s): Defined as the contextual feature vector of the current interaction, including user sentiment, intent, profile, real-time state, etc.

[0135] Action (a): Defined as the weight vector of the scoring model Fine-tuning, such as making small increases or decreases in the direction of a certain basis vector;

[0136] Reward (r): Calculated after each interaction. The reward signal combines objective business metrics and subjective satisfaction; an example of its calculation formula is as follows: .in, , , As a harmonic coefficient, the business completion indicator comes from feedback from the backend system (such as work order closure), and the user satisfaction score comes from manual or automatic evaluation at the end of the interaction;

[0137] The learning algorithm employs online learning algorithms such as policy gradient methods (e.g., the REINFORCE algorithm) or asynchronous advantage action evaluation (A3C). The agent (i.e., the weight optimization module) optimizes the weights based on the current state. Select Action (Fine-tuning weights), generating post-decision strategies and user interactions, and receiving rewards after the interaction ends. This reward is used to evaluate the merits of weight adjustment actions and updates the agent's parameters through policy gradients, making it more inclined to choose weight adjustment policies that yield higher long-term cumulative rewards. Weights The updates are performed asynchronously in the background, and the newly learned weights are periodically (e.g., every N interactions) synchronized to the online decision engine.

[0138] This hybrid mechanism ensures that the scoring model possesses both robust initial business logic and the ability to adaptively optimize decision preferences through continuous interaction with the environment, ultimately improving overall service efficiency and user satisfaction. The dynamic optimization module can be started and stopped independently, and its hyperparameters such as learning rate and exploration rate can be adjusted through the configuration interface.

[0139] A response strategy generator that transforms decision outcomes into concrete, presentable response strategy tuples;

[0140] Policy tuple structure: A complete response policy definition is a structured data object that includes, but is not limited to, the following fields:

[0141] Core objective: The primary business objective to be achieved in this interaction (e.g., "answering questions", "soothing emotions", "facilitating processing").

[0142] Content Component Sequence: An ordered list of components, each defining a specific content type, key information points, and tone. For example: [Component Type: Apology and Empathy, Tone: Sincerity]; [Component Type: Factual Statement, Key Information: Planned Maintenance, Recovery Time]; [Component Type: Action Recommendation, Key Information: Subscription to Power Outage Notification Service];

[0143] Channel and Modality Assignment: Specify the recommended output modality (such as voice broadcast, screen graphics, push notifications) and specific channel (such as app pop-ups, SMS) for each content component;

[0144] Anticipated subsequent interaction points: Predict possible subsequent user actions and link them to the corresponding service nodes to prepare for continuous dialogue;

[0145] Personalized population: The generator populates variables in the strategy tuple based on user profiles and real-time data. For example, it fills in the "Estimated Recovery Time" field with the specific time obtained from the production management system; and it provides customized suggestions in the "Energy Saving Suggestions" field based on the user's historical electricity consumption habits.

[0146] Output: This structured strategy tuple is sent to the "Multimodal Response Execution and Feedback Module" as a blueprint for generating the final voice, text, and interface interactions;

[0147] The multimodal response execution and feedback module is connected to the intelligent business decision and generation module. It transforms the response strategy tuple into specific multimodal output content, drives the physical device output through a multi-channel collaborative controller, and records the data of the entire interaction process to form a feedback learning closed loop.

[0148] The multimodal response execution and feedback module is responsible for transforming the structured policy tuples output by the decision-making module into user-perceptible multimodal interactions, and collecting feedback data to drive continuous system optimization. The module consists of three parts: a content generation and scheduler, a multi-channel collaborative output controller, and a feedback learning data pipeline.

[0149] The content generator and scheduler receives policy tuples from the response policy generator and is responsible for instantiating the abstract instructions in them into specific media content and interaction instructions;

[0150] Multimodal content generation engine: Based on the sequence of content components in the strategy tuple, it calls the corresponding content generator, specifically including:

[0151] Natural Language Generation (NLG): For voice broadcasting and on-screen text display, a template- and rule-based approach is employed. The system has a built-in atomic voice / text template library, with pre-reserved variable slots (such as {user name}, {recovery time}, {electricity bill}). Based on the selected template type (such as "appeasement apology", "factual explanation", "action guidance") and the specified variables, the engine generates grammatically correct and contextually appropriate natural language sentences in real time. For complex explanations (such as electricity bill composition), pre-arranged graphic scripts with associated tags can be invoked.

[0152] Visual content generation: For visual content such as infographics and animations, the system has a built-in series of chart templates (such as pie charts, bar charts, and map overlays) and animation sequences. Based on strategies and data, the engine generates corresponding chart data and rendering instructions in real time by calling visualization libraries (such as ECharts and D3.js wrapper interfaces) or by retrieving them from the cache. For example, it can generate an interactive line chart showing a user's electricity consumption trend over the past 12 months.

[0153] Interactive interface element assembly: Based on strategy requirements, generate corresponding graphical user interface (GUI) instructions, including but not limited to the layout, style, and associated business logic callback function identifiers for buttons, forms, lists, and pop-ups. These instructions are passed to the front-end rendering framework in a structured description language (such as JSON-formatted UI descriptors);

[0154] Virtual Avatar Behavior-Driven: If the terminal is equipped with a virtual avatar, the system schedules corresponding animation sequences and facial expression parameters from the behavior library based on the user's emotional intensity and strategic tone. The behavior library includes various preset actions such as neutral, smiling, nodding, and apologetic gestures, which are fine-tuned through parameters (such as nodding amplitude and speech rate) to match the emotional intensity.

[0155] The generator is responsible for arranging the content elements of the different modalities into a synchronous timeline according to the content component sequence order and timing requirements defined in the policy tuple, so as to ensure the continuity and consistency of multimodal output.

[0156] The multi-channel collaborative output controller is responsible for accurately distributing the orchestrated content timeline to different physical output devices and managing the synchronization between channels, specifically including:

[0157] Device Abstraction Layer: This layer defines a unified device control interface, abstracting specific hardware such as display screens, speaker arrays, programmable LED strips, and linear motors. Each device driver implements standardized playback (content segment, start timestamp, parameters) and stop interfaces.

[0158] Synchronization and Triggering Mechanism: The controller maintains a high-precision internal clock. At the start of output, a synchronization start signal is broadcast to all relevant devices. Each device triggers its output at a precise moment based on pre-allocated timestamps and content segments on the orchestrated timeline. For example, when the text-to-speech (TTS) program plays the word "peak electricity cost," the corresponding "peak" section in the screen graph will highlight and flash synchronously. This is achieved by aligning the time stamps of the speech waveform with the keyframes of the graphic animation on the same timeline.

[0159] Cross-modal redundancy and complementarity: To ensure reliable information delivery, the controller can implement cross-modal redundancy (e.g., important notifications are simultaneously prompted with voice, text, and flashing lights) or complementarity (e.g., voice describes overall trends, while on-screen charts display detailed data) based on the strategy. In situations with limited channels (e.g., noisy environments), the proportion of visual information can be automatically increased.

[0160] The feedback learning closed-loop data pipeline is responsible for collecting data from the entire interaction chain, providing a data foundation for the optimization of upstream models.

[0161] The system uses a unique session ID as an index and, with user authorization and privacy anonymization, records the following data in a structured manner for a complete interaction:

[0162] Input log: Feature summary of the original multimodal signal (not original audio and video), and primary semantic fragments output by the fusion processing layer;

[0163] Analysis Log: The sentiment and intent intelligent analysis module outputs sentiment polarity, intensity, intent list, and confidence level;

[0164] Decision log: Knowledge subgraph retrieved by the dynamic decision engine, candidate strategies and their scores, and the final selected strategy tuple;

[0165] Execution log: The actual content template called, the specific text / chart generated, the output device channel, and the precise timestamp;

[0166] User behavior logs: Real-time reactions of users during and after the interaction, including: voice interruption, touch operation (click, swipe), facial expression changes (through real-time analysis), human satisfaction rating after the session ends (if any), and subsequent key business actions (such as processing recommended services or clicking on provided links).

[0167] Log data is stored in the data lake in both structured and unstructured formats. Through an ETL process, the logs are transformed into datasets in a specific format for model retraining, or into business metrics for evaluating strategy effectiveness (such as the conversion rate of strategy A among users with negative sentiment).

[0168] Feedback learning can be performed offline in batches or online in real time for fine-tuning, mainly including:

[0169] Offline batch learning: Regularly (e.g., weekly) using accumulated new data to train sentiment analysis models (especially pre-trained feature models). , , Perform incremental training or fine-tuning;

[0170] Online reinforcement learning: The user's subsequent behavior (such as whether to accept the suggestion) is used as a real-time reward signal and fed back to the weight optimization module of the multi-factor scoring model in the dynamic decision engine (as described above) to achieve online adaptation of the decision strategy.

[0171] Furthermore, to ensure the robustness and continuity of the service, a fault tolerance mechanism including Quality of Service (QoS) monitoring and graded degradation strategies is designed when a certain modality generation fails (such as chart rendering timeout) or the output device malfunctions, in order to cope with abnormal situations such as component failure and performance fluctuations.

[0172] Quality of Service (QoS) monitoring: The system has a built-in lightweight real-time monitoring agent that continuously tracks key performance indicators (KPIs).

[0173] Output channel health status: The connection status and availability of each physical output device (such as displays and speakers) are monitored through regular heartbeat and response latency tests. Simultaneously, the process status and resource usage of each software content generation service (such as chart rendering engine and TTS engine) are monitored.

[0174] End-to-end response latency: In each interaction, record the total time from the end of user input to the start of the system's multimodal response output. The system sets maximum allowable latency thresholds (SLAs) for different priority services (e.g., fault reporting is high priority, and regular queries are normal priority).

[0175] Content Generation Status: Monitors the completion status and time taken for each modal content generation task. For example, it can be set that chart rendering must be completed within X milliseconds, otherwise it is considered a timeout;

[0176] When the monitoring system detects a fault or substandard performance, the multi-channel collaborative output controller will automatically adjust its response according to a preset, tiered degradation strategy. Its decision-making follows these hierarchical levels:

[0177] Level 1 Degradation (Single-Modal Failure): When a single output device or content generation service fails, the system migrates information to other available channels with a brief explanation. For example, if the display screen fails, all visual information (charts, text) is converted into a detailed voice description, with the message: "Relevant information has been read aloud for you." If the TTS engine fails, all voice text is converted to on-screen text and a clear visual cue (such as screen flashing) is triggered.

[0178] Level 2 Degradation (Performance Timeout): When content generation time approaches or exceeds the SLA threshold, the system activates "Simplified Output" mode. For example, if a complex energy efficiency analysis chart times out, it is automatically replaced with a concise data table or a text summary of key conclusions to ensure that core information is output within the time limit. For virtual avatars, if the behavioral computation load is too high, complex body animations are paused, retaining only basic lip-sync and facial expressions.

[0179] Level 3 Degradation (Severe Resource Constraints): When network bandwidth is extremely low or computing resources are severely insufficient, the system switches to "Minimum Guarantee Information" mode. In this mode, only the most reliable single channel (usually voice or basic text) is used to output the most concise and unadorned business results, and the user is advised: "The current network is poor. A detailed report has been sent to your APP message center. Please check back later."

[0180] Whenever a degradation action is triggered, the system generates a standard status code and a brief log. Additionally, if the degradation might impact user experience, the controller will dispatch a mild, non-technical notification (such as "For faster response, the display has been simplified for you") to inform the user of the change in the current interaction mode.

[0181] Based on real-time analysis of historical average response latency and current system load, the system can proactively predict potential risks at the initial interaction stage and pre-select content templates of appropriate complexity. For example, during peak system load periods, for non-urgent power consumption analysis requests, it may directly provide conclusive text instead of interactive charts. This pre-adjustment, combined with monitoring-based passive degradation, ensures stable service capabilities and an acceptable user experience under various operating conditions.

[0182] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An AI-powered intelligent power supply service acceptance system based on multimodal interaction, characterized in that, include: The module is connected in sequence to the following modules: multimodal interaction perception module, emotion and intent intelligent analysis module, intelligent business decision and generation module, and multimodal response execution and feedback module; The multimodal interaction perception module collects the user's voice, visual and touch signals, and maps the original behavioral data into primary fused semantic fragments based on interaction syntax rules, and outputs them to the emotion and intention intelligent analysis module. The emotional and intention intelligent analysis module receives the primary fused semantic fragment, parses it through a hierarchical multi-feature fused emotional analysis model, and outputs the user's emotional state, refined intention, and confidence level to the intelligent business decision and generation module. The intelligent business decision-making and generation module retrieves and reasons in the power grid service business knowledge graph based on the emotional state, refined intent and multi-source context information, generates a structured personalized response strategy tuple, and outputs it to the multimodal response execution and feedback module. The multimodal response execution and feedback module transforms the response strategy tuple into specific multimodal output content and outputs it to the user through a physical device, while recording interaction data to form a feedback learning loop.

2. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claim 1, characterized in that, The multimodal interactive perception module includes a physical input layer and an intelligent fusion processing layer; The physical input layer integrates an audio acquisition unit, a visual acquisition unit, a touch acquisition unit, and an optional auxiliary sensing unit, used to capture native multimodal physical signals generated by user interaction; The intelligent fusion processing layer is used to perform data preprocessing, feature extraction, spatiotemporal alignment and association on the physical signal, and to perform preliminary fusion based on predefined interaction syntax rules to generate the primary fusion semantic fragment.

3. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claim 1, characterized in that, The emotion and intention intelligent analysis module includes a feature pre-training submodule, a hierarchical dynamic fusion submodule, and a context understanding submodule; The feature pre-training submodule contains multiple pre-trained models, which are used to extract spoken text features, multimodal implicit expression features, and long contextual structured features, respectively. The hierarchical dynamic fusion submodule fuses the features extracted by the pre-trained model with the main network through a dynamic routing control mechanism, and outputs an abstract feature sequence. The context understanding submodule integrates the abstract feature sequence through a sequence model and outputs sentiment state, refined intent, and confidence level using a multi-task learning framework.

4. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claim 1, characterized in that, The intelligent business decision-making and generation module includes a business knowledge graph, a dynamic decision engine, and a response strategy generator. The business knowledge graph stores entities, attributes, and relationships within the power grid service domain; The dynamic decision engine is used to receive the emotional state and refined intent, combine multi-source contextual information, perform retrieval, reasoning and candidate strategy ranking in the knowledge graph, and select the optimal response strategy. The response strategy generator is used to transform the selected strategy into a structured response strategy tuple containing core objectives, content component sequences, channel allocation, and personalized fill fields.

5. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claim 4, characterized in that, The dynamic decision engine adopts a hybrid reasoning architecture that combines rule-based reasoning, case-based reasoning, and graph pathfinding, and is equipped with a rule conflict resolution mechanism.

6. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claim 4, characterized in that, The dynamic decision engine also includes a multi-factor scoring model for ranking candidate strategies. The factor weights of this model are adjusted using a hybrid mechanism that combines static initialization with dynamic online optimization based on reinforcement learning.

7. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claim 1, characterized in that, The multimodal response execution and feedback module includes a content generator and scheduler, a multi-channel collaborative output controller, and a feedback learning data pipeline. The content generator and scheduler is used to instantiate the response strategy tuple into specific text, charts, interfaces, and behavioral instructions. The multi-channel collaborative output controller is used to distribute instantiated content to the corresponding physical output devices according to the timeline and manage multi-channel synchronization; The feedback learning data pipeline is used to collect interaction data across the entire process, providing a data source for training and optimization of the upstream model.

8. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claim 7, characterized in that, The multi-channel collaborative output controller also includes a service quality monitoring and graded degradation fault tolerance mechanism, which is used to automatically adjust the response mode to ensure service continuity when an output failure or performance timeout is detected.

9. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claims 1-8, characterized in that, The multimodal interaction perception module, emotion and intent intelligent analysis module, intelligent business decision and generation module, and multimodal response execution and feedback module are connected sequentially through a data interface to form a closed-loop processing flow from user signal input to multimodal response output.

10. The AI-powered intelligent power supply service acceptance system based on multimodal interaction according to claim 9, characterized in that, The feedback learning data pipeline feeds back the user behavior data and interaction logs collected by the multimodal response execution and feedback module to the emotion and intent intelligent analysis module and the intelligent business decision and generation module for incremental training of the model and online optimization of decision strategies, forming a continuous self-optimizing learning loop.

Images