A multi-modal human-computer interaction platform system and method based on a central coordination layer

Abstract

This invention discloses a multimodal human-computer interaction platform system and method based on a central coordination layer. The invention uniformly converts multimodal input data from edge interaction units into structured semantic event objects. These objects encapsulate at least semantic intent, emotional context, privacy compliance tags, and policy identifiers, serving as the sole transmission unit for cross-module and cross-device interaction. Within the central coordination layer, capabilities for emotion understanding, personalized agents, privacy compliance verification, and dynamic scheduling of computing resources are integrated and exposed externally through standardized interfaces. This allows third-party developers to directly access platform capabilities without repeatedly building underlying models, privacy compliance, and cross-device consistency infrastructure, effectively ensuring consistency, security, and efficiency in cross-application and cross-device interaction, and significantly reducing development complexity and cost.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more particularly to multimodal human-computer interaction and artificial intelligence platform technology. More specifically, this invention provides a multimodal human-computer interaction platform system and method based on a central coordination layer. The system and method are used to achieve efficient semantic event object construction, personalized response generation, and intelligent scheduling based on edge or cloud hybrid architecture. The platform's core capabilities are exposed to the outside world in the form of service interfaces, while supporting privacy classification and multi-tenant policy control. Background Technology

[0002] With the development of artificial intelligence, especially the advancements in natural language processing (NLP) and computer vision (CV), multimodal human-computer interaction is becoming an important direction for the evolution of smart terminals. Smart speakers, companion robots, and interactive devices with AI functions have emerged in the market, possessing voice, vision, and environmental perception capabilities.

[0003] Existing multimodal interaction implementation paths mainly fall into two categories: The first is the edge-integrated solution, which directly deploys inference models such as Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) on the terminal device to reduce response latency and avoid data leakage. However, this solution is limited by terminal computing power, power consumption, and cost, making it difficult to support large-scale models and complex inference, and its personalization and contextual understanding capabilities are limited. The second is the cloud processing solution, which uploads environmental data such as audio and images collected by the terminal to the cloud, where cloud-based models generate responses and then distribute them. While this solution has strong understanding and generation capabilities, it relies on a stable network connection, and its performance significantly decreases in weak network or offline scenarios. It also faces high cloud computing costs and end-to-end latency, and introduces stricter privacy and compliance constraints, as raw environmental data often needs to be uploaded to a remote location for processing.

[0004] However, even under the two paths mentioned above, existing multimodal interaction systems still generally suffer from several technical bottlenecks:

[0005] Multimodal timing and consistency issues. Data from microphones, cameras, and environmental sensors often differ in sampling frequency, arrival time, and confidence level. The lack of stable cross-modal alignment, fusion, and consistency resolution mechanisms makes it difficult to form high-level semantic representations that can be stably consumed by upper layers.

[0006] Personalization and long-term memory are difficult to retain and continue across devices. User preferences, historical conversations, and contextual states are usually stored separately in proprietary formats for each device or application, lacking portable and reusable long-term memory summaries. This results in unstable personalization quality across different scenarios and difficulty in maintaining it.

[0007] Privacy controls are coarse-grained. Existing implementations are mostly simple start-stop or link-level protections, lacking data hierarchical labels and tenant / application policy identifiers that can be bound to specific semantic units, making it difficult to implement fine-grained, auditable access constraints across different processing paths or different callers.

[0008] The scheduling of computing power across endpoints, edges, and the cloud lacks an auditable dynamic routing mechanism. Current systems often struggle to perform fine-grained adaptive allocation and fallback control between local lightweight models, edge computing nodes, and large cloud models, making it difficult to achieve a controllable balance between response latency, intelligent experience, and computing power / bandwidth costs.

[0009] There is a lack of standardized platform interfaces. Many products adopt a "siloed" (vertically fragmented) architecture, which tightly couples perception, reasoning, strategy, and business logic, lacking a unified capability abstraction and standardized application programming interfaces (APIs). This means that third-party developers often need to solve complex issues such as multimodal fusion, model deployment optimization, and privacy compliance on their own, resulting in high development and maintenance barriers, difficulty in reusing capabilities, and difficulty in expanding the ecosystem.

[0010] Insufficient observability and traceability. Existing systems lack structured records and replayability of the basis for reasoning / scheduling decision-making processes (such as resource routing, privacy constraint judgment, and policy rollback triggering), making it difficult to meet the needs of compliant delivery, accountability, and continuous optimization.

[0011] Based on the above problems, there is an urgent need in this field for a new technical architecture: on the one hand, it should be able to reliably fuse multimodal inputs into high-level semantic representations that can be consumed by machines, and dynamically schedule them among local devices, edge nodes, and cloud resources based on latency, cost, and compliance constraints; on the other hand, it should refine capabilities such as multimodal perception, long-term memory accumulation, personalized response generation, privacy classification, and multi-tenant policy control into standardized, auditable, and reusable service interfaces, so that third-party developers can reuse these capabilities without directly accessing the original identifiable data, thereby achieving consistency, security, and maintainability across applications, devices, and scenarios.

[0012] In view of this, the present invention is proposed. Summary of the Invention

[0013] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a multimodal human-computer interaction platform system and method based on a central coordination layer, thereby solving the problems mentioned in the background art.

[0014] To solve the above-mentioned technical problems, the basic concept of the technical solution adopted by the present invention is as follows:

[0015] A multimodal human-computer interaction platform system and method based on a central coordination layer includes: a semantic event conversion module, a personality proxy module, and a developer interface module. The semantic event conversion module receives multimodal input data from voice, image, and sensors, and converts the input into structured semantic event objects. These semantic event objects serve as unified transmission units across modules and devices. The personality proxy module combines the semantic event objects with the user's historical interaction records and behavioral preferences to generate personalized interactive responses. The developer interface module provides the semantic event objects and personalized interactive responses to the outside world in the form of standardized application programming interfaces (APIs) or service interfaces. The output is only exposed in the form of de-identified and digested structured objects. Furthermore, the service interface supports multiple types of function calls, and the output granularity and access permissions are constrained by privacy level tags and tenant / application policy identifiers. This allows developers to focus solely on application business logic while calling platform capabilities to build scenario-based applications. All external responses from the platform are derived from a non-sensitive subset view of the semantic event objects.

[0016] Preferably, the system includes one or more edge interaction units, a multimodal acquisition interface, a local preprocessing module, and a communication module. The edge interaction unit is used to acquire multimodal input data and perform preliminary processing. The edge interaction unit communicates with the central coordination layer through a network to transmit semantic event objects and / or receive personalized response strategies. The multimodal acquisition interface is used to acquire voice, image, and / or environmental sensing data. The local preprocessing module is used to perform wake-up detection, automatic speech recognition and / or multimodal feature extraction, time alignment and / or confidence estimation and / or cross-modal fusion, and privacy desensitization processing, and reports the processing results in the form of standardized features or directly generated semantic event objects. The communication module is used to interact with the central coordination layer for data and control commands.

[0017] Preferably, the developer interface module further includes multiple service interfaces. The service interfaces only output objects that have been de-identified and digested, do not expose the original audio / image byte streams, and are subject to access control by privacy level tags and tenant / application policy identifiers. The service interfaces include at least a semantic event interface, a memory management interface, an execution policy interface, a user insight interface, a context management interface, and a model adaptation interface.

[0018] A multimodal semantic event interaction method based on a central coordination layer includes the following steps:

[0019] Step 1, Multimodal Input Acquisition: Voice, image and / or environmental sensing data are acquired by one or more edge interaction units, and preprocessing is performed to remove noise, perform time alignment and privacy desensitization;

[0020] Step 2, Semantic Event Generation: The preprocessed data is analyzed using the semantic event transformation module to generate structured semantic event objects. The semantic event objects include at least a weighted fusion calculation of sentiment indicators, a context summary generated based on historical dialogues, and a privacy level label for access control.

[0021] Step 3: Personalized Strategy Generation: The personality proxy module combines semantic event objects, user long-term memory archives, and personality parameter sets to generate personalized interaction response strategies.

[0022] Step 4: Service Interface Encapsulation: The semantic event object and personalized response strategy are encapsulated into a service interface through the developer interface module. The service interface supports multiple function calls, including semantic event interface, memory management interface, execution strategy interface, user insight interface and context management interface.

[0023] Step 5: Scenario-based application invocation: The service interface is invoked by the application, allowing developers to focus solely on writing business logic and quickly build scenario-based applications using the platform's capabilities.

[0024] Preferably, the semantic event conversion module is used to implement step two of the multimodal semantic event interaction method to generate a structured semantic event object, the personality proxy module is used to implement step three of the multimodal semantic event interaction method to generate a personalized interaction response strategy, and the developer interface module is used to implement steps four and five of the multimodal semantic event interaction method to encapsulate the service interface and provide it for application calls.

[0025] Preferably, the central coordination layer includes a scheduling module for calculating a synthetic routing score based on emotional urgency assessment and task complexity assessment, and selecting computing resources accordingly to generate and output personalized interactive responses. Specifically, the emotional urgency assessment is calculated based on the emotional recognition results in the semantic event object; the task complexity assessment is based on the weight of attribute keywords, context length, and / or estimation of required model resources; the synthetic routing score can be obtained at least by weighting emotional urgency and task complexity with configurable weights, not limited to a linear form, and compared with multi-level thresholds to dynamically select computing resources; the scheduling module further performs online adaptive updates of the weights and thresholds based on network latency, device load, and cost budget; the scheduling and resource selection process follows privacy and compliance information, which is attached to the semantic event object and includes at least a privacy level label and a tenant / application policy identifier; the scheduling module associates and stores decision logs and indicators with the corresponding semantic event objects for auditing and subsequent optimization; the scheduling module supports parallel pre-routing and prefetching to reduce cold start latency, and performs resource selection and result merging for different subtasks in multi-path composite response scenarios.

[0026] Preferably, the central coordination layer includes a personalized execution service module, used to provide the personalized interactive response decision-making process to external parties as a service, specifically including:

[0027] The personalized execution service module receives a structured semantic event object provided by the semantic event interface, and combines it with the user's long-term memory profile and personality parameter set to generate a personalized response strategy. The strategy includes at least tone / style parameters, output carrier selection, and interaction rhythm.

[0028] Personalized response strategies and corresponding execution constraints (including security / compliance rules and privacy classifications) are encapsulated together as callable execution strategy interfaces for application triggering, and the strategies can be refined into speech synthesis output, device control commands or scenario-based linkage operations.

[0029] The personalized execution service module supports policy self-adaptation and version management in different application scenarios, and records the basis for policy generation, scoring indicators and call logs to facilitate auditing, reproduction and continuous optimization.

[0030] Preferably, the central coordination layer includes a user insight service module, used to provide user profiles / growth representations generated based on long-term interaction data as a service, specifically including:

[0031] The user insight service module constructs a long-term memory archive based on structured semantic event objects provided by the semantic event interface. The archive includes at least an interaction event timeline, a weighted record of sentiment intensity and confidence, behavioral preference tags, and a context summary, and supports data deduplication, standardization, and alignment from multiple edge interaction units.

[0032] The user insight service module calculates longitudinal metrics, including but not limited to sentiment trends, interaction frequency / time period distribution, task completion rate, compliance / churn risk score, and scenario preference. These metrics can be updated by combining time decay and smoothing strategies.

[0033] Based on privacy level labels and access policies, the user insight service module provides user profiles to the public in an anonymized and summarized form, prohibiting the output of original audio / images and data that can directly identify individuals. It can optionally apply differential privacy noise or binning processing to the summary results.

[0034] The user insight service module provides versioning and traceability capabilities, records the data range, model / rule version and scoring criteria on which the profile is generated, and outputs auditable generation logs;

[0035] The User Insights service module provides query and subscription capabilities through the User Insights interface, supporting filtering and aggregation by time window, scenario domain, and metric dimension, which can be used by developers or content providers for personalized strategy configuration and effect evaluation.

[0036] Preferably, the central coordination layer includes a context management service module, used to provide the method for fusing multimodal inputs and generating unified context objects as a service, specifically including:

[0037] The module receives asynchronous data streams from one or more edge interaction units, including voice, image and / or environmental sensing data, and performs time alignment and source identification on each data stream;

[0038] Feature extraction and confidence evaluation are performed on the aligned data stream, and a unified context object is generated by the unified context generation unit according to a configurable weighted decision / learning fusion strategy. The object includes at least emotional state, spatial / temporal context, environmental conditions and task priority.

[0039] During the integration process, privacy level labels and access policies are followed to minimize processing and de-identify output, prohibiting the exposure of original audio / images and data that can directly identify individuals.

[0040] The unified context object and semantic event interface are linked to provide query, subscription and replay capabilities. It supports filtering and aggregation based on time window, spatial region and scene domain, and performs deduplication / consistency parsing for duplicate or conflicting events.

[0041] The context management service module supports scenario switching and collaboration: when multiple edge interaction units report events simultaneously, the dominant context is selected based on device proximity, sensor coverage, and event confidence. Buffering, degradation, and rollback strategies are executed when the network fluctuates or the device is unavailable to ensure context continuity.

[0042] A non-transitory computer-readable medium, characterized in that: it stores computer instructions thereon, which, when executed by a processor, cause a central coordination layer and / or one or more edge interaction units to perform a method for multimodal semantic event interaction, specifically including:

[0043] Generate semantic event objects, which include fused sentiment indicators, context summaries based on historical dialogues, and privacy level tags;

[0044] A response strategy is generated based on the semantic event object, long-term memory, and personality parameters, and the output granularity is controlled according to the privacy level label.

[0045] The routing score is calculated based on the urgency of the emotion and the complexity of the task, and resources are dynamically selected between local lightweight models, edge units or cloud models through multi-level thresholds.

[0046] The routing scores, resource selections, latency and cost metrics are associated with and stored with the corresponding semantic event objects for auditing and optimization.

[0047] Processing results are provided externally as a service through semantic events, memory management, execution strategies, user insights, or context management interfaces.

[0048] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all of the following advantages at the same time:

[0049] The core concept of this multimodal human-computer interaction platform system and method based on a central coordination layer lies in uniformly converting multimodal input data from edge interaction units into structured semantic event objects. These objects encapsulate at least semantic intent, emotional context, privacy compliance tags, and policy identifiers, serving as the sole transmission unit for cross-module and cross-device interactions. Within the central coordination layer, capabilities for emotion understanding, personalized agents, privacy compliance verification, and dynamic scheduling of computing resources are integrated and exposed externally through standardized interfaces. This allows third-party developers to directly access platform capabilities without repeatedly building underlying models, privacy compliance, and cross-device consistency infrastructure, effectively ensuring consistency, security, and efficiency in cross-application and cross-device interactions, and significantly reducing development complexity and costs.

[0050] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0051] The accompanying drawings described below are merely some embodiments. Those skilled in the art can obtain other drawings based on these drawings without any creative effort. In the drawings:

[0052] Figure 1 This is a schematic diagram of the architecture of the multimodal human-computer interaction platform system of the central coordination layer of the present invention;

[0053] Figure 2 This is a schematic diagram of the structural form of the edge interaction unit of the present invention;

[0054] Figure 3 This is a schematic diagram of the semantic event object data structure of the present invention;

[0055] Figure 4 This invention provides a multimodal semantic event interaction method for the central coordination layer.

[0056] Figure 5 This is the timing diagram for multimodal timing alignment and fusion in this invention;

[0057] Figure 6 This is a schematic diagram of the decision-making process of the scheduling module of the present invention;

[0058] Figure 7 This is a schematic diagram of the developer interface of the present invention;

[0059] Figure 8 This is a schematic diagram of the personalized execution service module of the present invention;

[0060] Figure 9 A structural diagram of the user insight service module;

[0061] Figure 10 A schematic diagram of the context management service module;

[0062] Figure 11 This is a schematic diagram of the structure of the non-transitory computer-readable medium of the present invention. Detailed Implementation

[0063] The invention will now be described in further detail with reference to the accompanying drawings. Please refer to... Figure 1-11 As shown, the multimodal human-computer interaction platform system of this embodiment includes an edge interaction unit 200, a central coordination layer 110, a data and memory storage 160, and a developer / application 300. The central coordination layer 110 internally includes a semantic event conversion module 120, a personality proxy module 130, a scheduling module 140, a developer interface module 150, a privacy and compliance module 170, and an application programming interface (API) / service module 180, and uses the semantic event object 400 as the core data unit for cross-module transmission.

[0064] I. Components and Functions:

[0065] Edge interaction unit 200: includes a multimodal acquisition interface 210, a local preprocessing module 220, and a communication module 230, used to acquire voice, image, and / or environmental sensing data and perform noise reduction, wake-up detection, and speech recognition related processing locally. It should be understood that... Figure 1 This is a module-level schematic diagram; the internal processing flow of the local preprocessing module 220 is not shown in detail in the diagram. In some embodiments, local preprocessing only performs Voice Activity Detection (VAD), endpoint detection, and voice feature extraction and uploading of streaming features; in another embodiment, offline command word recognition is supported; complete end-to-end Automatic Speech Recognition (ASR) can be performed by a central coordination layer or a cloud model to achieve a balance between resource consumption and privacy protection.

[0066] Semantic event conversion module 120: converts preprocessed data from edge interaction unit 200 into semantic event object 400, wherein the semantic event object includes at least sentiment index, context summary and privacy level label; optionally, it may also include tenant / application policy identifier for multi-tenant isolation and access control.

[0067] Personality Proxy Module 130: Receives semantic event objects 400 and generates personalized interactive response strategies by combining long-term memory, user behavior preferences and personality parameters; where long-term memory is provided by data and memory storage 160.

[0068] Scheduling module 140: Receives the strategy points from semantic event object 400 and personality agent module 130, calculates a comprehensive routing score based on emotional urgency and task complexity, and selects a local lightweight model, edge computing unit, or cloud-based large model to perform inference under privacy, network, and cost constraints; the comprehensive routing score can be generated by a pluggable scoring model, which can be optimized online based on rule calculation, machine learning, or reinforcement learning, and triggers degradation and rollback under resource unavailability or timeout conditions.

[0069] Developer Interface Module 150: Encapsulates the anonymized and summarized content, personalized response strategies, and inference results of the semantic event object 400 into standardized interface capabilities for external developers / applications 300 to call.

[0070] Data and memory storage 160: Stores long-term memory archives, interaction summaries, and user profiles, which may include user interaction preferences in the form of vector embeddings, structured archives (such as attributes, frequently used commands, and contextual summaries), and time-series-based behavioral pattern indexes to support retrieval and reasoning; and records routing scores, resource selections, and performance metrics generated by the scheduling module 140 for use by the personality agent module 130 and the scheduling module 140.

[0071] Privacy and Compliance Module 170: Apply control policies to Semantic Event Conversion Module 120, Personality Proxy Module 130, Scheduling Module 140, Developer Interface Module 150, and Application Programming Interface (API) / Service Module 180: When Semantic Event Conversion Module 120 generates Semantic Event Object 400, a privacy tag is attached; before resource selection in Scheduling Module 140, computational paths that do not meet the requirements are eliminated based on the tag; before external output by Developer Interface Module 150 and Application Programming Interface (API) / Service Module 180, attribute trimming and permission verification are performed.

[0072] Application Programming Interface (API) / Service Module 180: As a unified external access point, it provides service routing, interface management, multi-tenant isolation and access control, enabling developers / applications 300 to access platform capabilities in the form of a unified application programming interface (API) / service.

[0073] Developer / Application 300: Based on its own business logic, it calls the platform's capabilities to obtain standardized results or control instructions encapsulated and output by the developer interface module 150 through the application programming interface (API) / service module 180.

[0074] Semantic event object 400: generated by semantic event conversion module 120 and passed to personality agent module 130 and scheduling module 140, serving as the core data unit for cross-module processing; when updating, a new object is generated or an annotation is added to avoid the same object going back and forth between modules.

[0075] II. Master Data / Response Stream:

[0076] During operation, multimodal input is sent from the edge interaction unit 200 to the semantic event conversion module 120; the semantic event conversion module 120 outputs a semantic event object 400 and passes it to the personality agent module 130 and the scheduling module 140; the personality agent module 130 combines the long-term memory of the data and memory storage 160 to generate strategy points; the scheduling module 140 calculates the routing score and selects resources to generate a response; the result is encapsulated by the developer interface module 150 and provided to the developer / application 300 through the application programming interface (API) / service module 180. The personality agent module 130 and the data and memory storage 160 perform long-term memory read and write operations, and the scheduling module 140 writes decision and performance records to the data and memory storage 160.

[0077] III. Control Flow:

[0078] The privacy and compliance module 170 applies policy control during the processing of the semantic event conversion module 120, scheduling module 140, developer interface module 150, and application programming interface (API) / service module 180: at the semantic event conversion module 120, it associates the semantic event object 400 with a privacy tag; at the scheduling module 140, it uses the tag as a resource selection constraint; and at the developer interface module 150 and application programming interface (API) / service module 180, it performs granular control of output content and permission verification based on the tag and tenant / application policy.

[0079] IV. Auditing and Traceability:

[0080] Optionally, in some embodiments, the system may further include a monitoring and auditing module 190, used to structurally record the routing scores and resource selection of the scheduling module 140, and the interface calls and external outputs of the developer interface module 150 and the application programming interface (API) / service module 180, and associate them with the corresponding semantic event objects 400. The monitoring and auditing module 190 may also record key read and write events to the data and memory storage 160 to support audit traceability, effect evaluation, and reproduction.

[0081] like Figure 2 As shown, the edge interaction unit 200 provided in one embodiment of the present invention can be implemented as a module, an embedded platform, or a mobile application according to different deployment requirements, and its structure includes the following parts:

[0082] I. Structural Composition:

[0083] Multimodal acquisition interface 210

[0084] Used to receive multimodal input data from voice input, image sensors, and environmental sensors;

[0085] In modular form, this interface is typically implemented using standard hardware pins or bus protocols such as I²S, MIPI, I²C, and SPI.

[0086] In the form of mobile applications, data can be acquired through the microphone and camera application programming interfaces (APIs) provided by the operating system;

[0087] It is configured to provide a device-independent unified abstraction layer to adapt and standardize the output of sensors from different manufacturers / models. The output includes timestamp alignment and sampling rate specifications, and can publish capability descriptors to support plug-and-play and dynamic expansion.

[0088] The local preprocessing module 220 is used to perform basic data cleaning and compression, including speech endpoint detection, noise suppression, feature extraction and time alignment; when computing power allows, it can perform partial speech command word recognition or lightweight emotion feature extraction; the design boundaries of this module can be flexibly configured: for example, in a resource-constrained module form, it can only perform low-power feature extraction; while in an embedded platform form, it can support partial offline automatic speech recognition (ASR) or multimodal fusion processing;

[0089] Optionally, local alerts can be supported, such as threshold alarms and keyword triggers for abnormal sound / images, to reduce uplink and wake-up latency. During preprocessing, localized desensitization processing is performed on data containing privacy-sensitive information, such as face images and raw audio, including but not limited to face blurring / masking, voice fingerprint hashing, speaker anonymization and entity noun deletion, and only feature data or desensitized structured data is uploaded.

[0090] The communication module 230 is used to transmit preprocessed structured data or feature streams to the central coordination layer 110 and receive personalized response strategies. It can be implemented through wireless protocols such as Wi-Fi, Bluetooth, ZigBee, Z-Wave, cellular networks, or wired protocols such as Ethernet and USB. It adaptively adjusts the uplink strategy based on the current network conditions, bandwidth, latency, packet loss rate, and service latency budget: when the network is good, it can transmit high-fidelity features and rich context summaries; when the network is limited, it switches to high-compression features / critical instructions or event-level summaries, and can enable caching and delayed uploading (store-and-forward), batch uploading, and priority queuing to ensure the reliability of critical interactions. The communication process can adopt end-to-end encryption and retry / rollback mechanisms, and supports asynchronous transmission and flow control through gRPC / HTTP, MQTT, WebSocket, or equivalent protocols.

[0091] II. Differences and Application Scenarios of the Three Forms

[0092] Module Form: Embedded in the form of hardware modules into lamps, cameras, toys or other terminal devices, suitable for standardized integration by OEM / ODM;

[0093] Embedded Platform: Based on a SoC platform with computing and voice processing capabilities, it can support more localized AI functions and is suitable for smart speakers, security terminals or educational companion devices;

[0094] Mobile App Form: This form acquires microphone and camera data from iOS / Android applications and combines it with cloud or edge services, making it suitable for rapid deployment or providing value-added application services.

[0095] III. Collaborative Operation Theory

[0096] The edge interaction unit 200 in this embodiment, through the flexible deployment of the above three forms, ensures that it can effectively complete multimodal data collection and preliminary processing under different computing power, privacy and network conditions, and work in coordination with the central coordination layer 110.

[0097] like Figure 3 As shown, the Semantic Event Object 400 in this embodiment is a structured data unit transmitted between the edge interaction unit 200 and the central coordination layer 110. This object is generated by local speech processing or multimodal preprocessing, following the principles of "privacy first" and "minimum visibility," and does not contain raw audio / images or directly identifiable plaintext identifiers. The Semantic Event Object 400 includes three parts: metadata 410, semantic core 420, and control and compliance 430.

[0098] Metadata 410: Supports event tracking, timing, device ownership, and integrity verification, including at least:

[0099] Event identifier (410a) event_id: UUID / string, globally unique, used for auditing and logging;

[0100] Timestamp (410b): in ISO 8601 format, may include time zone or monotonic clock offset;

[0101] Source device identifier (410c) source_device_id: A one-way hash value that uniquely identifies the device; it cannot be reversed externally.

[0102] Modality (410d): Indicates the modality of the input data, such as audio, video, sensor, multimodal, etc., which is used for interface selection and routing of subsequent modules;

[0103] Integrity signature (410e) integrity_sig: Optional, a digest or information authentication code calculated from a normalized sequence of events that does not contain sensitive fields, used to detect transmission tampering and replay.

[0104] Optionally, the metadata 410 may carry extended key-value pairs such as delay budget, cost limit, scheduling hints, etc., but such extensions should not violate the above-mentioned desensitization and integrity requirements.

[0105] Semantic Core 420, used to characterize the task semantics, emotional dimension, and contextual relevance of an event, includes at least:

[0106] Semantic intent (420a): A string or JSON object, such as “request:play_music” or “query:weather”, which may contain the target object, slot and priority.

[0107] Emotional context (420b): This is the result of multimodal emotion fusion, which includes at least valence (ranging from -1 to 1), arousal (ranging from 0 to 1), and confidence, and may also include discrete emotion labels and time decay coefficients.

[0108] Dialogue context summary (420c) dialog_context: Summary text or keywords and citation identifiers of the history interactions, without original audio / images or complete transcripts.

[0109] Control and compliance (430) are used to impose hard constraints on the externally visible fields of an event, response granularity, data retention, and cross-border strategies, including at least:

[0110] Privacy tag set (430a) privacy_tags: consists of one or more tags, each tag may contain type, level and scope, which are used to indicate the degree of desensitization and the boundaries of external disclosure;

[0111] Policy ID (430b) policy_id: The policy version for the tenant or application, used for auditing and tracing multi-tenant isolation, data residency, retention period, and access control;

[0112] Output granularity control (430c): This indicates the level of fineness of the external interface response, such as full, masked, summary_only, etc. When the privacy level reaches a predetermined threshold, the system automatically downgrades to masked or summary_only, and can apply bucketing or differential privacy perturbation.

[0113] In a preferred embodiment, the semantic event object 400 is submitted to the central semantic event coordination layer via wireless or wired communication, and operates according to the following relationships and execution flow:

[0114] External input: The semantic core 420 serves as the input to the personality agent module 130 and the scheduling module 140. The former generates personalized response styles and content based on it, while the latter determines model routing, resource allocation, and device coordination based on it.

[0115] Compliance Pre-processing: Privacy and Compliance Module 170 reads Control and Compliance 430, desensitizes and trims the fields contained in 400, and applies constraints to Scheduling Module 140 and subsequent external interfaces;

[0116] Memory writing: Under the premise of satisfying the strategy shown in 430, the scheduling module 140 may write the processed summary content into the long-term memory module 160; fields marked as restricted fields may not be written, or may only be stored as reference identifiers such as history_ref;

[0117] External output: When an external interface requests a retrieval or responds, the system only returns a non-sensitive subset of the semantic core (420) and meta-information (410) that has been de-identified, anonymized and summarized, and does not contain the original audio / image or plaintext identifiers that can directly identify individuals; the output granularity is controlled by 430c.

[0118] For data security and replaceability implementation, to enhance security, integrity signature 410e can be implemented based on a pre-shared key information authentication code algorithm or a public-private key signing mechanism; the hash map table of the source device identifier 410c is stored only in a trusted, isolated domain and is subject to audited access. In other embodiments, semantic core 420 can be generated by a fusion model of audio, image, and environmental sensing data; the tagging system of control and compliance 430 can be adapted to the data protection specifications of different jurisdictions, but all should ensure that external responses do not contain the original multimedia data.

[0119] like Figure 4 As shown, this method generates, schedules, and outputs semantic events while adhering to privacy and compliance requirements. The system consists of an edge interaction unit 200, a semantic event conversion module 120, a semantic event object 400, a privacy and compliance module 170, a personality proxy module 130, a scheduling module 140, data and memory storage 160, and optional developer interface module 150 and application programming interface (API) / service module 180, providing external services to developers / applications 300. In the diagram, solid lines represent data / response flows, and dashed lines represent control / policy and audit channels.

[0120] S1 Data Acquisition and Preprocessing

[0121] The edge interaction unit 200 acquires multimodal input, completes necessary preprocessing and secure transmission, and sends the purified input to the semantic event conversion module 120.

[0122] S2 Semantic Event Generation

[0123] The semantic event conversion module 120 generates a semantic event object 400. The object contains metadata and semantic core, and carries control and compliance information for subsequent policy determination; it does not contain original audio / video or directly identifiable plaintext identifiers, and the device / user identifier adopts one-way hashing or anonymization.

[0124] S3 Personalized Strategy Generation

[0125] The privacy and compliance module 170 reads the control and compliance information carried by the semantic event object 400, generates policy tokens such as masking, visibility range, retention period, and geographic residency, and sends them via a dotted line to the scheduling module 140, developer interface module 150, application programming interface (API) / service module 180, and data and memory storage 160, without transmitting the data ontology. The personality proxy module 130 obtains features from the semantics and context of the semantic event object 400 to form personalized parameters for scheduling use.

[0126] S4 Scheduling and Storage

[0127] The scheduling module 140 receives the semantic event object 400 and personalized parameters, and completes model / resource routing. Under the constraints of the issued policy, it only writes the summary or reference identifier to the data and memory storage 160, and implements access control according to the preset retention / regional policy. The semantic event object 400 does not directly establish a write channel with the data and memory storage 160.

[0128] S5 encapsulation and external calls

[0129] The scheduling result is encapsulated by the developer interface module 150, which implements access control authentication / authorization, scope, frequency limits, and auditing, and then outputs it to the developer / application 300 via the application programming interface (API) / service module 180. A privacy tag is attached when the semantic event object is generated at 120. Data writing is only implemented by 140→160; 200 does not directly supply data to 130; the input to 130 comes from 400; 170 is the policy executor and issuer and does not output the data ontology. The above steps can be implemented in a single-machine or distributed environment and do not constitute a limitation on the scope of protection of this invention.

[0130] like Figure 5 As shown, in one embodiment, the semantic event conversion module 120 performs preprocessing and fusion on multimodal inputs from microphone 200a, camera 200b, and sensor 200c to generate a dialogue context summary field 420c in the semantic core 420 of the semantic event object 400. The processing includes any one or more of the following steps, which can be combined as needed, and the order is not limited:

[0131] Feature extraction: Generate de-identifiable feature representations for audio, image and environmental data, such as acoustic embedding, keywords, key image regions / frame-level embedding, IMU / environmental statistics, etc., without outputting the original audio or original image frames.

[0132] Time alignment: Within a predetermined time window Δt, a unified time reference is established for different modes. If necessary, resampling and / or interpolation are performed to eliminate sampling rate differences and achieve cross-modal time synchronization.

[0133] Confidence estimation: Calculate confidence scores based on modal quality metrics, such as audio signal-to-noise ratio, endpoint detection confidence, image sharpness / face quality, sensor stability, etc., for subsequent weighting.

[0134] Cross-modal fusion: Based on the confidence level and task type, multimodal features are weighted, attention-based fusion, and / or voting-based fusion to obtain a unified contextual representation.

[0135] Deduplication and consistency parsing (optional): Removes overlapping segments and conflicting signals, and outputs consistent context results.

[0136] Desensitization and summary output: The above results are written into or referenced in the semantic core 420 of the semantic event object 400, especially the dialogue context summary field 420c. The field may include summary text (e.g., TL;DR form), keyword set and / or history_ref reference ID, as input to the subsequent personality agent module 130 and scheduling module 140.

[0137] To meet privacy and compliance requirements, uploads and storage only include de-identified summaries or features; source identifiers such as source_device_id are represented in one-way hash or anonymized form, and the return of original audio / video and plaintext identifiers that can directly identify individuals is prohibited. The above processing can be performed by the edge interaction unit 200 and / or the semantic event conversion module 120; when executed at the edge, only the minimum necessary summaries / features are sent to the central coordination layer 110. The window Δt, visible fields, and upload granularity can be configured according to system policies and do not constitute a necessary limitation of this invention.

[0138] like Figure 6 The diagram illustrates the detailed S4 scheduling and storage process. The metadata 410 and semantic core 420 in the semantic event object 400 serve as inputs and can be combined with the personalized parameters of the personality agent module 130. The privacy and compliance module 170 generates a policy token based on the privacy and compliance information carried by the object and sends it to the scheduling module 140 as a dashed control signal.

[0139] S4-1 Privacy and Compliance Pre-Filtering Hard Constraints

[0140] The scheduling module 140 first eliminates candidate computing resources that are incompatible with the regional residency, retention period, visible field / output granularity or access policy according to the policy issued by the privacy and compliance module 170, and does not transmit the data body.

[0141] S4-2 Emotional Urgency Assessment

[0142] Based on the emotional context 420b in the semantic core 420 and combined with the timestamp of the meta-information 410 for time decay, a normalized urgency index is obtained.

[0143] S4-3 Task Complexity Assessment

[0144] By combining the semantic intent 420a and dialogue context summary 420c in the semantic core 420, and by combining the modality / latency budget / resource hints in the meta-information 410, the parameter size, video memory / memory usage and inference overhead can be estimated to obtain a normalized complexity index; the personality agent module 130 can be used as a bias or weight in the evaluation.

[0145] S4-4 Composite Routing Score

[0146] The system combines urgency and complexity, along with optional personality / modal matching signals, in a configurable or learning-based manner to form a decision score, which is not limited to a linear form.

[0147] S4-5 Multi-level threshold comparison maps this score

[0148] The decision scores are mapped to resource layers 600: local lightweight model 600a, edge computing / caching 600b, and cloud-based large model 600c, outputting the selected layer identifier (tier id). Online adaptive latency / load / cost are fed back to S4-4 / 5 / 6 as runtime monitoring signals via dashed lines for dynamically adjusting weights, thresholds, and fallback strategies.

[0149] S4-6 Layer Model / Engine Selection and Logs

[0150] The scheduling module 140 determines the specific model / inference engine within the selected resource layer based on task characteristics and the capability matrix, and outputs it to the developer interface module 150 for encapsulation and external provision via the application programming interface (API) / service module 180. Simultaneously, it associates decision logs and metrics such as routing scores, threshold comparison results, selected resource layer and policy / model version identifiers, latency and cost measurements with the semantic event object 400 and writes them to the data and memory storage 160 and the monitoring and auditing module 190. To meet privacy requirements, only summaries / citations such as TL;DR, keywords, or history_ref are allowed to be written; storing raw audio / video or plaintext identifiers that can directly identify individuals is prohibited. When a candidate resource is unavailable, times out, or conflicts with a policy, a degradation, fallback, and retry strategy is triggered.

[0151] like Figure 7 As shown, an embodiment of the present invention provides a developer interface architecture including a developer interface module 150, an application programming interface (API) / service module 180, and an external developer / application 300. The developer interface module 150 encapsulates the semantic event object 400, memory summary, and personalization strategy generated internally by the platform into a standardized interface, and provides it externally in a unified manner through the application programming interface (API) / service module 180.

[0152] In a preferred embodiment, the Application Programming Interface (API) / Service Module 180 serves as a security boundary and gateway control point, responsible for authenticating, authorizing, and rate-limiting external requests, and maintaining access isolation in a multi-tenant environment. All external calls must first pass through this gateway API / Service Module 180 to avoid bypassing the control layer to directly access internal resources; the returned results are constrained by policies issued by the Privacy and Compliance Module 170, and only de-identified, anonymized, and digested attributes are output.

[0153] Application Programming Interface (API) interfaces include at least five categories:

[0154] The Semantic Events API is used to report or query semantic event objects and related structured fields.

[0155] The Memory Management API is used for synchronizing, retrieving, and deleting memories across modules.

[0156] The Execution Policy API is used to issue or query personalized policies, including tone, style, and interaction rhythm.

[0157] The User Insights API is used to obtain anonymized and aggregated user behavior statistics and growth trends.

[0158] The Context Management API is used for context configuration and scheduling of contextualized tasks.

[0159] Optionally, the Application Programming Interface (API) / Service Module 180 may also provide a System API, including health checks, system metrics, and version queries, to support operation and maintenance needs. In another optional embodiment, the system may further include an application development platform (not shown in the figure), coupled to the developer interface module 150 and the API / Service Module 180, to provide developers with a low-code or graphical development environment, enabling them to define interaction logic and business rules based on the aforementioned APIs, and deploy the encapsulated application to edge interaction units 200 of different forms.

[0160] like Figure 8 As shown, a personalized execution service module 700 provided in one embodiment of the present invention is used to uniformly encapsulate and output personalized interaction logic and execution constraints in a service-oriented manner after the semantic event object 400 is parsed.

[0161] In a preferred embodiment, the semantic intent and context summary 420 carried by the semantic event object 400, combined with the personalized parameters generated by the personality agent module 130 and the historical memory items provided by the data and memory storage 160, are input into the personalized execution service module 700.

[0162] Response strategy generation unit 710: used to generate a personalized response strategy by integrating the above inputs. The strategy may include tone, style, output carrier selection and interaction rhythm.

[0163] Execution constraint unit 720: Used to constrain and tailor the response policy based on privacy level, compliance rules and multi-tenant access policy to ensure that the output meets security and compliance requirements.

[0164] The Personalized Execution Service Module 700 supports multiple output formats, including but not limited to:

[0165] Text-to-Speech (TTS) output: converts response strategies into spoken form for playback;

[0166] Device control: Generate control commands for interaction with IoT devices or smart terminals;

[0167] Scene linkage: Triggering coordinated responses from lights, alarms, images, or other external systems in cross-modal scenarios.

[0168] In addition, the personalized execution service module 700 supports version management, self-adaptation and logging, which is used to dynamically adjust strategy rules in different application scenarios and save key indicators and call logs of the execution process for auditing, reproduction and subsequent optimization.

[0169] In other embodiments, the personalized execution service module 700 can be implemented as a service component that works in conjunction with the developer interface module 150 and the application programming interface (API) / service module 180, allowing developers to call personalized response strategies through standardized interfaces, thereby integrating them into third-party applications or edge interaction units 200.

[0170] like Figure 9 As shown, an embodiment of the present invention provides a user insight service module 800, whose input source is a semantic event object 400, and which forms a user insight service that can be called by developers or applications through a multi-layer processing unit.

[0171] Preprocessing unit 801: Semantic event objects 400 are first deduplicated and standardized by the preprocessing unit 801 to eliminate duplicate fragments across devices, align timestamps, and unify feature formats, thereby constructing a consistent long-term interaction archive. This step corresponds to claim 8a, ensuring multimodal consistency and sustainable use of data sources.

[0172] Indicator generation unit 802: The pre-processed data enters the indicator generation unit 802 to calculate multi-dimensional longitudinal indicators, including sentiment trends, interaction frequency and time period distribution, task completion, compliance / churn risk, and scene preference; the indicators can be dynamically updated by combining time decay and smoothing strategies, corresponding to the description of longitudinal behavioral indicators and trend generation in claim 8.

[0173] Desensitized Output Unit 803: The generated indicators and profiles are processed by the desensitized output unit 803. Based on the privacy level label carried by the semantic event object and the tenant / application policy, the output results are provided in a de-identified, de-identified, and summarized manner, prohibiting the inclusion of original audio, images, or data that can directly identify individuals. Optionally, unit 803 may apply differential privacy noise or binning processing to further enhance privacy protection; the above capabilities correspond to the description in claim 8 regarding the output of de-identified profiles based on privacy level labels.

[0174] Versioning and traceability: The de-identified output unit 803 is also responsible for versioning and traceability management, recording the data range, model or rule version and scoring basis, and generating auditable logs, corresponding to the description of auditable traceability and version management in claim 8.

[0175] External Provision and Storage: The anonymized output unit 803 transmits the processed output to the developer interface module 150 and the application programming interface (API) / service module 180 for external developers or applications 300 to call; simultaneously, the user insight service module 800 stores the anonymized and summarized indicator data into the data and memory storage 160 for continuous learning and strategy optimization. The query and subscription capabilities support filtering and aggregation by time window, scenario domain, and indicator dimension, corresponding to the description of the query capabilities for filtering, aggregation, and subscription by time window, scenario domain, or indicator dimension in claim 8.

[0176] like Figure 10 As shown, an embodiment of the present invention provides a context management service module 820 comprising multiple functional units for realizing the fusion of multimodal inputs and the generation and output of a unified context object 500. The semantic event object 400, after being generated by the semantic event conversion module, enters the context management service module 820 as input. The module internally includes:

[0177] Time alignment and source identification unit 820a: performs time alignment on asynchronously input voice, image and / or sensor data, and adds a source identification to each data stream.

[0178] Feature extraction and confidence estimation unit 820b: Extracts semantic features from aligned multimodal data and calculates the corresponding confidence index.

[0179] Fusion Strategy Unit 820c: Fuses features from different modalities based on configurable weighting rules or learning algorithms.

[0180] Unified Context Generation Unit 820d: Generates a unified context object 500 from the fused results. This object contains at least emotional state, spatial / temporal context, environmental conditions, and task priority.

[0181] Privacy and De-identification Control Unit 820e: Performs de-identification and minimization processing before output to ensure that the Uniform Context Object does not contain raw audio or video or data that can directly identify an individual.

[0182] Query / Replay / Filtering and Consistency Unit 820f: Supports querying and aggregating unified context objects by time window, spatial domain or scene domain, and performs deduplication and consistency resolution.

[0183] Multi-unit collaboration and continuity control unit 820g: When multiple edge interaction units report simultaneously, it selects the dominant context and executes buffering, degradation and fallback strategies to maintain context continuity when there are network fluctuations or device unavailability.

[0184] The generated unified context object 500 is then provided to the scheduling module 140, the user insight service module 800, and the personalized execution service module 700 for subsequent task routing, user profile generation, and personalized response.

[0185] like Figure 11 As shown, an embodiment of the present invention provides a non-transitory computer-readable medium 900 on which computer instructions are stored; when executed by a processor or a system-on-a-chip (SoC) 910, the instructions cause the system to perform the following functions:

[0186] Generate semantic event objects, each of which must contain at least a sentiment index, a contextual summary, and a privacy level label.

[0187] Output a personalized response strategy, combining user long-term memory with a personality parameter set generation strategy;

[0188] Routing scores are calculated based on emotional urgency and task complexity, and resource selection is performed using multi-level thresholds.

[0189] Record and store information such as routing scores, selected resource categories, latency / cost metrics for logging and auditing purposes;

[0190] The above processing results are provided to the outside world in a standardized form through application programming interfaces (APIs) and service interfaces (Service Interfaces).

[0191] In a preferred embodiment, the processor / SoC (910) can be implemented in different types of carriers, including but not limited to servers, edge devices or terminals, and is not limited to a single implementation method, thereby supporting the portability and scalability of the platform in a variety of deployment environments.

[0192] This invention is not limited to the embodiments described above. Anyone should understand that structural changes made under the guidance of this invention, and any technical solutions that are the same as or similar to this invention, fall within the protection scope of this invention. Technical aspects, shapes, and structures not described in detail in this invention are all publicly known technologies.

Claims

1. A multimodal human-computer interaction platform system based on a central coordination layer, characterized in that, include: The module includes a semantic event conversion module, a personality proxy module, a scheduling module, and a developer interface module. The semantic event conversion module is used to receive multimodal input data from voice, image and sensor, and convert the input into structured semantic event objects. The semantic event object is a unified transmission unit across modules and devices. The semantic event object carries identification information for access control and compliance governance. The identification information includes at least privacy level labels and / or tenant / application policy identifiers. The personality proxy module is used to combine semantic event objects with the user's historical interaction records and behavioral preferences to generate personalized interactive responses; The scheduling module is used to select an execution path among local, edge, or cloud resources based on the context and constraint information contained in the semantic event object and output the corresponding results. The developer interface module is used to provide semantic event objects and personalized interactive responses to the outside world in the form of standardized application programming interfaces (APIs) or service interfaces. The output is only exposed in the form of structured objects that have been de-identified and digested. The service interface supports multiple types of function calls, and the output granularity and access permissions are constrained by privacy level labels and tenant / application policy identifiers. This allows developers to focus only on application business logic and call platform capabilities to build scenario-based applications. Furthermore, any response from the platform to the outside world is derived from a non-sensitive subset view of the semantic event object.

2. The multimodal human-computer interaction platform system based on a central coordination layer according to claim 1, characterized in that, It also includes one or more edge interaction units, a multimodal acquisition interface, a local preprocessing module, and a communication module, wherein: The edge interaction unit is used to collect multimodal input data and perform preliminary processing; the edge interaction unit communicates with the central coordination layer through the network to transmit semantic event objects and / or receive personalized response strategies; The multimodal acquisition interface is used to acquire voice, image and / or environmental sensing data; The local preprocessing module is used to perform at least one local processing operation on the multimodal input data. The local processing operation includes: wake-up detection, automatic speech recognition, multimodal feature extraction, time alignment, confidence estimation, cross-modal fusion, and privacy desensitization processing. The processing results are reported to the central coordination layer in the form of standardized features or directly generated semantic event objects. The communication module is used to interact with the central coordination layer for data and control commands, and to perform irreversible desensitization on personally identifiable multimedia fields locally, and to prohibit the uploading of original identifiable audio / image byte streams.

3. The multimodal human-computer interaction platform system based on a central coordination layer according to claim 2, characterized in that, The developer interface module further includes multiple service interfaces; wherein, the service interfaces only output objects that have been de-identified and digested, and do not expose the original audio / image byte streams, and are subject to access control by privacy level tags and tenant / application policy identifiers. The service interfaces include at least a semantic event interface, a memory management interface, an execution policy interface, a user insight interface, a context management interface, and a model adaptation interface.

4. The multimodal human-computer interaction platform system based on a central coordination layer according to claim 3, characterized in that, The semantic event conversion module is used to execute the semantic event generation step of claim 4 to generate a structured semantic event object. The personality proxy module is used to execute the personalization strategy generation step of claim 4 to generate a personalized interaction response strategy. The developer interface module is used to execute the service interface encapsulation step and the scenario-based application invocation step of claim 4 to encapsulate the service interface and make it available for application calls.

5. A multimodal human-computer interaction platform system based on a central coordination layer according to claim 4, characterized in that, The central coordination layer includes a scheduling module, used to calculate a synthetic routing score based on emotional urgency assessment and task complexity assessment, and select computing resources accordingly to generate and output personalized interactive responses, specifically including: Emotional urgency assessment is calculated based on the emotion recognition results in semantic event objects; Task complexity assessment is based on attribute keyword weights, context length, and / or required model resource estimation; The synthetic route score can be obtained by weighting emotional urgency and task complexity with configurable weights, not limited to a linear form, and compared with multi-level thresholds to dynamically select computing resources. The scheduling module further performs online adaptive updates to the weights and thresholds based on network latency, device load, and cost budget. The scheduling and resource selection process follows privacy and compliance information, which is attached to the semantic event object and includes at least a privacy level label and a tenant / application policy identifier; The scheduling module associates and stores decision logs and metrics with corresponding semantic event objects, and records the routing scores, resource selection results, fallback and / or degradation triggering situations, network latency and cost measurements in a structured manner. It saves the decision basis only in anonymized and summarized form without retaining the original identifiable audio / image byte streams, for use in compliant delivery, accountability and subsequent optimization. The scheduling module supports parallel pre-routing and pre-fetching to reduce cold start latency, and performs resource selection and result merging for different subtasks in multi-path composite response scenarios.

6. A multimodal human-computer interaction platform system based on a central coordination layer according to claim 5, characterized in that, The central coordination layer includes a personalized execution service module, which provides the personalized interactive response decision-making process as a service to external parties, specifically including: The personalized execution service module receives a structured semantic event object provided by the semantic event interface, and combines it with the user's long-term memory profile and personality parameter set to generate a personalized response strategy. The strategy includes at least tone / style parameters, output carrier selection, and interaction rhythm. Personalized response strategies and corresponding execution constraints (including security / compliance rules and privacy classifications) are encapsulated together as callable execution strategy interfaces for application triggering, and the strategies can be refined into speech synthesis output, device control commands or scenario-based linkage operations. The personalized execution service module supports policy self-adaptation and version management in different application scenarios, and records the basis for policy generation, scoring indicators and call logs to facilitate auditing, reproduction and continuous optimization.

7. A multimodal human-computer interaction platform system based on a central coordination layer according to claim 6, characterized in that, The central coordination layer includes a user insight service module, which is used to provide user profiles / growth representations generated based on long-term interaction data as a service, specifically including: The user insight service module constructs a long-term memory archive based on structured semantic event objects provided by the semantic event interface. The archive includes at least an interaction event timeline, a weighted record of sentiment intensity and confidence, behavioral preference tags, and a context summary, and supports data deduplication, standardization, and alignment from multiple edge interaction units. The user insight service module calculates longitudinal metrics, including but not limited to sentiment trends, interaction frequency / time period distribution, task completion rate, compliance / churn risk score, and scenario preference. These metrics can be updated by combining time decay and smoothing strategies. Based on privacy level labels and access policies, the user insight service module provides user profiles to the public in an anonymized and summarized form, prohibiting the output of original audio / images and data that can directly identify individuals. It can optionally apply differential privacy noise or binning processing to the summary results. The user insight service module provides versioning and traceability capabilities, records the data range, model / rule version and scoring criteria on which the profile is generated, and outputs auditable generation logs; The User Insights service module provides query and subscription capabilities through the User Insights interface, supporting filtering and aggregation by time window, scenario domain, and metric dimension, which can be used by developers or content providers for personalized strategy configuration and effect evaluation.

8. A multimodal human-computer interaction platform system based on a central coordination layer according to claim 7, characterized in that, The central coordination layer includes a context management service module, which provides the method for fusing multimodal inputs and generating unified context objects as a service. Specifically, it includes: The module receives asynchronous data streams from one or more edge interaction units, including voice, image and / or environmental sensing data, and performs time alignment and source identification on each data stream; Feature extraction and confidence evaluation are performed on the aligned data stream, and a unified context object is generated by the unified context generation unit according to a configurable weighted decision / learning fusion strategy. The object includes at least emotional state, spatial / temporal context, environmental conditions and task priority. During the integration process, privacy level labels and access policies are followed to minimize processing and de-identify output, prohibiting the exposure of original audio / images and data that can directly identify individuals. The unified context object and semantic event interface are linked to provide query, subscription and replay capabilities. It supports filtering and aggregation based on time window, spatial region and scene domain, and performs deduplication / consistency parsing for duplicate or conflicting events. The context management service module supports scenario switching and collaboration: when multiple edge interaction units report events simultaneously, the dominant context is selected based on device proximity, sensor coverage, and event confidence. Buffering, degradation, and rollback strategies are executed when the network fluctuates or the device is unavailable to ensure context continuity.

9. A method for a multimodal human-computer interaction platform based on a central coordination layer, characterized in that, Includes the following steps Multimodal input acquisition steps: Speech, image and / or environmental sensing data are acquired by one or more edge interaction units, and preprocessing is performed to remove noise, perform time alignment and privacy desensitization; Semantic event generation steps: The preprocessed data is analyzed using the semantic event transformation module to generate a structured semantic event object. The semantic event object includes at least a weighted fusion calculation of the sentiment index, a context summary generated based on historical dialogue, and a privacy level label for access control. Personalized strategy generation steps: The personality agent module combines semantic event objects, user long-term memory archives and personality parameter sets to generate personalized interaction response strategies; Service interface encapsulation steps: The semantic event object and personalized response strategy are encapsulated into a service interface through the developer interface module. The service interface supports multiple function calls, including semantic event interface, memory management interface, execution strategy interface, user insight interface and context management interface. Scenario-based application invocation steps: The service interface is invoked by the application, allowing developers to focus only on writing business logic and quickly build scenario-based applications using the platform's capabilities.

10. A non-transitory computer-readable medium, characterized in that: It stores computer instructions that, when executed by a processor, cause the central coordination layer and / or one or more edge interaction units to perform the method of claim 4, specifically including: Generate semantic event objects, which include fused sentiment indicators, context summaries based on historical dialogues, and privacy level tags; A response strategy is generated based on the semantic event object, long-term memory, and personality parameters, and the output granularity is controlled according to the privacy level label. The routing score is calculated based on the urgency of the emotion and the complexity of the task, and resources are dynamically selected between local lightweight models, edge units or cloud models through multi-level thresholds. The routing scores, resource selections, latency and cost metrics are associated with and stored with the corresponding semantic event objects for auditing and optimization. The processing results are provided to the outside world as a service through semantic events, memory management, execution strategies, user insights, or context management interfaces.

Images