A lightweight knowledge graph and edge computing human-computer interaction system

Abstract

The application relates to the technical field of human-computer interaction, in particular to a lightweight knowledge graph and edge computing human-computer interaction system, which comprises a data acquisition module, a graph query module, a task scheduling module, an inference adaptation module and a closed-loop updating module; the data acquisition module acquires multi-modal interaction data streams such as voice recognition result texts, gesture trajectory coordinates and terminal posture angle change sequences; the graph query module searches for function service nodes and attribute constraints matched with interaction intents; the task scheduling module adjusts execution priorities in combination with terminal posture angles; the inference adaptation module generates an inference task queue adapted to a computing power environment; and the closed-loop updating module forms a closed-loop control. The system can accurately capture multi-modal interaction intents, improve the adaptability of inference tasks and edge computing power environments, enhance the real-time performance and continuity of human-computer interaction, and adapt to interaction requirements in complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to a lightweight knowledge graph and edge computing human-computer interaction system. Background Technology

[0002] In existing lightweight knowledge graph and human-computer interaction technologies, the processing of multimodal interaction data is mostly limited to single-modal parsing or simple multimodal information splicing. It has not yet achieved unified input processing of speech recognition results (text), gesture trajectory coordinates captured by visual sensors, and the posture angle change sequence of the terminal device. The task scheduling mechanisms at edge computing nodes also generally adopt fixed priorities or scheduling logic based solely on single computing power or network status, failing to incorporate the posture angle changes of the terminal device into the criteria for judging the interaction state.

[0003] In existing technical solutions, the query interface of lightweight knowledge graphs can only serve nodes based on the retrieval function of a single semantic parsing result. The lack of multimodal information leads to biases in capturing interaction intent, making it difficult to accurately match the actual interaction needs of users. At the same time, the generation of inference task queues on edge computing nodes lacks a dynamic adaptation mechanism, and cannot adjust task execution strategies based on changes in terminal device posture and real-time network bandwidth. This results in insufficient matching between the execution of inference tasks and the edge computing environment, thus affecting the real-time performance and accuracy of the human-computer interaction system.

[0004] Existing technologies have not provided effective solutions for the integrated processing of multimodal interactive data streams, the dynamic scheduling of edge computing nodes, and the generation of inference queues. They also lack a closed-loop state feedback update mechanism, which makes it difficult for lightweight knowledge graphs combined with edge computing to meet the interaction needs of complex scenarios in practical applications. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a lightweight human-computer interaction system based on knowledge graphs and edge computing.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a lightweight knowledge graph and edge computing human-computer interaction system, comprising: The data acquisition module acquires a multimodal interaction data stream in the target scene. The multimodal interaction data stream includes the text of the speech recognition result, the coordinates of the gesture trajectory captured by the visual sensor, and the posture angle change sequence of the terminal device. The graph query module inputs the multimodal interaction data stream to the query interface of the pre-built lightweight knowledge graph. The query interface of the lightweight knowledge graph retrieves the functional service nodes related to the current interaction intent from the entity nodes of the graph according to the semantic parsing results, and outputs the functional service nodes and their associated attribute constraints. The task scheduling module receives the functional service node and its associated attribute constraints from the task scheduler deployed on the edge computing node, determines the current interaction state by combining the attitude angle change sequence of the terminal device, and adjusts the execution priority of the functional service node on the edge computing node accordingly. The inference adaptation module loads the lightweight inference model corresponding to the functional service node on the edge computing node, compares the adjusted execution priority with the current network bandwidth usage, and generates an inference task queue adapted to the current computing power environment. The closed-loop update module updates the status feedback returned by the query interface of the lightweight knowledge graph according to the execution progress of the inference task queue, forming a closed-loop control human-computer interaction process.

[0007] As a further aspect of the present invention, the acquisition of the multimodal interaction data stream in the target scene includes: The system collects the user's original speech signal through a microphone array, performs noise reduction and endpoint detection on the original speech signal, extracts the effective speech segments, and converts them into the speech recognition result text. The system acquires a video frame sequence containing the user's hand using a depth camera, segments the hand region in the video frame sequence, calculates the coordinates of the gesture trajectory using optical flow, and filters out abnormal coordinate points caused by hand tremors. The terminal device's acceleration and angular velocity data in the spatial coordinate system are acquired by the inertial measurement unit, and the Kalman filter algorithm is used to fuse and solve the acceleration and angular velocity data to generate the attitude angle change sequence. The timestamp-aligned speech recognition result text, gesture trajectory coordinates, and posture angle change sequence are packaged and encapsulated into the multimodal interaction data stream.

[0008] As a further aspect of the present invention, the multimodal interaction data stream is input to the query interface of a pre-constructed lightweight knowledge graph. Based on the semantic parsing results, the query interface of the lightweight knowledge graph retrieves functional service nodes related to the current interaction intent from the entity nodes of the graph, and outputs the functional service nodes and their associated attribute constraints, including: The natural language processing engine is invoked to perform dependency parsing and semantic role labeling on the speech recognition result text, identify the operation verbs and target objects, and generate the semantic parsing result; The semantic parsing results are fused with the trajectory features of the gesture trajectory coordinates across modalities to form a composite interactive intent vector; The composite interaction intent vector is input into the query interface of the lightweight knowledge graph. The query interface traverses the entity nodes and relation edges in the graph and filters out candidate functional service nodes that match the composite interaction intent vector. The selected candidate functional service nodes are scored with confidence, and the functional service node with the highest confidence and its associated attribute constraints are selected as the query result output.

[0009] As a further aspect of the present invention, the step of receiving the functional service node and its associated attribute constraints from the task scheduler deployed on the edge computing node, determining the current interaction state by combining the attitude angle change sequence of the terminal device, and adjusting the execution priority of the functional service node on the edge computing node accordingly includes: Analyze the resource requirements and latency tolerance indicators contained in the attribute constraints; Analyze the rate of change and acceleration of the attitude angle change sequence. If the rate of change is higher than a preset threshold and the acceleration shows an upward trend, then the current interaction state is determined to be an urgent interaction state. For scenarios in the urgent interaction state, the execution priority of the functional service node is raised to the highest level, and it is allocated exclusive edge computing resources. For scenarios with non-urgent interaction, the execution priority of the functional service nodes is dynamically adjusted according to the latency tolerance index, so that they can queue up and wait for execution without blocking high-priority tasks.

[0010] As a further aspect of the present invention, the step of loading the lightweight inference model corresponding to the functional service node on the edge computing node, comparing the adjusted execution priority with the current network bandwidth usage, and generating an inference task queue adapted to the current computing power environment includes: Based on the type identifier of the functional service node, the corresponding lightweight inference model is retrieved from the local model library. The lightweight inference model is a neural network model that has undergone pruning and quantization processing. Periodically monitor the network connection status between the edge computing node and the external cloud to obtain the current network bandwidth usage; Establish a mapping table between execution priority and network bandwidth usage. If the execution priority is high and the network bandwidth usage is below the lower limit threshold, then the remote data download task required during the inference process will be merged into the inference task queue. Based on task dependencies and timestamp order, all pending inference tasks are sorted to generate the inference task queue adapted to the current computing power environment.

[0011] As a further aspect of the present invention, the status feedback returned by the query interface of the lightweight knowledge graph is updated according to the execution progress of the inference task queue, forming a closed-loop control human-computer interaction process, including: When each task in the inference task queue is started, a signaling message indicating that the task has started execution is sent to the query interface of the lightweight knowledge graph. The running status and completion percentage of each task in the inference task queue are monitored in real time, and the latest running status is written into the status attribute field of the corresponding functional service node in the lightweight knowledge graph. After all tasks in the inference task queue have been completed, the final inference result data is aggregated and encapsulated into a status feedback data packet, which is then sent to the query interface of the lightweight knowledge graph. After receiving the status feedback data packet, the query interface of the lightweight knowledge graph triggers a refresh command for the human-computer interaction interface, completing a closed-loop interaction.

[0012] As a further aspect of the present invention, the step of fusing and calculating the acceleration and angular velocity data using a Kalman filter algorithm to generate the attitude angle change sequence includes: Initialize the Kalman filter model, where the state vector contains the attitude angle of the terminal device and the gyroscope zero bias, and the measurement vector contains the triaxial acceleration and angular velocity data collected by the inertial measurement unit; The acceleration and angular velocity data are acquired by the accelerometer and gyroscope of the inertial measurement unit, respectively, and then input into the Kalman filter model as the system's observations. In the prediction step of the Kalman filter, based on the system state equation, the predicted value of the attitude angle at the current moment is recursively calculated using the attitude angle at the previous moment and the gyroscope measurement value. In the update step of the Kalman filter, the projection of the gravity vector in the device coordinate system is calculated using the current accelerometer measurement value, and compared with the gravity vector calculated from the attitude angle prediction value to obtain the observation residual; Based on the calculated observation residuals and the observation noise covariance matrix in the Kalman filter model, the covariance matrix of the attitude angle prediction is updated, and the Kalman gain is calculated. The Kalman gain is used to correct the predicted attitude angle value to obtain the optimal attitude angle estimate after fusion calculation, and the gyroscope zero bias estimate is updated simultaneously. The optimal attitude angle estimates calculated at multiple consecutive timestamps are arranged in chronological order to generate the attitude angle change sequence.

[0013] As a further aspect of the present invention, the rate of change and acceleration of the attitude angle change sequence are analyzed. If the rate of change is higher than a preset threshold and the acceleration shows an upward trend, the current interaction state is determined to be an urgent interaction state, including: The attitude angle change sequence is divided into sliding windows. Within each sliding time window, the changes in the roll angle, pitch angle, and yaw angle of the terminal device around its three coordinate axes are calculated. Differentiate the changes in roll angle, pitch angle, and yaw angle respectively to obtain the corresponding attitude angle change rate sequence; Calculate the increment of the rate of change sequence in adjacent time intervals to obtain the acceleration of the attitude angle change; The maximum value in the sequence of rates of change of roll angle, pitch angle, and yaw angle is compared with the preset threshold. At the same time, trend analysis is performed on the acceleration of the roll angle, pitch angle, and yaw angle to determine whether they remain positive within the preset time interval; If the rate of change of the attitude angle is higher than the preset threshold, and the corresponding acceleration of change shows a continuous upward trend within the preset time interval, the condition determination mechanism is triggered to generate an urgent interaction state trigger signal. The task scheduler determines the current interaction state as the urgent interaction state based on the received urgent interaction state trigger signal.

[0014] As a further aspect of the present invention, it also includes the step of updating the lightweight knowledge graph based on an edge-side incremental learning mechanism: After each human-computer interaction process is completed, the multimodal interaction data stream generated in this interaction, the functional service node information, and the inference result data are used as a new training sample. The new training samples are desensitized and feature extracted to generate feature vectors for model fine-tuning; The lightweight inference model is updated with parameters using an online learning algorithm on an edge computing node. The parameter update is only applied to the network layer weights related to the current interaction function. When the performance evaluation index of the lightweight inference model improves beyond the preset convergence criterion, the updated network layer weights are deserialized and stored in the model asset library of the lightweight knowledge graph, replacing the original old weights.

[0015] As a further aspect of the present invention, the lightweight inference model is updated with parameters, wherein the parameter update is only applied to the network layer weights related to the current interaction function, including: In the lightweight knowledge graph, an independent model version number index is established for each type of functional service node; Based on the type of the functional service node corresponding to the current interaction function, lock the weight parameters of the type-related convolutional or fully connected layers in the lightweight inference model. Freeze the weight parameters of the remaining network layers in the lightweight inference model, except for the locking layer; The generated feature vector is input into the unfrozen network, and the gradient descent algorithm is executed to update only the weight parameters of the locked layer, while the weights of the other layers remain unchanged.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The three types of multimodal interactive data streams—speech recognition results (text), gesture trajectory coordinates captured by visual sensors, and posture angle change sequences of the terminal device—are uniformly input into a pre-built lightweight knowledge graph query interface. This interface retrieves functional service nodes related to the current interactive intent from the entity nodes of the knowledge graph based on semantic parsing results, and outputs the functional service nodes and their associated attribute constraints. The comprehensive input of multimodal information replaces single-modal or simple concatenation processing methods. Semantic parsing results can more comprehensively map interactive intents, enabling the retrieval process of functional service nodes to accurately match the user's actual interactive needs. The synchronous output of attribute constraints also provides constraints that perfectly match the intent for interaction execution.

[0017] After receiving the functional service nodes and their attribute constraints, the task scheduler deployed on the edge computing nodes determines the current interaction state by combining the attitude angle change sequence of the terminal device, and adjusts the execution priority of the functional service nodes on the edge computing nodes accordingly. The inference adaptation module loads the corresponding lightweight inference model on the edge computing nodes, compares the adjusted execution priority with the current network bandwidth usage, and generates an inference task queue adapted to the current computing power environment. The dynamic characteristics of the interaction scenario reflected by the terminal attitude angle change are incorporated into the priority adjustment criteria, making the execution priority setting more closely match the actual interaction process. The generation of the inference task queue combines dynamic priority and real-time network bandwidth, making the allocation of computing resources on the edge computing nodes more suitable for the current operating environment, and improving the execution efficiency and adaptability of inference tasks.

[0018] The closed-loop update module updates the status feedback returned by the query interface of the lightweight knowledge graph based on the execution progress of the inference task queue, forming a closed-loop control of the human-computer interaction process. This closed-loop mechanism allows the operating status of the interactive system to be continuously fed back and adjusted, further ensuring the continuity and accuracy of the interaction process. This enables the human-computer interaction system that combines lightweight knowledge graphs with edge computing to better adapt to complex and ever-changing real-world application scenarios. Attached Figure Description

[0019] Figure 1 This is a timing diagram of a lightweight knowledge graph and edge computing human-computer interaction system according to the present invention. Figure 2 A flowchart for knowledge graph query and functional service node retrieval; Figure 3 A bar chart comparing the execution efficiency of each task under different interaction states; Figure 4 A trend analysis chart of signal strength in multimodal interactive data streams; Figure 5 This is a time series analysis chart of performance indicators. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] See Figure 1The data acquisition module acquires a multimodal interaction data stream from the target scene. This data stream includes the speech recognition result text, the gesture trajectory coordinates captured by the visual sensor, and the posture angle change sequence of the terminal device. The graph query module inputs the multimodal interaction data stream into the query interface of a pre-built lightweight knowledge graph. Based on the semantic parsing results, the query interface retrieves functional service nodes related to the current interaction intent from the entity nodes of the graph and outputs the functional service nodes and their associated attribute constraints. The task scheduling module receives the functional service nodes and their associated attribute constraints from the task scheduler deployed on the edge computing node, determines the current interaction state based on the posture angle change sequence of the terminal device, and adjusts the execution priority of the functional service nodes on the edge computing node accordingly. The inference adaptation module loads the lightweight inference model corresponding to the functional service node on the edge computing node, compares the adjusted execution priority with the current network bandwidth usage, and generates an inference task queue adapted to the current computing power environment. The closed-loop update module updates the status feedback returned by the query interface of the lightweight knowledge graph according to the execution progress of the inference task queue, forming a closed-loop control human-computer interaction process.

[0023] In one embodiment of the invention, a microphone array acquires the raw speech signal emitted by the user. The raw speech signal is processed by a noise reduction algorithm and endpoint detection to extract valid speech segments, which are then converted into speech recognition result text by a speech recognition engine. A depth camera acquires a video frame sequence containing the user's hand, which is processed by a hand detection and segmentation model to segment the hand region. Optical flow is used to calculate the displacement of hand feature points between consecutive frames in the hand region to obtain the gesture trajectory coordinates, and Kalman filtering is used to filter out abnormal coordinate points caused by physiological hand tremors. An inertial measurement unit acquires the acceleration and angular velocity data of the terminal device in a spatial coordinate system, and uses a Kalman filtering algorithm to fuse and solve the acceleration and angular velocity data to generate a posture angle change sequence. In a specific implementation, the timestamp-aligned speech recognition result text, gesture trajectory coordinates, and posture angle change sequence are packaged and encapsulated into a multimodal interactive data stream.

[0024] In some embodiments, the process of fusing acceleration and angular velocity data using the Kalman filter algorithm includes an initialization step. The Kalman filter model is initialized. The state vector of the Kalman filter model contains the attitude angle of the terminal device and the gyroscope's zero bias, while the measurement vector contains the three-axis acceleration and angular velocity data acquired by the inertial measurement unit (IMU). Acceleration data is acquired through the IMU's accelerometer, and angular velocity data is acquired through the IMU's gyroscope. These acceleration and angular velocity data are input into the Kalman filter model as system observations. In the prediction step of the Kalman filter, based on the system state equation, the predicted value of the attitude angle at the current moment is recursively calculated using the attitude angle and gyroscope angular velocity measurements from the previous moment. In the update step of the Kalman filter, the projection of the gravity vector in the device coordinate system is calculated using the current accelerometer measurement, and compared with the gravity vector calculated from the attitude angle prediction value to obtain the observation residual. Based on the calculated observation residual and the observation noise covariance matrix in the Kalman filter model, the covariance matrix of the attitude angle prediction value is updated, and the Kalman gain is calculated. The attitude angle predictions are corrected using Kalman gain to obtain the optimal attitude angle estimate after fusion, and the gyroscope bias estimate is updated simultaneously. The observation update equation of the Kalman filter can be expressed as:

[0025] in: This represents the attitude angle updated at time k and the gyroscope's zero-bias state vector. This represents the attitude angle predicted at time k and the gyroscope's zero-bias state vector. This represents the Kalman gain matrix at time k. This represents the observation vector calculated from the accelerometer measurements at time k. The observation matrix represents the state vector mapped to the observation space. After calculating the Kalman gain matrix, the covariance matrix of the attitude angle prediction values ​​is also updated synchronously. The optimal attitude angle estimates calculated at multiple consecutive time stamps are arranged in chronological order to generate an attitude angle change sequence.

[0026] In one embodiment of the present invention, see [reference] Figure 2The system invokes a natural language processing (NLP) engine to process the speech recognition results text in the multimodal interaction data stream. The NLP engine performs dependency parsing and semantic role labeling on the speech recognition results text, identifying the operational verbs and target objects to generate semantic parsing results. For example, if a user utters a statement containing an operation instruction, the speech recognition result text is "increase volume." The NLP engine, through dependency parsing, determines "increase" as the operational verb and "volume" as the target object; the semantic parsing result represents the intent to "increase the volume." The trajectory features of the gesture trajectory coordinates are encoded into a fixed-dimensional feature vector through a feature extraction network. The semantic vector corresponding to the semantic parsing result is fused across modally with the trajectory feature vector of the gesture trajectory coordinates to form a composite interaction intent vector. In some embodiments, cross-modal fusion is achieved through vector concatenation and linear transformation. The semantic parsing result is converted into a semantic vector. The trajectory features of the gesture trajectory coordinates are converted into feature vectors. Composite interaction intent vector Calculated using the following formula:

[0027] in: Represents the fusion weight matrix. Represents the fusion bias vector, symbol This represents the concatenation operation of vectors along a dimensional direction. The calculated composite interactive intent vector is... Used for subsequent map lookups.

[0028] In practice, the composite interaction intent vector is input into the query interface of the lightweight knowledge graph. The query interface traverses the entity nodes and relation edges in the graph, calculating the similarity between the composite interaction intent vector and the embedding vector of each functional service node in the graph. It can be understood that the embedding vector of a functional service node is pre-generated and used to represent the functional semantics provided by that node in the vector space. The query interface filters out candidate functional service nodes whose similarity to the composite interaction intent vector is higher than a set threshold. A confidence score is applied to the filtered candidate functional service nodes, taking into account multiple factors such as vector similarity, node historical call frequency, and current context relevance. The functional service node with the highest confidence and its associated attribute constraints are selected as the query result output. For example, the composite interaction intent vector may match both the "adjust media volume" and "adjust call volume" functional service nodes. The query interface, through confidence scoring, determines that the "adjust media volume" node has a higher score, and ultimately outputs the "adjust media volume" node and its associated volume change range and execution device identifier, among other attribute constraints. In some embodiments, attribute constraints are encapsulated in the query results as key-value pairs.

[0029] Optionally, when performing semantic role labeling in the natural language processing engine, not only are the operational verbs and target objects identified, but also the adverbial components related to the operation. For example, for the speech recognition result text "slowly reduce brightness," the semantic parsing result will include the operational verb "reduce," the target object "brightness," and the adverbial "slowly." The adverbial information can be used to refine and assign values ​​to the attribute constraints of the functional service nodes obtained in subsequent queries. These attribute constraints may include the parameter "speed level." It can be understood that the trajectory features of gesture coordinates include the shape, direction, and speed features of the trajectory. The fusion of these features with the semantic parsing result can more accurately represent the user's complex interaction intent. For example, a circle-drawing trajectory combined with the voice "select this" can point to the "region selection" function rather than a simple "click" function.

[0030] In one embodiment of the present invention, the task scheduler receives functional service nodes and their associated attribute constraints from the graph query module. The task scheduler parses the resource requirements and latency tolerance indicators contained in the attribute constraints. The resource requirements are quantified by processor utilization and memory usage, and the latency tolerance indicator is quantified by the maximum allowable latency time. The task scheduler determines the current interaction state by combining the attitude angle change sequence of the terminal device. Analyzing the rate of change and acceleration of the attitude angle change sequence, if the rate of change is higher than a preset threshold and the acceleration shows an upward trend, the current interaction state is determined to be an urgent interaction state. For scenarios in urgent interaction states, the task scheduler raises the execution priority of the functional service nodes to the highest level and allocates exclusive edge computing resources to the functional service nodes. For scenarios in non-urgent interaction states, the task scheduler dynamically adjusts the execution priority of the functional service nodes according to the latency tolerance indicator, so that they can queue for execution without blocking high-priority tasks.

[0031] In some embodiments, the process of analyzing the rate of change and acceleration of the attitude angle change sequence includes dividing the attitude angle change sequence into sliding windows. Within each sliding time window, the changes in the roll, pitch, and yaw angles of the terminal device around its three coordinate axes are calculated. Differential operations are performed on the changes in roll, pitch, and yaw angles respectively to obtain the roll angle change rate sequence, pitch angle change rate sequence, and yaw angle change rate sequence. The increments of the roll angle change rate sequence, pitch angle change rate sequence, and yaw angle change rate sequence within adjacent time intervals are calculated to obtain the roll angle change acceleration, pitch angle change acceleration, and yaw angle change acceleration. The maximum values ​​in the roll angle change rate sequence, pitch angle change rate sequence, and yaw angle change rate sequence are compared with preset thresholds. It can be understood that the preset thresholds are set empirically to distinguish between normal operation and rapid, urgent operation. Simultaneously, trend analysis is performed on the roll angle change acceleration, pitch angle change acceleration, and yaw angle change acceleration to determine whether these accelerations remain positive continuously within a preset time interval. Trend analysis can be completed by calculating the slope of the acceleration sequence within the preset time interval or by performing linear regression. If the rate of change of attitude angles exceeds a preset threshold, and the corresponding acceleration shows a continuous upward trend within the preset time interval, a condition determination mechanism is triggered, generating an urgent interaction state trigger signal. The task scheduler determines the current interaction state as urgent based on the received urgent interaction state trigger signal. (Attitude angle change acceleration) The calculation can be expressed as:

[0032] in: Indicates time The acceleration of attitude angle change about a certain coordinate axis (e.g., roll axis). Indicates time The rate of change of attitude angle about the corresponding coordinate axis is obtained by differentiating the change of attitude angle. This indicates the derivative with respect to time.

[0033] In practical implementation, the criteria for determining an urgent interaction state require simultaneous fulfillment of both the rate of change condition and the acceleration trend condition. Optionally, the preset thresholds for roll angle, pitch angle, and yaw angle can be set to the same value, or they can be set to different values ​​according to interaction habits. For example, when determining an urgent interaction state, the preset threshold for the roll angle rate of change may be less than the preset threshold for the yaw angle rate of change. It can be understood that the preset time interval length is a configurable parameter, and its length affects the sensitivity to judging the persistence of acceleration trends. In an optional implementation, when the rate of change of multiple attitude angles simultaneously exceeds their respective preset thresholds and the acceleration trends all meet the conditions, the strength of the urgent interaction state trigger signal can be weighted and amplified based on the magnitude of the exceedance. After receiving the urgent interaction state trigger signal, the task scheduler immediately interrupts the currently executing low-priority task and reallocates computing resources to the urgent interaction function service node marked as the highest priority. For non-urgent interaction states, the task scheduler calculates dynamic priority based on the latency tolerance index. The shorter the allowable delay time indicated by the latency tolerance index, the higher the calculated dynamic priority value, and the higher the position of the functional service node in the waiting queue.

[0034] See Figure 3 This is a bar chart comparing the execution efficiency of various tasks under different interaction states, primarily showcasing the difference in execution efficiency between urgent and non-urgent interaction states for five types of tasks. Red bars represent urgent interaction states, corresponding to high-priority, resource-intensive task scenarios; blue bars represent non-urgent interaction states, corresponding to regular queuing tasks. Among all task types, the execution efficiency of urgent interaction states is significantly higher than that of non-urgent interaction states, with the difference remaining stable at 17-18 percentage points, reflecting the system's resource allocation and priority guarantee for urgent interactions. Pose calculation maintains the highest efficiency in both states, making it the most stable task module in the system. Graph query has the lowest efficiency in both states, possibly limited by the inherent complexity of knowledge graph retrieval. All tasks have an efficiency below 80% in non-urgent states, further illustrating the strong guarantee effect of resource scheduling strategies on urgent interactions.

[0035] In one embodiment of the present invention, the inference adaptation module retrieves the corresponding lightweight inference model from the model library deployed locally on the edge computing node based on the type identifier of the functional service node. The lightweight inference model is a neural network model that has undergone pruning and quantization. The inference adaptation module periodically monitors the network connection status between the edge computing node and the external cloud, and obtains the current network bandwidth usage by sending probe data packets to the cloud server and measuring round-trip latency and throughput. The inference adaptation module establishes a mapping table between execution priority and network bandwidth usage. If the execution priority is high and the network bandwidth usage is below the lower limit threshold, the remote data download task required during the inference process is added to the inference task queue. The inference adaptation module sorts all pending inference tasks according to task dependencies and timestamp order, generating an inference task queue adapted to the current computing power environment. When each task in the inference task queue starts, the inference adaptation module sends a signal to the query interface of the lightweight knowledge graph to start task execution. The inference adaptation module monitors the running status and completion percentage of each task in the inference task queue in real time, and writes the latest running status into the status attribute field of the corresponding functional service node in the lightweight knowledge graph. Once all tasks in the inference task queue have been completed, the inference adaptation module aggregates the final inference result data and encapsulates it into a status feedback data packet, which is then sent to the lightweight knowledge graph's query interface. Upon receiving the status feedback data packet, the lightweight knowledge graph's query interface triggers a refresh command on the human-computer interaction interface, completing a closed-loop interaction.

[0036] In some embodiments, the mapping relationship between execution priority and network bandwidth usage is defined as a decision table. This mapping table specifies the processing strategy for remote data download tasks under different combinations of execution priorities and network bandwidth usage ranges. Refer to Table 1, which shows one possible mapping table.

[0037] Table 1: Mapping Relationship between Execution Priority and Network Bandwidth Usage

[0038] The inference adaptation module determines the corresponding processing strategy by querying a mapping table based on the execution priority of the functional service nodes issued by the task scheduler and the real-time monitored network bandwidth utilization. The lower and upper thresholds of network bandwidth utilization can be configured according to the actual network environment and application requirements of the edge nodes. When generating the inference task queue, the task order must follow the dependencies of a directed acyclic graph. For example, if an image recognition task depends on a previously completed model loading task and a data download task, then the model loading task and the data download task must be scheduled before the image recognition task. Among tasks with dependencies, the timestamp order has lower priority than the dependency relationship; for independent tasks without dependencies, they are arranged according to their timestamp order. Optionally, when the inference adaptation module writes the running status to the status attribute field of the lightweight knowledge graph, the field value can include status indicators such as "waiting," "in execution," "completed XX%," "completed," and "failed." The refresh command of the human-computer interaction interface carries key result data from the status feedback data packet, driving the interface to update the displayed content or execute corresponding control actions.

[0039] In practice, the generation of the inference task queue is a dynamic calculation process. The inference adaptation module calculates a comprehensive priority score for each task to be scheduled. This is used to determine the order when there are no dependency conflicts. (Combined priority score) It is determined by both the execution priority weight and the network bandwidth adaptation coefficient, and its calculation formula is as follows:

[0040] in: This represents the overall priority score of the task; the higher the score, the higher the position in the queue. This represents the normalized value of the execution priority set by the task scheduler; This represents the normalized value of the current network bandwidth utilization, ranging from 0 to 1; and These are weighting coefficients used to balance the impact of execution priority and network conditions. In the formula This means that the less busy the network, the greater the contribution to the score, encouraging task scheduling when the network is good. The inference adaptation module follows... The scores are sorted from highest to lowest for parallelizable tasks, generating the final inference task queue. During the closed-loop update process, when the lightweight knowledge graph query interface receives a status feedback data packet, in addition to triggering a UI refresh, the complete log of this interaction is stored in an edge database for subsequent analysis.

[0041] See Figure 4This is a multimodal interaction data stream signal strength trend analysis chart, showing the dynamic changes of three types of interaction data within 0-60 seconds. The posture angle change exhibits a typical sinusoidal fluctuation characteristic, reaching double peaks at 8s and 58s, and bottoming out at 32s, reflecting the periodic rapid changes in the terminal device's posture, which is the core signal source triggering the "urgent interaction state." The gesture trajectory coordinates initially have the highest intensity, then continuously decrease to a trough at 40s, before gradually recovering, reflecting the process of user gesture operation from active to stable and then back to active. The voice signal intensity fluctuates smoothly overall, gradually rising from 70% to a peak at 15s, then slowly decaying to a trough at 45s, and finally slightly recovering, conforming to the typical rhythm of voice interaction: "initiation-continuation-end." The three types of signal intensity show a clear temporal shift, providing continuous interactive input to the system and avoiding interaction interruptions caused by the failure of a single modality.

[0042] In one embodiment of the present invention, after each human-computer interaction process based on lightweight knowledge graph and edge computing, the closed-loop update module initiates an incremental learning mechanism based on the edge side. The incremental learning mechanism uses the multimodal interaction data stream, functional service node information, and inference result data generated in this interaction as a new training sample. The new training sample is sent to the data processing unit, which performs desensitization and feature extraction on the new training sample. The desensitization operation removes any personal identification information that may be contained in the new training sample, and the feature extraction operation extracts feature vectors for model fine-tuning from the new training sample. Using an online learning algorithm deployed on the edge computing node, the parameters of the lightweight inference model invoked in this interaction are updated. The parameter update only applies to the network layer weights related to the function of this interaction. When the performance evaluation index of the lightweight inference model improves beyond a preset convergence criterion, the incremental learning mechanism deserializes the updated network layer weights and stores them in the model asset library of the lightweight knowledge graph, replacing the original old weights in the model asset library of the lightweight knowledge graph, thus completing the update of the model components in the lightweight knowledge graph.

[0043] In some embodiments, desensitization and feature extraction of new training samples include replacing or deleting sensitive words in the speech recognition result text of the multimodal interaction data stream, and blurring frames in the gesture trajectory coordinates captured by the visual sensor that may contain user background environment information. The feature extraction operation uses a feature encoder with the same structure as the front-end layer of the lightweight inference model to map the processed multimodal interaction data stream into high-dimensional feature vectors. The online learning algorithm uses mini-batch gradient descent or its variants to update the parameters of the lightweight inference model at edge computing nodes using a loss function composed of the newly generated feature vectors and the inference result data. It can be understood that the performance evaluation metric can be the improvement in accuracy, precision, recall, or F1 score on a small validation set held locally by the model. The preset convergence criterion can be an absolute improvement threshold, such as an accuracy improvement of more than 0.5%, or the metric no longer decreasing after a number of consecutive incremental learning iterations. Once the convergence criteria are met, the updated network layer weights are serialized into binary files and written to the model storage path associated with the corresponding functional service node in the model asset library through the lightweight knowledge graph management interface, overwriting the old version of the weight file.

[0044] In practice, updating the parameters of the lightweight inference model, focusing solely on the network layer weights relevant to the current interactive function, involves index creation and layer locking. A separate model version number index is established for each type of functional service node in the lightweight knowledge graph. This index records the currently active model weight file version and historical versions. Based on the functional service node type corresponding to the current interactive function, the incremental learning mechanism locks the weight parameters of the type-sensitive convolutional or fully connected layers in the lightweight inference model. For example, for a "image classification" type functional service node, the last fully connected classification layer in the lightweight inference model is locked; for a "object detection" type, the weight parameters of the network layer responsible for region proposal might be locked. The weight parameters of the remaining network layers in the lightweight inference model, except for the locked layer, are frozen to ensure that these weights remain unchanged during incremental learning. The generated feature vectors for model fine-tuning are input into the unfrozen network, and the gradient descent algorithm in the online learning algorithm is executed. Only the weight parameters of the locked layer are updated; the gradients of the weights of the remaining layers are masked during backpropagation, and their values ​​remain unchanged. The weight parameter update process follows the formula below:

[0045] in: Indicates the first The set of network layer weight parameters that are locked and related to the current interactive function during each iteration (or training step); This represents the updated set of corresponding weight parameters; This represents the learning rate set in the online learning algorithm, controlling the step size for updating parameters; Indicates based on the current mini-batch training data Calculated parameters of the locking layer The loss function gradient; the minus sign indicates updating the parameters in the opposite direction of the gradient to minimize the loss. This can be understood as mini-batch training data. It consists of feature vectors generated from this interaction and several recent interactions, along with the corresponding inference results.

[0046] Optionally, the model version number index in the lightweight knowledge graph is managed using an incrementing integer. Each successful weight update increments the model version number of the corresponding functional service node. When locking network layers, the system operates based on a predefined "node type-network layer" mapping table, which is stored in the lightweight knowledge graph's configuration information. Freezing the weight parameters of other network layers is achieved by setting the `requires_grad` attribute of the corresponding parameter to `False` in the deep learning framework. It can be understood that updating only a portion of the network layer weights significantly reduces the computational and memory requirements for incremental learning, making continuous model optimization possible even in environments with limited edge computing node resources.

[0047] See Figure 5 This is a time-series performance analysis chart, showing the dynamic trends of the system's core evaluation metrics during the 0-100 step training cycle, and their comparison with the preset performance target. Accuracy peaks at 95% from steps 40-50, consistently exceeding the 90% performance target, before falling back to 90% later, consistent with the "convergence and stabilization" characteristic of incremental learning, indicating overall performance remains up to standard. Precision initially has the highest value, representing extremely high accuracy in recognizing positive samples. It continuously declines with increasing iteration steps, reaching a low of 55% at step 80. Although it recovers somewhat later, it doesn't return to the initial level, suggesting that the system needs to strengthen the monitoring and regularization of precision during multimodal intent fusion. Recall peaks at 95% around step 30, then rapidly declines, falling below 45% at step 80. This reflects a significant decrease in the model's recall ability for marginal cases / rare interaction intents in later stages, possibly due to "catastrophic forgetting" of old knowledge during incremental learning.

[0048] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A lightweight human-computer interaction system based on knowledge graphs and edge computing, characterized in that, The system includes: The data acquisition module acquires a multimodal interaction data stream in the target scene. The multimodal interaction data stream includes the text of the speech recognition result, the coordinates of the gesture trajectory captured by the visual sensor, and the posture angle change sequence of the terminal device. The graph query module inputs the multimodal interaction data stream to the query interface of the pre-built lightweight knowledge graph. The query interface of the lightweight knowledge graph retrieves the functional service nodes related to the current interaction intent from the entity nodes of the graph according to the semantic parsing results, and outputs the functional service nodes and their associated attribute constraints. The task scheduling module receives the functional service node and its associated attribute constraints from the task scheduler deployed on the edge computing node, determines the current interaction state by combining the attitude angle change sequence of the terminal device, and adjusts the execution priority of the functional service node on the edge computing node accordingly. The inference adaptation module loads the lightweight inference model corresponding to the functional service node on the edge computing node, compares the adjusted execution priority with the current network bandwidth usage, and generates an inference task queue adapted to the current computing power environment. The closed-loop update module updates the status feedback returned by the query interface of the lightweight knowledge graph according to the execution progress of the inference task queue, forming a closed-loop control human-computer interaction process.

2. The lightweight knowledge graph and edge computing human-computer interaction system according to claim 1, characterized in that, The acquisition of the multimodal interaction data stream in the target scenario includes: The system collects the user's original speech signal through a microphone array, performs noise reduction and endpoint detection on the original speech signal, extracts the effective speech segments, and converts them into the speech recognition result text. The system acquires a video frame sequence containing the user's hand using a depth camera, segments the hand region in the video frame sequence, calculates the coordinates of the gesture trajectory using optical flow, and filters out abnormal coordinate points caused by hand tremors. The terminal device's acceleration and angular velocity data in the spatial coordinate system are acquired by the inertial measurement unit, and the Kalman filter algorithm is used to fuse and solve the acceleration and angular velocity data to generate the attitude angle change sequence. The timestamp-aligned speech recognition result text, gesture trajectory coordinates, and posture angle change sequence are packaged and encapsulated into the multimodal interaction data stream.

3. The lightweight knowledge graph and edge computing human-computer interaction system according to claim 2, characterized in that, The multimodal interaction data stream is input to the query interface of a pre-built lightweight knowledge graph. Based on the semantic parsing results, the query interface retrieves functional service nodes related to the current interaction intent from the entity nodes of the graph, and outputs the functional service nodes and their associated attribute constraints, including: The natural language processing engine is invoked to perform dependency parsing and semantic role labeling on the speech recognition result text, identify the operation verbs and target objects, and generate the semantic parsing result; The semantic parsing results are fused with the trajectory features of the gesture trajectory coordinates across modalities to form a composite interactive intent vector; The composite interaction intent vector is input into the query interface of the lightweight knowledge graph. The query interface traverses the entity nodes and relation edges in the graph and filters out candidate functional service nodes that match the composite interaction intent vector. The selected candidate functional service nodes are scored with confidence, and the functional service node with the highest confidence and its associated attribute constraints are selected as the query result output.

4. The lightweight knowledge graph and edge computing human-computer interaction system according to claim 3, characterized in that, The process of receiving the functional service node and its associated attribute constraints from the task scheduler deployed on the edge computing node, determining the current interaction state by combining the attitude angle change sequence of the terminal device, and adjusting the execution priority of the functional service node on the edge computing node accordingly includes: Analyze the resource requirements and latency tolerance indicators contained in the attribute constraints; Analyze the rate of change and acceleration of the attitude angle change sequence. If the rate of change is higher than a preset threshold and the acceleration shows an upward trend, then the current interaction state is determined to be an urgent interaction state. For scenarios in the urgent interaction state, the execution priority of the functional service node is raised to the highest level, and it is allocated exclusive edge computing resources. For scenarios with non-urgent interaction, the execution priority of the functional service nodes is dynamically adjusted according to the latency tolerance index, so that they can queue up and wait for execution without blocking high-priority tasks.

5. A lightweight knowledge graph and edge computing human-computer interaction system according to claim 4, characterized in that, The process of loading the lightweight inference model corresponding to the functional service node on the edge computing node, comparing the adjusted execution priority with the current network bandwidth usage, and generating an inference task queue adapted to the current computing power environment includes: Based on the type identifier of the functional service node, the corresponding lightweight inference model is retrieved from the local model library. The lightweight inference model is a neural network model that has undergone pruning and quantization processing. Periodically monitor the network connection status between the edge computing node and the external cloud to obtain the current network bandwidth usage; Establish a mapping table between execution priority and network bandwidth usage. If the execution priority is high and the network bandwidth usage is below the lower limit threshold, then the remote data download task required during the inference process will be merged into the inference task queue. Based on task dependencies and timestamp order, all pending inference tasks are sorted to generate the inference task queue adapted to the current computing power environment.

6. A lightweight knowledge graph and edge computing human-computer interaction system according to claim 5, characterized in that, Based on the execution progress of the inference task queue, the status feedback returned by the query interface of the lightweight knowledge graph is updated to form a closed-loop control human-computer interaction process, including: When each task in the inference task queue is started, a signaling message indicating that the task has started execution is sent to the query interface of the lightweight knowledge graph. The running status and completion percentage of each task in the inference task queue are monitored in real time, and the latest running status is written into the status attribute field of the corresponding functional service node in the lightweight knowledge graph. After all tasks in the inference task queue have been completed, the final inference result data is aggregated and encapsulated into a status feedback data packet, which is then sent to the query interface of the lightweight knowledge graph. After receiving the status feedback data packet, the query interface of the lightweight knowledge graph triggers a refresh command for the human-computer interaction interface, completing a closed-loop interaction.

7. A lightweight knowledge graph and edge computing human-computer interaction system according to claim 6, characterized in that, The step of fusing and solving the acceleration and angular velocity data using the Kalman filter algorithm to generate the attitude angle change sequence includes: Initialize the Kalman filter model, where the state vector contains the attitude angle of the terminal device and the gyroscope zero bias, and the measurement vector contains the triaxial acceleration and angular velocity data collected by the inertial measurement unit; The acceleration and angular velocity data are acquired by the accelerometer and gyroscope of the inertial measurement unit, respectively, and then input into the Kalman filter model as the system's observations. In the prediction step of the Kalman filter, based on the system state equation, the predicted value of the attitude angle at the current moment is recursively calculated using the attitude angle at the previous moment and the gyroscope measurement value. In the update step of the Kalman filter, the projection of the gravity vector in the device coordinate system is calculated using the current accelerometer measurement value, and compared with the gravity vector calculated from the attitude angle prediction value to obtain the observation residual; Based on the calculated observation residuals and the observation noise covariance matrix in the Kalman filter model, the covariance matrix of the attitude angle prediction is updated, and the Kalman gain is calculated. The Kalman gain is used to correct the predicted attitude angle value to obtain the optimal attitude angle estimate after fusion calculation, and the gyroscope zero bias estimate is updated simultaneously. The optimal attitude angle estimates calculated at multiple consecutive timestamps are arranged in chronological order to generate the attitude angle change sequence.

8. A lightweight knowledge graph and edge computing human-computer interaction system according to claim 7, characterized in that, Analyzing the rate of change and acceleration of the attitude angle change sequence, if the rate of change is higher than a preset threshold and the acceleration shows an upward trend, then the current interaction state is determined to be an urgent interaction state, including: The attitude angle change sequence is divided into sliding windows. Within each sliding time window, the changes in the roll angle, pitch angle, and yaw angle of the terminal device around its three coordinate axes are calculated. Differentiate the changes in roll angle, pitch angle, and yaw angle respectively to obtain the corresponding attitude angle change rate sequence; Calculate the increment of the rate of change sequence in adjacent time intervals to obtain the acceleration of the attitude angle change; The maximum value in the sequence of rates of change of roll angle, pitch angle, and yaw angle is compared with the preset threshold. At the same time, trend analysis is performed on the acceleration of the roll angle, pitch angle, and yaw angle to determine whether they remain positive within the preset time interval; If the rate of change of the attitude angle is higher than the preset threshold, and the corresponding acceleration of change shows a continuous upward trend within the preset time interval, the condition determination mechanism is triggered to generate an urgent interaction state trigger signal. The task scheduler determines the current interaction state as the urgent interaction state based on the received urgent interaction state trigger signal.

9. A lightweight knowledge graph and edge computing human-computer interaction system according to claim 8, characterized in that, It also includes the step of updating the lightweight knowledge graph using an edge-based incremental learning mechanism: After each human-computer interaction process is completed, the multimodal interaction data stream generated in this interaction, the functional service node information, and the inference result data are used as a new training sample. The new training samples are desensitized and feature extracted to generate feature vectors for model fine-tuning; The lightweight inference model is updated with parameters using an online learning algorithm on an edge computing node. The parameter update is only applied to the network layer weights related to the current interaction function. When the performance evaluation index of the lightweight inference model improves beyond the preset convergence criterion, the updated network layer weights are deserialized and stored in the model asset library of the lightweight knowledge graph, replacing the original old weights.

10. A lightweight knowledge graph and edge computing human-computer interaction system according to claim 9, characterized in that, The lightweight inference model is updated with parameters, which are only applied to the network layer weights relevant to this interaction function, including: In the lightweight knowledge graph, an independent model version number index is established for each type of functional service node; Based on the type of the functional service node corresponding to the current interaction function, lock the weight parameters of the type-related convolutional or fully connected layers in the lightweight inference model. Freeze the weight parameters of the remaining network layers in the lightweight inference model, except for the locking layer; The generated feature vector is input into the unfrozen network, and the gradient descent algorithm is executed to update only the weight parameters of the locked layer, while the weights of the other layers remain unchanged.

Images