Intelligent exhibition and teaching device adaptive scheduling method and system based on multi-modal interaction

Abstract

The application discloses a multi-modal interaction-based intelligent exhibition teaching equipment adaptive scheduling method and system, and belongs to the technical field of multi-modal interaction. In a baseline learning stage, multi-modal interaction scene data is collected, resource occupation features are extracted, a clustering algorithm is used to divide typical interaction mode clusters, a mean vector and a covariance matrix are calculated, a multivariate Gaussian sampler is constructed to generate simulated competition feature vectors, and a scheduling prior knowledge base is formed; in an online monitoring stage, a sliding time window is used to extract a real-time resource competition feature vector sequence, the sequence is matched with a simulated feature vector set to calculate a cumulative Mahalanobis distance, and a Viterbi decoding is used in combination with a transition probability matrix to identify a current interaction mode category. The application builds a knowledge base containing simulated features through offline learning, and solves the problem of high-concurrency interaction data sparsity; online monitoring uses Mahalanobis distance matching based on simulated features and hidden Markov timing decoding, and precise identification of complex multi-modal interaction modes is realized.

Description

Technical Field

[0001] This invention relates to the field of multimodal interaction technology, specifically to an adaptive scheduling method and system for smart educational equipment based on multimodal interaction. Background Technology

[0002] Smart exhibition and education equipment (such as interactive digital booths, multimedia guide terminals, and immersive interactive exhibits) is widely used in science and technology museums, museums, exhibition halls, and other venues. These devices typically support multiple human-computer interaction modalities, including voice commands, gestures, touch clicks, and eye contact. Visitors interact naturally with the devices through these modalities, while the devices must respond in real time and allocate computing resources (CPU, GPU, memory, network bandwidth) to process various interaction requests.

[0003] However, in real-world multimodal concurrent interaction scenarios, the scheduling technology of existing educational equipment has the following shortcomings: Traditional scheduling strategies typically employ fixed priority ordering (e.g., voice over gestures). When multiple visitors interact with the same exhibit simultaneously using different modalities, high-priority modalities consume processing resources for extended periods, while low-priority modalities experience significant delays in response. This results in some visitors' interaction intentions being ignored, leading to a decline in user experience.

[0004] Multi-user, multimodal concurrent interaction generates a large number of resource contention feature vectors. When the number of concurrent users exceeds a preset threshold, the device response latency jumps from milliseconds to seconds, and even device freezes. Existing technologies lack prior knowledge of high-load interaction modes and cannot predict resource demands in advance or perform preventative scheduling.

[0005] The educational equipment accumulates a large number of historical interaction logs during operation, but existing technologies have failed to extract a reusable interaction pattern library from them. At the same time, the cost of data collection in real high-concurrency interaction scenarios is high and the samples are sparse, resulting in insufficient generalization ability of the scheduling model.

[0006] Therefore, there is an urgent need for a scheduling method that can learn the patterns of historical interaction, identify the current load type online, and adaptively allocate computing resources to improve the smoothness and reliability of smart exhibition and education equipment in multimodal interaction scenarios. Summary of the Invention

[0007] The purpose of this invention is to provide an adaptive scheduling method for intelligent educational exhibition equipment based on multimodal interaction, so as to solve the problems mentioned in the background art.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: An adaptive scheduling method for smart educational equipment based on multimodal interaction, comprising the following steps: Baseline learning phase: Collect data from various preset interaction scenarios, extract resource occupancy features corresponding to each interaction event, and use a clustering algorithm based on the resource occupancy features to divide the interaction events into several typical interaction pattern clusters, and generate typical feature distribution parameters for each typical interaction pattern cluster; construct a simulated feature generator based on the typical feature distribution parameters to generate simulated competition feature vectors, and summarize and store the simulated feature generators and simulated competition feature vectors corresponding to all typical interaction pattern clusters to form a scheduling prior knowledge base; In the online monitoring phase: Multimodal interactive sensing data streams are captured using a sliding time window approach. Real-time resource contention feature vectors within the current time window are extracted and arranged in chronological order to form a real-time feature sequence. Each real-time resource contention feature vector in the real-time feature sequence is matched with the simulated contention feature vector set of each typical interaction mode cluster in the scheduling prior knowledge base, and the cumulative Mahalanobis distance is calculated. Combined with a preset transition probability matrix, the Viterbi decoding algorithm is used to solve for the hidden state sequence that minimizes the total cost of the entire real-time feature sequence. The typical interaction mode cluster with the highest frequency of occurrence is selected from this hidden state sequence as the interaction mode category of the current time window.

[0009] It should be noted that in the above method, a prior knowledge base for scheduling is constructed through offline baseline learning, and the sample diversity of high-concurrency scenarios is amplified by using a simulated feature generator; during online monitoring, the current interaction pattern is identified based on Mahalanobis distance and Hidden Markov Model, providing accurate category determination for subsequent adaptive scheduling decisions, thereby avoiding resource contention rigidity and response crashes.

[0010] Furthermore, the specific implementation process of the baseline learning phase includes: before the exhibition and education equipment is put into online scheduling operation, collecting various preset interactive scenario data in an offline environment. The interactive scenario data includes sensor signal streams generated by different concurrent numbers of visitors interacting with the exhibition and education equipment using at least two interaction modalities, such as voice commands, gestures, touch clicks, and eye gaze; performing modal separation and time alignment on each sensor signal stream, and extracting resource occupancy features corresponding to each interactive event. The resource occupancy features include modal type identifier, event duration, cumulative processor occupancy time, memory access frequency, and communication bandwidth occupancy value; based on the resource occupancy features, using a clustering algorithm to divide the interactive events into several typical interactive mode clusters, and generating typical feature distribution parameters, wherein each typical interactive mode cluster corresponds to a sample resource competition feature vector; constructing a simulated feature generator according to the typical feature distribution parameters, generating simulated competition feature vectors, and forming a scheduling prior knowledge base.

[0011] Furthermore, the baseline learning phase collects various preset interaction scenario data, specifically including: constructing standardized test scripts that include four types of scenarios: single-person single-modal, multi-person single-modal, single-person multi-modal, and multi-person multi-modal; driving the standardized test scripts during the interaction between the testers and the educational equipment; and synchronously recording the system resource call logs and interaction response timestamps triggered by each interaction event.

[0012] Furthermore, the clustering algorithm is used to divide the interaction events into several typical interaction pattern clusters. Specifically, the sample resource competition feature vector corresponding to each interaction event is regarded as a data point in a high-dimensional space, and the data points corresponding to all interaction events constitute the original dataset. A neighborhood radius threshold and a density threshold are set. For each data point in the original dataset, the number of other data points contained in the hypersphere centered on a certain data point and with the neighborhood radius threshold as the radius is counted. This number is used as the neighborhood density value of the hypersphere constructed with the certain data point as the center. Data points with neighborhood density values ​​greater than or equal to the density threshold are selected and marked as core points. For any core point, all other core points and boundary points located within the neighborhood radius threshold of any core point are assigned to the same initial cluster. All core points are traversed, and the disjoint-set data structure algorithm is used to merge mutually reachable initial clusters to obtain multiple candidate clusters. Candidate clusters with a number of data points lower than the preset minimum sample number are removed as noise clusters, and the remaining candidate clusters are the typical interaction pattern clusters.

[0013] Furthermore, the specific implementation process for generating typical feature distribution parameters includes: the components of the sample resource competition feature vector include: event duration component, processor cumulative occupancy duration component, memory access frequency component, and communication bandwidth occupancy value component; wherein, the value of the event duration component is the time length between the triggering time of the interactive event and the completion time of the response output; the value of the processor cumulative occupancy duration component is the cumulative number of microseconds occupied by the central processing unit and the graphics processing unit during the event duration; the value of the memory access frequency component is the weighted sum of the number of page faults and the number of memory page replacements during the event duration; and the value of the communication bandwidth occupancy value component is the value of the event duration... The total amount of data sent and received by the network interface of the educational equipment within a given time period; for any typical interaction mode cluster, calculate the arithmetic mean of the resource competition feature vector of all samples within the cluster at each component, and combine the arithmetic means of all components in a fixed order to form the mean vector of the typical interaction mode cluster; for any typical interaction mode cluster, calculate the covariance between any two different components within the cluster, and arrange the variance of each component itself and the covariance between different components in matrix form to form the covariance matrix of the typical interaction mode cluster; the mean vector and the covariance matrix are used together as the typical feature distribution parameters of the typical interaction mode cluster.

[0014] Furthermore, the specific implementation process of constructing the scheduling prior knowledge base includes: for any typical interaction pattern cluster, using the mean vector and covariance matrix of the typical interaction pattern cluster as the statistical benchmark for generating simulated competition feature vectors; constructing a multivariate Gaussian sampler as the simulated feature generator for the typical interaction pattern cluster, wherein the multivariate Gaussian sampler uses the mean vector as the expectation center and the covariance matrix as the covariance structure between the components of the sample resource competition feature vector; the multivariate Gaussian sampler generates simulated competition feature vectors in the following way: independently sampling a random noise vector with the same dimension as the sample resource competition feature vector from the standard normal distribution, performing Choliski decomposition on the covariance matrix to obtain a lower triangular matrix, multiplying the random noise vector with the lower triangular matrix, and then adding it to the mean vector, the result of which is a simulated competition feature vector; summarizing and storing the multivariate Gaussian samplers of all typical interaction pattern clusters and the set of simulated competition feature vectors generated by the multivariate Gaussian samplers, which together constitute the scheduling prior knowledge base.

[0015] Furthermore, the specific implementation process of the online monitoring phase includes: continuously sliding the time window during the online operation of the educational equipment with a preset time window length and window overlap rate; for the current time window, performing event-level segmentation on the multimodal interactive sensing data stream captured within the current time window to identify each complete interactive event; for any identified complete interactive event, extracting the resource occupancy features of that complete interactive event and converting them into a real-time resource contention feature vector; arranging all the real-time resource contention feature vectors obtained within the current time window in chronological order of event occurrence to form a real-time feature sequence; if no complete interactive event is detected within the current time window, skipping the current time window; for the real-time features... For any real-time resource competition feature vector in the sequence, for any typical interaction pattern cluster, calculate the Mahalanobis distance between the real-time resource competition feature vector and each simulated competition feature vector in the simulated competition feature vector set of the typical interaction pattern cluster. The Mahalanobis distance is calculated using the covariance matrix of the typical interaction pattern cluster as the scaling matrix, and the minimum Mahalanobis distance is selected as the minimum Mahalanobis distance between the real-time resource competition feature vector and the typical interaction pattern cluster. For any typical interaction pattern cluster, accumulate the minimum Mahalanobis distances corresponding to all real-time resource competition feature vectors in the current real-time feature sequence to obtain the cumulative Mahalanobis distance of the typical interaction pattern cluster at the real-time feature sequence level.

[0016] Furthermore, the specific implementation process of the online monitoring phase also includes: pre-setting a transition probability matrix, where each element of the transition probability matrix represents the transition probability of transitioning from one typical interaction mode cluster to another between adjacent time windows. This transition probability is obtained by statistically analyzing the transition frequency of typical interaction modes in historical interaction scene data collected offline; using the cumulative Mahalanobis distance corresponding to each typical interaction mode cluster as the observation cost, and combining it with the transition probability matrix, the Viterbi decoding algorithm is used to solve for the hidden state sequence that minimizes the total cost of the entire real-time feature sequence. Each hidden state in the hidden state sequence corresponds to a typical interaction mode cluster; from the decoded hidden state sequence, the typical interaction mode cluster with the highest occurrence frequency is selected as the interaction mode category of the current time window.

[0017] The present invention also provides an adaptive scheduling system for intelligent educational equipment based on multimodal interaction, comprising: a multimodal interaction platform; at least one processor; at least one memory storing a computer program; when the computer program is executed by the at least one processor, it implements all the steps of the method described above; the multimodal interaction platform is used to schedule one or more computer programs, and when the one or more computer programs are executed by the one or more processors, the multimodal interaction platform implements the steps of the method described above.

[0018] The beneficial effects achieved by this invention are: (1) For the technical problem of poor generalization ability of scheduling model due to the sparse sample in high-concurrency interaction scenario using real samples in historical data, this invention automatically discovers typical interaction pattern clusters through density clustering, and uses a multivariate Gaussian sampler to generate a large number of simulated competitive feature vectors based on the mean vector and covariance matrix, thereby achieving effective expansion of the sample space at extremely low cost, so that the knowledge base can cover rare but critical load patterns.

[0019] (2) The present invention calculates the Mahalanobis distance between the real-time vector and each vector in the simulated feature set and takes the minimum value. It not only makes use of the richness of the simulated feature set, but also eliminates the influence of feature dimensions through the scaling transformation of the covariance matrix. The matching accuracy is significantly higher than that of the traditional method.

[0020] (3) For single-mode or multi-mode decision-making in a single time slice, which is susceptible to instantaneous interference and misjudgment, this invention uses the cumulative Mahalanobis distance as the observation cost, combines it with the mode transition probability matrix obtained from historical data statistics, and uses Viterbi decoding to solve the global optimal hidden state sequence, making full use of the continuity and transition law of the interaction mode in the time dimension.

[0021] In summary, the interaction mode categories output by this invention can be directly mapped to predefined resource scheduling strategies (such as CPU quota allocation and thread priority adjustment) without the need for manual parameter tuning. This enables adaptive scheduling of educational equipment in multi-user, multi-modal concurrent scenarios, thereby reducing equipment response latency and effectively improving user experience. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0023] Figure 1 This is a schematic diagram illustrating the steps of the adaptive scheduling method for intelligent educational equipment based on multimodal interaction according to the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In this first embodiment: an adaptive scheduling system for smart educational equipment based on multimodal interaction is provided. The system includes: a multimodal data acquisition module, an offline baseline learning module, an online monitoring and scheduling decision module, and a resource scheduling execution module.

[0026] The multimodal data acquisition module is configured to collect sensor signal streams such as voice commands, gestures, touch clicks, and eye gaze generated during the interaction between visitors and exhibition equipment, and to perform timestamp marking and modal separation.

[0027] The offline baseline learning module is configured as follows: Before the educational equipment is put into online operation, it receives various preset interactive scenario data collected by the multimodal data acquisition module, extracts resource occupancy features corresponding to each interactive event (including modality type identifier, event duration, cumulative processor occupancy time, memory access frequency, and communication bandwidth occupancy value), and uses a density-based clustering algorithm (such as DBSCAN) to divide the interactive events into several typical interactive pattern clusters. For each cluster, its mean vector and covariance matrix are calculated as typical feature distribution parameters. Then, a multivariate Gaussian sampler is constructed, using the mean vector and covariance matrix of the cluster as parameters, to generate simulated competition feature vectors. The multivariate Gaussian samplers of all clusters and the generated simulated competition feature vector set together constitute a scheduling prior knowledge base, which is stored in non-volatile memory.

[0028] The online monitoring and scheduling decision module is configured as follows: During the online operation of the educational equipment, it continuously captures multimodal interactive sensor data streams using a sliding time window approach. For each time window, event-level segmentation is performed, and the real-time resource contention feature vectors of each complete interactive event are extracted and arranged into a real-time feature sequence according to time sequence. Then, for each real-time feature vector, its minimum Mahalanobis distance with the simulated contention feature vector sets of each cluster in the scheduling prior knowledge base is calculated, and the cumulative Mahalanobis distance of each cluster is accumulated as the observation cost. Combining the transition probability matrix obtained from historical data statistics, the Viterbi decoding algorithm is used to solve for the optimal hidden state sequence, and the cluster with the highest occurrence frequency is selected as the interaction mode category of the current time window.

[0029] The resource scheduling execution module is configured to: based on the identified current interaction mode category, look up the corresponding resource scheduling strategy (such as the CPU quota of the speech recognition engine, the priority of the gesture recognition thread, etc.) in a table, and send it to the operating system resource manager for execution.

[0030] Please see Figure 1 In this second embodiment: an adaptive scheduling method for smart educational equipment based on multimodal interaction is provided, which includes the following steps: Step S1 (Baseline Learning Phase): S101: Before the exhibition and education equipment is put into online scheduling and operation, collect data of various preset interactive scenarios in an offline environment; specifically, construct standardized test scripts that include four types of scenarios: single-person single-modal, multi-person single-modal, single-person multi-modal, and multi-person multi-modal, drive testers to interact with the exhibition and education equipment according to the scripts, and synchronously record sensor signal streams such as voice, gestures, touch, and gaze, as well as system resource call logs.

[0031] S102: Perform modal separation and time alignment on each sensor signal stream, and extract the resource occupancy features corresponding to each interaction event; the resource occupancy features include: modal type identifier (e.g., voice is encoded as 1, gesture is encoded as 2, etc.), event duration (milliseconds), processor cumulative occupancy time (CPU + GPU microseconds), memory access frequency (weighted sum of page faults and page replacements), and communication bandwidth occupancy value (total amount of data sent and received).

[0032] S103: Based on the resource occupancy characteristics, a clustering algorithm is used to divide the interaction events into several typical interaction pattern clusters. In this embodiment, the DBSCAN algorithm is used: the resource competition feature vector of each interaction event is regarded as a data point, a neighborhood radius threshold ε and a density threshold MinPts are set, and the number of other points contained in the ε-neighborhood of each data point is calculated as the density value; points with a density value ≥ MinPts are marked as core points; the core points and all points in their ε-neighborhood are assigned to the same initial cluster; the initial clusters are determined by disjoint lookup sets and mutual reachability; noisy clusters with a number of data points lower than the preset minimum sample number are removed, and the remaining clusters are the typical interaction pattern clusters.

[0033] S104: Generate the typical feature distribution parameters for each typical interaction pattern cluster; calculate the arithmetic mean of all vectors within the cluster on each component to obtain the mean vector μ; calculate the covariance between any two different components to obtain the covariance matrix Σ.

[0034] S105: Construct a scheduling prior knowledge base. For each cluster, construct a multivariate Gaussian sampler with μ and Σ as parameters; generate simulated competition feature vectors by sampling random noise vector z (with the same dimension as the resource competition feature vector) from a standard normal distribution, performing Choliski decomposition on Σ to obtain a lower triangular matrix L, and calculating μ + L·z as the simulated competition feature vector; generate a preset number (e.g., 1000) of simulated competition feature vectors for each cluster, and store them together with the sampler as the scheduling prior knowledge base.

[0035] Step S2 (Online Monitoring Phase): S201: Capture multimodal interactive sensing data streams using a sliding time window method; set the time window length to W (e.g., 1000ms) and the window overlap rate to R (e.g., 50%); continuously slide the window while the educational device is running.

[0036] S202: For the current time window, perform event-level segmentation on the sensor data stream within the window to identify each complete interaction event (e.g., a voice command from wake word to end); extract the resource consumption features of each event and convert them into a real-time resource contention feature vector; arrange all real-time vectors within the window in chronological order to obtain a real-time feature sequence; if there is no complete interaction event within the window, skip the window.

[0037] S203: For each real-time resource competition feature vector x in the real-time feature sequence, for each typical interaction mode cluster k, calculate the Mahalanobis distance between x and each simulated vector in the simulated competition feature vector set of that cluster (using the covariance matrix Σk of the cluster as the scale matrix), and take the minimum value dmin(x, k); then, for each cluster k, sum the dmin values ​​of all real-time vectors in the current real-time feature sequence to obtain the cumulative Mahalanobis distance Dk.

[0038] S204: A preset transition probability matrix T is used, where T[i][j] represents the probability of transitioning from pattern cluster i to pattern cluster j, which is obtained by statistically analyzing the cluster transition frequency in offline historical data; using Dk as the observation cost, combined with T, the Viterbi decoding algorithm is used to solve for the hidden state sequence that minimizes the total cost of the entire real-time feature sequence; after decoding, the pattern cluster with the highest occurrence frequency is selected from this sequence as the interaction mode category of the current time window.

[0039] Step S3 (Scheduling Decision and Application): S301: Based on the interaction mode category identified in step S204, query the preset scheduling strategy mapping table to obtain the corresponding resource scheduling instructions (e.g., if it is identified as "multi-person dual-modal concurrent" mode, set the voice engine priority to high, the gesture recognition thread CPU quota to 30%, and the touch response thread frequency to low, etc.).

[0040] S302: Send the scheduling command to the operating system resource manager of the exhibition and education equipment for execution, and complete the adaptive scheduling.

[0041] It should be noted that the core principle of the technical solution of the present invention is to model the multimodal interaction process of the exhibition and education equipment as a probability mapping problem between an observable sequence of resource competition features and an unobservable sequence of interaction mode states.

[0042] Specifically, multimodal interaction events (such as multiple people issuing voice commands simultaneously, concurrent gesture touch, etc.) will trigger competition for device computing resources. This competition can be quantitatively characterized by resource usage characteristics (processor duration, memory access frequency, bandwidth usage, etc.). However, it is difficult to directly determine the complex interaction mode behind a single interaction event (such as "single-person single-modal" or "multi-person bimodal concurrency") from the feature vector of a single interaction event, because the feature distributions of different modes may overlap.

[0043] To address this issue, this invention first employs a density clustering algorithm (DBSCAN) in the offline phase to automatically discover multiple typical interaction pattern clusters from historical data. Each cluster corresponds to a statistical distribution of resource competition feature vectors, and the precise distribution parameters are obtained by calculating the mean vector and covariance matrix. Furthermore, considering the scarcity of samples in high-concurrency scenarios, this invention utilizes a multivariate Gaussian sampler to generate a large number of simulated competition feature vectors based on these distribution parameters, thereby expanding the sample space of each pattern cluster and constructing a comprehensive and reliable scheduling prior knowledge base.

[0044] During the online monitoring phase, this invention continuously captures real-time interactive events using a sliding time window, converting them into a real-time resource competition feature vector sequence. To determine the interaction mode of the current window, this invention designs a dual matching mechanism: First, it calculates the minimum Mahalanobis distance between each real-time vector and the simulated feature vector set of each mode cluster in the knowledge base—this distance metric eliminates the dimensional differences of each feature component at the scale of the covariance matrix, while simultaneously characterizing the correlation structure between vectors, and more accurately reflects the similarity of resource competition modes than Euclidean distance. Then, it accumulates the minimum distances of all vectors within the same window to obtain the cumulative Mahalanobis distance of each mode cluster as the observation cost. Finally, it introduces a Hidden Markov Model, using the transition probability matrix to describe the mode switching pattern between adjacent time windows, and uses the Viterbi decoding algorithm to solve for the hidden state sequence that minimizes the total cost of the entire observation sequence, thereby outputting the interaction mode category of the current time window.

[0045] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0046] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive scheduling method for intelligent educational exhibition equipment based on multimodal interaction, characterized in that, The method includes the following steps: In the baseline learning phase, data from various preset interaction scenarios are collected, and resource occupancy features corresponding to each interaction event are extracted. Based on the resource occupancy features, a clustering algorithm is used to divide the interaction events into several typical interaction pattern clusters, and typical feature distribution parameters for each typical interaction pattern cluster are generated. A simulated feature generator is constructed based on the typical feature distribution parameters to generate simulated competition feature vectors. The simulated feature generators and simulated competition feature vectors corresponding to all typical interaction pattern clusters are summarized and stored to form a scheduling prior knowledge base. In the online monitoring phase, multimodal interactive sensing data streams are captured using a sliding time window approach. Real-time resource contention feature vectors within the current time window are extracted and arranged chronologically to form a real-time feature sequence. Each real-time resource contention feature vector in the real-time feature sequence is matched with the simulated contention feature vector set of each typical interaction mode cluster in the scheduling prior knowledge base, and the cumulative Mahalanobis distance is calculated. Combined with a preset transition probability matrix, the Viterbi decoding algorithm is used to solve for the hidden state sequence that minimizes the total cost of the entire real-time feature sequence. The typical interaction mode cluster with the highest frequency of occurrence is selected from the hidden state sequence as the interaction mode category of the current time window. The specific implementation process of constructing the scheduling prior knowledge base includes: For any typical interaction pattern cluster, the mean vector and covariance matrix of the typical interaction pattern cluster are used as the statistical benchmark for generating simulated competition feature vectors. A multivariate Gaussian sampler is constructed as a simulated feature generator for any typical interaction pattern cluster. The multivariate Gaussian sampler takes the mean vector as the expectation center and the covariance matrix as the covariance structure between the components of the sample resource competition feature vector. The multivariate Gaussian sampler generates a simulated competition feature vector in the following way: it independently samples a random noise vector with the same dimension as the sample resource competition feature vector from the standard normal distribution, performs Choliski decomposition on the covariance matrix to obtain a lower triangular matrix, multiplies the random noise vector with the lower triangular matrix, and then adds it to the mean vector. The result is a simulated competition feature vector. The multivariate Gaussian samplers of all typical interaction pattern clusters and the simulated competition feature vector sets generated by the multivariate Gaussian samplers are summarized and stored to form the scheduling prior knowledge base.

2. The adaptive scheduling method for intelligent educational equipment based on multimodal interaction according to claim 1, characterized in that, The specific implementation process of the baseline learning phase includes: Before the exhibition and education equipment is put into online scheduling and operation, a variety of preset interactive scenario data are collected in an offline environment. The interactive scenario data includes the sensor signal stream generated by different numbers of visitors interacting with the exhibition and education equipment using at least two of the following interactive modes: voice commands, gestures, touch clicks, and eye gaze. Modal separation and time alignment are performed on each sensor signal stream, and resource occupancy features corresponding to each interaction event are extracted. The resource occupancy features include modal type identifier, event duration, cumulative processor occupancy time, memory access frequency, and communication bandwidth occupancy value. Based on the resource occupancy characteristics, a clustering algorithm is used to divide the interactive events into several typical interactive pattern clusters and generate typical feature distribution parameters, wherein each typical interactive pattern cluster corresponds to a sample resource competition feature vector. Generative adversarial networks are used to learn the distribution of sample resource competition feature vectors within each typical interaction pattern cluster, generating simulated competition feature vectors that are homologous to the typical interaction pattern clusters but have expanded sample space, thus forming a scheduling prior knowledge base.

3. The adaptive scheduling method for intelligent educational equipment based on multimodal interaction according to claim 2, characterized in that, The baseline learning phase collects data from various preset interaction scenarios, including: constructing standardized test scripts for four types of scenarios: single-person single-modal, multi-person single-modal, single-person multimodal, and multi-person multimodal. During the interaction between the testers and the educational equipment, the standardized test scripts are driven, and the system resource call logs and interaction response timestamps triggered by each interaction event are recorded synchronously.

4. The adaptive scheduling method for intelligent educational equipment based on multimodal interaction according to claim 2, characterized in that, The method of using clustering algorithms to divide interactive events into several typical interaction pattern clusters is as follows: Each interactive event corresponds to a sample resource competition feature vector as a data point in a high-dimensional space, and all the data points corresponding to interactive events constitute the original dataset. Set a neighborhood radius threshold and a density threshold. For each data point in the original dataset, count the number of other data points (excluding the selected data point) contained in the hypersphere centered on a certain data point and with the neighborhood radius threshold as the radius. Use this number as the neighborhood density value of the hypersphere constructed with the data point as the center. Data points with neighborhood density values ​​greater than or equal to the density threshold are selected and marked as core points; For any core point, all other core points and boundary points located within the neighborhood radius threshold of that core point are assigned to the same initial cluster. Traverse all core points and use the disjoint-set data structure algorithm to merge mutually reachable initial clusters to obtain multiple candidate clusters; Candidate clusters with fewer data points than the preset minimum number of samples are eliminated as noise clusters, and the remaining candidate clusters are the typical interaction pattern clusters.

5. The adaptive scheduling method for intelligent educational equipment based on multimodal interaction according to claim 4, characterized in that, The specific implementation process for generating typical feature distribution parameters includes: The components of the sample resource competition feature vector include: event duration component, processor cumulative occupancy time component, memory access frequency component, and communication bandwidth occupancy value component. The event duration component is the time length between the triggering time and the completion time of the response output of the interactive event; the processor cumulative occupancy time component is the cumulative number of microseconds occupied by the central processing unit and the graphics processing unit during the event duration; the memory access frequency component is the weighted sum of the number of page faults and memory page replacements during the event duration; and the communication bandwidth occupancy component is the total amount of data sent and received by the network interface of the teaching and learning equipment during the event duration. For any typical interaction pattern cluster, calculate the arithmetic mean of the resource competition feature vector of all samples in the typical interaction pattern cluster at each component, and combine the arithmetic mean of all components in a fixed order to form the mean vector of the typical interaction pattern cluster. For any typical interaction pattern cluster, calculate the covariance between any two different components within the typical interaction pattern cluster, and arrange the variance of each component itself and the covariance between different components in matrix form to form the covariance matrix of the typical interaction pattern cluster. The mean vector and the covariance matrix are used together as the typical feature distribution parameters of this typical interaction pattern cluster.

6. The adaptive scheduling method for intelligent educational equipment based on multimodal interaction according to claim 1, characterized in that, The specific implementation process of the online monitoring phase includes: The time window is continuously slid during the online operation of the educational equipment, with a preset time window length and window overlap rate. For the current time window, perform event-level segmentation on the multimodal interactive sensing data stream captured within the current time window to identify each complete interactive event; For any complete interaction event identified, extract the resource consumption features of that complete interaction event and convert them into a real-time resource contention feature vector; All real-time resource contention feature vectors obtained within the current time window are arranged in chronological order of event occurrence to form a real-time feature sequence; if no complete interaction event is detected within the current time window, the current time window is skipped. For any real-time resource competition feature vector in the real-time feature sequence, for any typical interaction mode cluster, calculate the Mahalanobis distance between the real-time resource competition feature vector and each simulated competition feature vector in the simulated competition feature vector set of the typical interaction mode cluster. The Mahalanobis distance is calculated using the covariance matrix of the typical interaction mode cluster as the scaling matrix, and the minimum value of the Mahalanobis distance is selected as the minimum Mahalanobis distance between the real-time resource competition feature vector and the typical interaction mode cluster. For any typical interaction pattern cluster, the minimum Mahalanobis distances corresponding to all real-time resource competition feature vectors in the current real-time feature sequence are accumulated to obtain the cumulative Mahalanobis distance of that typical interaction pattern cluster at the real-time feature sequence level.

7. The adaptive scheduling method for intelligent educational equipment based on multimodal interaction according to claim 6, characterized in that, The specific implementation process of the online monitoring phase also includes: A transition probability matrix is ​​preset, and each element in the transition probability matrix represents the transition probability of transitioning from one typical interaction mode cluster to another typical interaction mode cluster between adjacent time windows. The transition probability is obtained by statistically analyzing the transition frequency of typical interaction modes in offline collected historical interaction scene data. Using the cumulative Mahalanobis distance corresponding to each typical interaction pattern cluster as the observation cost, and combining it with the transition probability matrix, the Viterbi decoding algorithm is used to solve for the hidden state sequence that minimizes the total cost of the entire real-time feature sequence. Each hidden state in the hidden state sequence corresponds to a typical interaction pattern cluster. From the hidden state sequence obtained by decoding, the cluster of typical interaction patterns that appears most frequently is selected as the interaction pattern category for the current time window.

8. An adaptive scheduling system that implements the adaptive scheduling method for intelligent educational equipment based on multimodal interaction as described in any one of claims 1-7, characterized in that, The system includes: At least one processor; At least one memory, wherein a computer program is stored; When the computer program is executed by the at least one processor, it implements the steps of the adaptive scheduling method for smart educational equipment based on multimodal interaction.

9. The adaptive scheduling system according to claim 8, characterized in that, The system also includes: A multimodal interaction platform is used to schedule one or more computer programs, which, when executed by one or more processors, enable the multimodal interaction platform to implement the steps of the adaptive scheduling method for smart educational equipment based on multimodal interaction.

Images