AI-based user desktop interactive control device and method

Abstract

The present invention relates to the field of communication technology, and in particular to an AI-based user desktop interaction control device and method thereof, wherein the method comprises the following steps: obtaining an original interaction data sequence, performing outlier detection and timestamp sorting preprocessing on the data, and generating an ordered original interaction data sequence; performing time domain and frequency domain transformation on the original interaction data sequence, and extracting multi-scale frequency features to construct a feature vector space for obtaining feature data, wherein the feature data is used to characterize the time-frequency characteristics of the interaction signal; performing self-attention mechanism processing on the feature data, performing weighted calculation through a multi-head attention weight matrix and position encoding parameters, and obtaining context-aware weighted feature data; inputting the weighted feature data into a long short-term memory network, combining the forget gate, input gate and output gate parameters of a gated recurrent unit, and outputting time-series-associated interaction behavior prediction parameters.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to an AI-based user desktop interaction control device and method thereof. Background Art

[0002] With the development of computer technology, the way users interact with desktop devices has gradually evolved from traditional mouse and keyboard operations to intelligent and natural ones.

[0003] Currently, mainstream desktop interactive control technologies rely primarily on preset instruction sets or simple pattern matching, making it difficult to effectively understand users' complex operational intent. This leads to problems such as rigid interactive responses and insufficient adaptability. Furthermore, traditional interactive methods have significant limitations in data processing: Firstly, processing of user operational data (such as click coordinates and sliding trajectories) often remains limited to raw time-domain analysis, lacking the ability to mine signal frequency-domain features. This inability to fully characterize the time-frequency characteristics of interactive behavior. Secondly, traditional models struggle to capture long-range dependencies in highly sequential interactive data. For example, when identifying a user's underlying intent within a continuous sequence of operations, inadequate temporal feature extraction often leads to misjudgments.

[0004] When it comes to intent recognition and decision-making control, existing technologies typically rely on fixed rules or simple machine learning algorithms, failing to dynamically optimize strategies based on user behavior. For example, when user habits change or new interaction scenarios emerge, traditional systems struggle to adapt their response modes through autonomous learning, resulting in decreased interaction efficiency. Furthermore, most solutions lack closed-loop optimization mechanisms based on user feedback, making them unable to continuously improve the interaction experience in real-world use and meeting the demands of scenarios such as intelligent office work and efficient human-computer collaboration. Summary of the Invention

[0005] Based on this, it is necessary to provide an AI-based user desktop interaction control device and method to address the above technical problems, effectively solve the problems of shallow data processing, rigid intention understanding, and inability to learn strategies independently in traditional technologies, and improve the interaction efficiency and user experience in complex scenarios.

[0006] The present invention provides an AI-based user desktop interaction control method, the method comprising:

[0007] Obtaining an original interaction data sequence, performing outlier detection and timestamp sorting preprocessing on the data, and generating an ordered original interaction data sequence;

[0008] The original interaction data sequence is transformed into the time domain and the frequency domain, and multi-scale frequency features are extracted to construct a feature vector space to obtain feature data, wherein the feature data is used to characterize the time-frequency characteristics of the interaction signal;

[0009] Processing the feature data with a self-attention mechanism, performing weighted calculations using a multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data;

[0010] Inputting the weighted feature data into a long short-term memory network, combining the forget gate, input gate, and output gate parameters of the gated recurrent unit, and outputting a time-series-associated interactive behavior prediction parameter;

[0011] Based on the preset intent classifier parameters, the interaction behavior prediction parameters are mapped to the predefined intent space. The intent probability distribution is calculated using the softmax function, and the category corresponding to the maximum probability is taken as the intent category data.

[0012] Calling historical operation data to build a decision process model, combining the intention category data and the current system state, and calculating the optimal control parameter set through a reinforcement learning algorithm;

[0013] The desktop interactive control action is executed according to the optimal control parameter set, and the policy gradient is updated according to user feedback to continuously optimize the interactive response strategy.

[0014] In one embodiment, obtaining the original interaction data sequence, performing outlier detection and timestamp sorting preprocessing on the data, and generating an ordered original interaction data sequence includes:

[0015] Collect user operation data on desktop devices, including click coordinates, sliding trajectories, and pressing time, to construct a three-dimensional spatiotemporal data matrix;

[0016] Perform outlier detection on the collected raw data, calculate the deviation between the data point and the local mean, and eliminate abnormal data whose deviation exceeds the preset threshold;

[0017] Sort the processed data by timestamp to generate an ordered sequence of original interaction data.

[0018] In one embodiment, performing time domain and frequency domain transformation on the original interaction data sequence includes:

[0019] The original interaction data sequence is segmented and the mean, variance and standard deviation of each data segment are calculated within the time window;

[0020] Apply the difference operator to calculate the rate of change between adjacent data points and generate a first-order difference sequence;

[0021] Construct a sliding time window, calculate the cumulative sum and cumulative difference of the data in the window, and extract time domain features.

[0022] In one embodiment, extracting multi-scale frequency features to construct a feature vector space includes:

[0023] Calculate frequency characteristics at different scales, including peak frequency, center frequency, and bandwidth;

[0024] Construct a feature correlation matrix to analyze the correlation between different frequency features;

[0025] The principal component analysis algorithm is applied to project the high-dimensional feature vectors into a low-dimensional space to generate a feature vector space.

[0026] In one embodiment, the self-attention mechanism is performed on the feature data, and weighted calculation is performed using a multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data, including:

[0027] Construct a feature similarity matrix and calculate the cosine similarity between feature vectors;

[0028] Apply the weight normalization algorithm to convert the similarity matrix into the attention weight matrix;

[0029] The feature data is weighted and summed according to the attention weight matrix to generate context-aware weighted feature data.

[0030] In one embodiment, the step of inputting the weighted feature data into a long short-term memory network, combining the forget gate, input gate, and output gate parameters of a gated recurrent unit, and outputting temporally associated interaction behavior prediction parameters includes:

[0031] Normalize the weighted feature data and map the data to the [0, 1] interval;

[0032] Build a time series prediction model and calculate the hidden state of the current time step;

[0033] The forget gate, input gate, and output gate mechanisms are applied to control the flow and update of information and generate time-series-related interactive behavior prediction parameters.

[0034] In one embodiment, mapping the interaction behavior prediction parameters to a predefined intent space based on preset intent classifier parameters includes:

[0035] Load a pre-trained intent classifier model, which includes a multi-layer fully connected neural network and activation function parameters;

[0036] Performing dimension alignment processing on the interaction behavior prediction parameters to match the dimension of the classifier input layer;

[0037] Through the forward propagation algorithm, the processed prediction parameters are input into the classifier, and the intention space mapping vector is calculated using the preset weight matrix and bias parameters.

[0038] In one embodiment, the calculation of the intent probability distribution using the softmax function and taking the category corresponding to the maximum probability as the intent category data include:

[0039] Input the intent space mapping vector into the softmax function to calculate the index value of each predefined intent category;

[0040] Normalize all index values ​​to generate a probability distribution vector in the range [0, 1];

[0041] Traverse the probability distribution vector, extract the index position with the largest probability value, and match it with the predefined intent category mapping table to obtain the intent category data.

[0042] In one embodiment, the process of building a decision-making process model based on historical operation data, combining the intention category data and the current system state, and calculating the optimal control parameter set through a reinforcement learning algorithm includes:

[0043] Extract the operation sequence that matches the current intention from the historical operation database and construct a state-action pair training set;

[0044] Define the reinforcement learning environment state space and encode the current system state parameters and intention category data into the environment state vector;

[0045] The optimal action-value function is iteratively calculated based on the state space and training set to generate a set of control parameters.

[0046] The present invention further provides an AI-based user desktop interaction control device, which is applied to the AI-based user desktop interaction control method described in any of the above embodiments, comprising:

[0047] A first data processing module is used to obtain a raw interaction data sequence, perform outlier detection and timestamp sorting preprocessing on the data, and generate an ordered raw interaction data sequence;

[0048] A second data processing module is used to transform the original interaction data sequence into the time domain and the frequency domain, and extract multi-scale frequency features to construct a feature vector space to obtain feature data, wherein the feature data is used to characterize the time-frequency characteristics of the interaction signal;

[0049] A third data processing module is used to perform self-attention mechanism processing on the feature data, and perform weighted calculation through a multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data;

[0050] An input-output module, configured to input the weighted feature data into a long short-term memory network, combine the forget gate, input gate, and output gate parameters of the gated recurrent unit, and output a time-series-associated interactive behavior prediction parameter;

[0051] The mapping processing module is used to map the interaction behavior prediction parameters to the predefined intent space based on the preset intent classifier parameters, calculate the intent probability distribution through the softmax function, and take the category corresponding to the maximum probability as the intent category data;

[0052] A calculation and decision-making module is used to call historical operation data to build a decision-making process model, combine the intention category data and the current system state, and calculate the optimal control parameter set through a reinforcement learning algorithm;

[0053] The optimization module is used to execute desktop interactive control actions according to the optimal control parameter set, and to perform policy gradient updates based on user feedback to continuously optimize the interactive response strategy.

[0054] The above-mentioned AI-based user desktop interaction control device and method obtain the original interaction data sequence and perform outlier detection and timestamp sorting preprocessing to generate ordered data to ensure the reliability of the interaction signal and eliminate interference such as equipment noise; then the original data is transformed in the time domain and frequency domain and multi-scale frequency features are extracted to construct a feature vector space, thereby strengthening the representation of the time-frequency characteristics of the interaction behavior and capturing dynamic characteristics and periodic laws; then the feature data is processed by the self-attention mechanism, combined with the prediction parameters of the time series association of the long-term and short-term memory network output, to enhance the modeling of the long-distance dependency relationship of the interaction data, focus on key operations and memorize historical patterns; then based on the intent classifier and softmax function, the prediction parameters are mapped to the intent space to achieve accurate intent recognition; then the historical operation data is called in combination with the current intent and system status, and the optimal control parameter set is calculated through the reinforcement learning algorithm; finally, the control action is executed according to the optimal parameters and the policy gradient is updated based on user feedback to form a closed-loop optimization mechanism, which effectively solves the problems of shallow data processing, rigid intent understanding, and inability of strategy to learn independently in traditional technologies, thereby improving the interaction efficiency and user experience in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 Flowchart of the AI-based user desktop interaction control method provided by the present invention;

[0057] Figure 2 This is a module diagram of the AI-based user desktop interaction control device provided by the present invention. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0059] like Figure 1 As shown, an AI-based user desktop interaction control method includes the following steps:

[0060] Obtain the original interaction data sequence, perform outlier detection and timestamp sorting preprocessing on the data, and generate an ordered original interaction data sequence;

[0061] The original interaction data sequence is transformed into the time domain and frequency domain, and multi-scale frequency features are extracted to construct a feature vector space to obtain feature data. The feature data is used to characterize the time-frequency characteristics of the interaction signal.

[0062] The feature data is processed by the self-attention mechanism, and weighted calculation is performed through the multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data;

[0063] The weighted feature data is input into the long short-term memory network, and the forget gate, input gate, and output gate parameters of the gated recurrent unit are combined to output the time-series-related interaction behavior prediction parameters;

[0064] Based on the preset intent classifier parameters, the interaction behavior prediction parameters are mapped to the predefined intent space. The intent probability distribution is calculated using the softmax function, and the category corresponding to the maximum probability is taken as the intent category data.

[0065] Calling historical operation data to build a decision-making process model, combining intent category data and current system status, and calculating the optimal control parameter set through reinforcement learning algorithms;

[0066] Execute desktop interactive control actions according to the optimal control parameter set, and perform policy gradient updates based on user feedback to continuously optimize the interactive response strategy.

[0067] The above-mentioned AI-based user desktop interaction control method obtains the original interaction data sequence and performs outlier detection and timestamp sorting preprocessing to generate ordered data to ensure the reliability of the interaction signal and eliminate interference such as device noise; then the original data is transformed in the time domain and frequency domain and multi-scale frequency features are extracted to construct a feature vector space, thereby strengthening the representation of the time-frequency characteristics of the interaction behavior and capturing dynamic characteristics and periodic laws; then the feature data is processed by the self-attention mechanism, combined with the prediction parameters of the time series correlation of the long short-term memory network output, to enhance the modeling of the long-distance dependency relationship of the interaction data, focus on key operations and memorize historical patterns; then based on the intent classifier and softmax function, the prediction parameters are mapped to the intent space to achieve accurate intent recognition; then the historical operation data is called in combination with the current intent and system status, and the optimal control parameter set is calculated through the reinforcement learning algorithm; finally, the control action is executed according to the optimal parameters and the policy gradient is updated based on user feedback to form a closed-loop optimization mechanism, which effectively solves the problems of shallow data processing, rigid intent understanding, and inability of strategy to learn independently in traditional technologies, thereby improving the interaction efficiency and user experience in complex scenarios.

[0068] In one embodiment, obtaining a raw interaction data sequence, performing outlier detection and timestamp sorting preprocessing on the data, and generating an ordered raw interaction data sequence includes the following steps:

[0069] Collect user operation data on desktop devices, including click coordinates, sliding trajectories and pressing time, and build a three-dimensional spatiotemporal data matrix.

[0070] Specifically, the acquisition devices include a mouse, a touch screen, etc., and the sampling frequency is set to 100 Hz (i.e., 100 data points per second).

[0071] Constructing a three-dimensional spatiotemporal data matrix ,in is the number of samples, and the three dimensions are click coordinates and compression time , is the length of the time series.

[0072] If the user clicks three times in a row, the collected data can be expressed as , corresponding to the coordinates and pressing time of the three time points.

[0073] The collected raw data is detected for outliers. By calculating the deviation between the data point and the local mean, the abnormal data with deviation exceeding the preset threshold is eliminated.

[0074] Specifically, for the data point , calculate its difference with the local area The mean of the points deviation.

[0075] ,in

[0076] in, For the data points (such as click coordinates, press time, etc.), For data points The local neighborhood of The mean of the points, is the number of points in the local neighborhood (such as the number of nearest neighbors), For data points of The set of nearest neighbor points, For data points Deviation from the local mean.

[0077] In this embodiment, the preset threshold ,in, is the standard deviation of the local neighborhood data, when , remove the point.

[0078] If a click coordinate , whose local neighborhood mean is , standard deviation ,but , far exceeding , which is determined to be an outlier.

[0079] Sort the processed data by timestamp to generate an ordered sequence of original interaction data.

[0080] Specifically, by data collection timestamp Arrange in ascending order to generate an ordered sequence ,in .

[0081] In one embodiment, the original interaction data sequence is transformed into the time domain and the frequency domain, including the following steps:

[0082] The original interaction data sequence is segmented and the mean, variance and standard deviation within the time window of each segment of data are calculated.

[0083] Specifically, set the time window , with an overlap rate of 50%, the sequence is divided into M segments, and for each segment, the following is calculated:

[0084] Mean: ,in, The first data points, is the number of data points in the window, is the mean of the data in the window.

[0085] variance:

[0086] Standard Deviation: ,in, is the standard deviation

[0087] Apply the difference operator to calculate the rate of change between adjacent data points and generate a first-order difference series.

[0088] Specifically, the first-order difference ,in, is the rate of change of adjacent data points.

[0089] Construct a sliding time window, calculate the cumulative sum and cumulative difference of the data in the window, and extract time domain features.

[0090] Specifically, the cumulative sum is calculated within the sliding window and cumulative difference

[0091] In one embodiment, extracting multi-scale frequency features to construct a feature vector space includes the following steps:

[0092] Calculate frequency characteristics at different scales, including peak frequency, center frequency, and bandwidth.

[0093] Specifically, we define short-scale (100ms), medium-scale (500ms), and long-scale (1s) windows to extract the data at each scale. 、 、 , forming the feature vector:

[0094] , dimension is 3 3=9 dimensions.

[0095] Perform fast Fourier transform on the time domain sequence to obtain frequency domain representation .

[0096] Power spectral density ,in, is the frequency variable, is the time variable, is the length of the time domain series, that is, the total number of data points in the time series, is the first The signal value at a time point, which reflects the original interaction data at time The specific characteristics of the Is an imaginary unit.

[0097] Peak frequency:

[0098] Center Frequency:

[0099] bandwidth: ,in, is the frequency at which the power drops to half of its peak value.

[0100] Construct a feature correlation matrix to analyze the correlation between features of different frequencies.

[0101] Specifically, construct a correlation matrix ,in , analyze the linear correlation between features, such as the correlation between peak frequency and center frequency.

[0102] in, Features and features The correlation coefficient of and For different frequency characteristics, Features and features The covariance of Features and features The standard deviation of .

[0103] The principal component analysis algorithm is applied to project the high-dimensional feature vectors into a low-dimensional space to generate a feature vector space.

[0104] Specifically, perform principal component analysis on the eigenvector, retain 95% of the variance, and use the dimensionality reduction formula: ,in For the front The principal component matrix is ​​decomposed by eigenvalue get, satisfy

[0105] If the original feature is 100-dimensional, it may be 20-dimensional after dimensionality reduction.

[0106] In one embodiment, a self-attention mechanism is applied to feature data, and weighted calculation is performed using a multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data, including the following steps:

[0107] Construct a feature similarity matrix and calculate the cosine similarity between feature vectors.

[0108] Specifically, construct the query matrix , key matrix , value matrix , the dimensions are ,in, is the feature dimension, is the sequence length.

[0109] Cosine similarity matrix:

[0110] in, 、 is a matrix No. row, matrix No. row vector, For the Hedi The cosine similarity of the feature vectors.

[0111] Apply the weight normalization algorithm to convert the similarity matrix into the attention weight matrix.

[0112] Specifically, apply the softmax function: , get the attention weight matrix ,in .

[0113] in, For the The feature pair The attention weight of each feature.

[0114] The feature data is weighted and summed according to the attention weight matrix to generate context-aware weighted feature data.

[0115] Specific, context-aware features: , integrating the feature dependencies at each position in the sequence.

[0116] in, is the weighted feature after integrating the sequence dependency.

[0117] Set up 8 heads, each head calculates independently , and finally concatenated into

[0118] In one embodiment, weighted feature data is input into a long short-term memory network, and combined with the forget gate, input gate, and output gate parameters of a gated recurrent unit to output a time-series-associated interactive behavior prediction parameter, including the following steps:

[0119] Normalize the weighted feature data and map the data to the [0, 1] interval.

[0120] Specifically, we use min-max normalization to map weighted feature data to the range [0, 1]. This involves calculating the minimum and maximum values ​​of the feature data, min_val, and max_val, and then performing the multiplication (x - min_val) / (max_val - min_val) for each data point x. For example, if a feature value ranges from [-10, 20], it will be normalized to [0, 1]. This ensures uniform data scaling and prevents LSTM training instability caused by large variance in input values.

[0121] Among them, x is the original eigenvalue, min_val and max_val are the minimum and maximum values ​​of the feature respectively.

[0122] Build a time series prediction model and calculate the hidden state of the current time step.

[0123] Specifically, initialize the hidden layer dimension of the LSTM network (e.g., 128 dimensions) and set the sequence length (e.g., the interaction data of the last 50 time steps).

[0124] For the current time step t, the normalized weighted feature data is used as input, combined with the hidden state h(t-1) and cell state c(t-1) of the previous time step, and the current hidden state h(t) is calculated through the LSTM unit.

[0125] The hidden state calculation process captures the temporal dependencies of interaction data, such as the user's continuous click and drag action sequence patterns.

[0126] The forget gate, input gate, and output gate mechanisms are applied to control the flow and update of information and generate time-series-related interactive behavior prediction parameters.

[0127] Specifically, the forget gate generates a weight between 0 and 1 based on the current input and the previous hidden state, deciding which historical information to discard. For example, if the user has not input any keyboard data for a long time, the forget gate will reduce the weight of the earlier keyboard operation data.

[0128] Input gate: Filters the valid information in the current input, combines it with the candidate cell state, and updates the cell state c(t). For example, when a quick mouse slide is detected, the input gate will strengthen the feature weight of the action.

[0129] Output gate: Based on the updated cell state, it generates the hidden state h(t) for the current time step, which is the interaction behavior prediction parameter. This parameter includes the temporal correlation prediction of the user's next action, such as the coordinate range of the mouse click or the type of keyboard input.

[0130] Among them, h(t) is the hidden state of the current time step (interaction behavior prediction parameter), and c(t) is the cell state of the current time step.

[0131] In one embodiment, based on preset intent classifier parameters, mapping interaction behavior prediction parameters to a predefined intent space includes the following steps:

[0132] Load the pre-trained intent classifier model, which contains a multi-layer fully connected neural network and activation function parameters.

[0133] Specifically, the pre-trained model uses a three-layer fully connected neural network structure: the input layer dimension is consistent with the prediction parameter dimension of the LSTM output (for example, 128 dimensions), the middle layer contains 256 neurons and uses the ReLU activation function, and the output layer dimension matches the number of predefined intent categories (for example, 10 categories of intent).

[0134] The model is trained using historical interaction data, which includes user operation sequences and their corresponding true intent labels (such as "copy text", "switch window", "save file", etc.).

[0135] The interaction behavior prediction parameters are dimensionally aligned to match the classifier input layer dimension.

[0136] Specifically, if the prediction parameter dimension of the LSTM output (e.g., 128 dimensions) does not match the classifier input layer dimension (e.g., 256 dimensions), the following method is used to handle it:

[0137] If the output dimension is lower than the input dimension, zero is added to the feature vector or the dimension is increased through linear transformation of the fully connected layer.

[0138] If the output dimension is higher than the input dimension, use principal component analysis (PCA) to reduce the dimension or directly extract the key feature dimension to ensure that it is consistent with the input layer dimension.

[0139] Through the forward propagation algorithm, the processed prediction parameters are input into the classifier, and the intention space mapping vector is calculated using the preset weight matrix and bias parameters.

[0140] Specifically, the dimensionally aligned prediction parameters are input into the classifier:

[0141] The first fully connected layer calculates W1×input+b1 and outputs nonlinear features through the ReLU activation function.

[0142] The second fully connected layer calculates W2×hidden1+b2 in the same way and activates it again through ReLU.

[0143] The output layer calculates W3×hidden2+b3 to generate a mapping vector with the same dimension as the number of intent categories (e.g., 10 dimensions), where each element corresponds to the raw score (logits) of an intent category.

[0144] Among them, W is the weight matrix, b is the bias vector, and input is the input feature (such as the prediction parameter output by LSTM).

[0145] In one embodiment, the intent probability distribution is calculated using a softmax function, and the category corresponding to the maximum probability is taken as the intent category data, including the following steps:

[0146] The intent space mapping vector is input into the softmax function to calculate the index value of each predefined intent category.

[0147] Specifically, the intent space mapping vector (such as [2.1, -0.5, 3.2, ...]) is input into the softmax function, and the exponential value exp(logit) is calculated for each element. For example, the logarithmic value 3.2 is calculated as exp(3.2)≈24.53.

[0148] Sum all the exponents and divide each exponent by the sum to get a probability distribution vector in the range [0, 1]. For example, if the total exponent sum is 50 and an element has an exponent of 24.53, then the corresponding probability is 24.53 / 50≈0.49.

[0149] All exponential values ​​are normalized to generate a probability distribution vector in the range [0, 1].

[0150] Specifically, the predefined intent category mapping table adopts a dictionary structure, such as {0: "copy text", 1: "open file", 2: "save document", ...}.

[0151] Traverse the probability distribution vector, extract the index position with the largest probability value, and match it with the predefined intent category mapping table to obtain the intent category data.

[0152] Specifically, the probability distribution vector is traversed to find the element index with the highest probability value (for example, index 2 corresponds to a probability of 0.49). This is then matched against the mapping table to obtain the intent category data (for example, "Save Document"). This process converts numerical features into semantic intent, providing decision-making basis for subsequent desktop control actions.

[0153] In one embodiment, historical operation data is used to build a decision process model, and the optimal control parameter set is calculated using a reinforcement learning algorithm in combination with the intent category data and the current system state, including the following steps:

[0154] Extract the action sequence that matches the current intention from the historical action database and construct a state-action pair training set.

[0155] Specifically, the operation sequence that matches the current intention is extracted from the history database. For example, when the intention is "copy text", the mouse drag, keyboard Ctrl+C and other actions under the intention in the history are extracted to form a state-action pair. , where the status include:

[0156] System status: current window position , mouse position , keyboard status (key pressed).

[0157] Intent encoding: a one-hot vector of the intent category (e.g., the corresponding position in 10 dimensions is 1).

[0158] action :Such as mouse click coordinates , key input, etc., parameterized as vector (Click type: left click / right click).

[0159] Define the reinforcement learning environment state space and encode the current system state parameters and intention category data into the environment state vector.

[0160] Specifically, the state space : Concatenate system state and intent encoding into a 100-dimensional vector

[0161]

[0162] Action space A: a set of discrete actions, such as {left click, right click, drag, keyboard input}, where each action corresponds to a set of control parameters.

[0163] The optimal action-value function is iteratively calculated based on the state space and training set to generate a set of control parameters.

[0164] Specifically, iterative updates

[0165]

[0166] in, Status Next action The value function of is the current state (system state + intention encoding), For the current action, For the next state, For the next action, (learning rate), (discount factor), To perform an action The immediate reward after the intention is executed (e.g. +1 for correct execution and -1 for error).

[0167] Generate control parameter set: through greedy strategy The optimal action is selected, and its parameters such as click coordinates and drag distance constitute the control parameter set.

[0168] like Figure 2 As shown, an AI-based user desktop interaction control device is applied to the AI-based user desktop interaction control method of any of the above embodiments, including a first data processing module, a second data processing module, a third data processing module, an input and output module, a mapping processing module, a calculation and decision module, and an optimization module:

[0169] The first data processing module is used to obtain the original interaction data sequence, perform outlier detection and timestamp sorting preprocessing on the data, and generate an ordered original interaction data sequence;

[0170] The second data processing module is used to transform the original interaction data sequence into the time domain and frequency domain, and extract multi-scale frequency features to construct a feature vector space to obtain feature data. The feature data is used to characterize the time-frequency characteristics of the interaction signal;

[0171] The third data processing module is used to process the feature data through a self-attention mechanism, and perform weighted calculations using a multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data;

[0172] The input-output module is used to input weighted feature data into the long short-term memory network, combine the forget gate, input gate, and output gate parameters of the gated recurrent unit, and output the time-series-related interaction behavior prediction parameters;

[0173] The mapping processing module is used to map the interaction behavior prediction parameters to the predefined intent space based on the preset intent classifier parameters, calculate the intent probability distribution through the softmax function, and take the category corresponding to the maximum probability as the intent category data;

[0174] The calculation and decision-making module is used to call historical operation data to build a decision-making process model, combine the intention category data and the current system status, and calculate the optimal control parameter set through the reinforcement learning algorithm;

[0175] The optimization module is used to execute desktop interactive control actions according to the optimal control parameter set, and to perform policy gradient updates based on user feedback to continuously optimize the interactive response strategy.

[0176] The above-mentioned AI-based user desktop interaction control device obtains the original interaction data sequence and performs outlier detection and timestamp sorting preprocessing to generate ordered data to ensure the reliability of the interaction signal and eliminate interference such as equipment noise; then the original data is transformed in the time domain and frequency domain and multi-scale frequency features are extracted to construct a feature vector space, thereby strengthening the representation of the time-frequency characteristics of the interaction behavior and capturing dynamic characteristics and periodic laws; then the feature data is processed by the self-attention mechanism, combined with the prediction parameters of the time series correlation of the long short-term memory network output, to enhance the modeling of the long-distance dependency relationship of the interaction data, focus on key operations and memorize historical patterns; then based on the intent classifier and softmax function, the prediction parameters are mapped to the intent space to achieve accurate intent recognition; then the historical operation data is called and combined with the current intent and system status, and the optimal control parameter set is calculated through the reinforcement learning algorithm; finally, the control action is executed according to the optimal parameters and the policy gradient is updated according to user feedback to form a closed-loop optimization mechanism, which effectively solves the problems of shallow data processing, rigid intent understanding, and inability of strategy to learn independently in traditional technologies, thereby improving the interaction efficiency and user experience in complex scenarios.

[0177] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The above-described embodiments merely illustrate several embodiments of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A user desktop interactive control method based on AI, characterized in that: The method comprises: Obtaining an original interaction data sequence, performing outlier detection and timestamp sorting preprocessing on the data, and generating an ordered original interaction data sequence; The original interaction data sequence is transformed into the time domain and the frequency domain, and multi-scale frequency features are extracted to construct a feature vector space to obtain feature data, wherein the feature data is used to characterize the time-frequency characteristics of the interaction signal; Processing the feature data with a self-attention mechanism, performing weighted calculations using a multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data; Inputting the weighted feature data into a long short-term memory network, combining the forget gate, input gate, and output gate parameters of the gated recurrent unit, and outputting a time-series-associated interactive behavior prediction parameter; Based on the preset intent classifier parameters, the interaction behavior prediction parameters are mapped to the predefined intent space. The intent probability distribution is calculated using the softmax function, and the category corresponding to the maximum probability is taken as the intent category data. Calling historical operation data to build a decision process model, combining the intention category data and the current system state, and calculating the optimal control parameter set through a reinforcement learning algorithm; Executing desktop interactive control actions according to the optimal control parameter set, and performing policy gradient updates based on user feedback to continuously optimize the interactive response strategy; The step of obtaining the original interaction data sequence, performing outlier detection and timestamp sorting preprocessing on the data, and generating an ordered original interaction data sequence includes: Collect user operation data on desktop devices, including click coordinates, sliding trajectories, and pressing time, to construct a three-dimensional spatiotemporal data matrix; Perform outlier detection on the collected raw data, calculate the deviation between the data point and the local mean, and eliminate abnormal data whose deviation exceeds the preset threshold; Sort the processed data by timestamp to generate an ordered sequence of original interaction data; Mapping the interaction behavior prediction parameters to the predefined intent space based on the preset intent classifier parameters includes: Load a pre-trained intent classifier model, which includes a multi-layer fully connected neural network and activation function parameters; Performing dimension alignment processing on the interaction behavior prediction parameters to match the dimension of the classifier input layer; Through the forward propagation algorithm, the processed prediction parameters are input into the classifier, and the intention space mapping vector is calculated using the preset weight matrix and bias parameters.

2. The AI-based user desktop interaction control method according to claim 1, characterized in that: The performing time domain and frequency domain transformation on the original interaction data sequence includes: The original interaction data sequence is segmented and the mean, variance and standard deviation of each data segment are calculated within the time window; Apply the difference operator to calculate the rate of change between adjacent data points and generate a first-order difference sequence; Construct a sliding time window, calculate the cumulative sum and cumulative difference of the data in the window, and extract time domain features.

3. The AI-based user desktop interaction control method according to claim 2, characterized in that: The extracting multi-scale frequency features to construct a feature vector space includes: Calculate frequency characteristics at different scales, including peak frequency, center frequency, and bandwidth; Construct a feature correlation matrix to analyze the correlation between different frequency features; The principal component analysis algorithm is applied to project the high-dimensional feature vectors into a low-dimensional space to generate a feature vector space.

4. The AI-based user desktop interaction control method according to claim 3, characterized in that: The self-attention mechanism is performed on the feature data, and weighted calculation is performed through a multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data, including: Construct a feature similarity matrix and calculate the cosine similarity between feature vectors; Apply the weight normalization algorithm to convert the similarity matrix into the attention weight matrix; The feature data is weighted and summed according to the attention weight matrix to generate context-aware weighted feature data.

5. The AI-based user desktop interaction control method according to claim 4, characterized in that: The step of inputting the weighted feature data into a long short-term memory network, combining the forget gate, input gate, and output gate parameters of a gated recurrent unit, and outputting time-series-associated interaction behavior prediction parameters includes: Normalize the weighted feature data and map the data to the [0, 1] interval; Build a time series prediction model and calculate the hidden state of the current time step; The forget gate, input gate, and output gate mechanisms are applied to control the flow and update of information and generate time-series-related interactive behavior prediction parameters.

6. The AI-based user desktop interaction control method according to claim 5, characterized in that: The intent probability distribution is calculated by the softmax function, and the category corresponding to the maximum probability is taken as the intent category data, including: Input the intent space mapping vector into the softmax function to calculate the index value of each predefined intent category; Normalize all index values ​​to generate a probability distribution vector in the range [0, 1]; Traverse the probability distribution vector, extract the index position with the largest probability value, and match it with the predefined intent category mapping table to obtain the intent category data.

7. The AI-based user desktop interaction control method according to claim 6, characterized in that: The calling of historical operation data to construct a decision process model, combining the intention category data and the current system state, and calculating the optimal control parameter set through a reinforcement learning algorithm include: Extract the operation sequence that matches the current intention from the historical operation database and construct a state-action pair training set; Define the reinforcement learning environment state space and encode the current system state parameters and intention category data into the environment state vector; The optimal action-value function is iteratively calculated based on the state space and training set to generate a set of control parameters.

8. An AI-based user desktop interaction control device, applied to the AI-based user desktop interaction control method according to any one of claims 1 to 7, characterized in that: include: A first data processing module is used to obtain a raw interaction data sequence, perform outlier detection and timestamp sorting preprocessing on the data, and generate an ordered raw interaction data sequence; A second data processing module is used to transform the original interaction data sequence into the time domain and the frequency domain, and extract multi-scale frequency features to construct a feature vector space to obtain feature data, wherein the feature data is used to characterize the time-frequency characteristics of the interaction signal; A third data processing module is used to perform self-attention mechanism processing on the feature data, and perform weighted calculation through a multi-head attention weight matrix and position encoding parameters to obtain context-aware weighted feature data; An input-output module, configured to input the weighted feature data into a long short-term memory network, combine the forget gate, input gate, and output gate parameters of the gated recurrent unit, and output a time-series-associated interactive behavior prediction parameter; The mapping processing module is used to map the interaction behavior prediction parameters to the predefined intent space based on the preset intent classifier parameters, calculate the intent probability distribution through the softmax function, and take the category corresponding to the maximum probability as the intent category data; A calculation and decision-making module is used to call historical operation data to build a decision-making process model, combine the intention category data and the current system state, and calculate the optimal control parameter set through a reinforcement learning algorithm; The optimization module is used to execute desktop interactive control actions according to the optimal control parameter set, and to perform policy gradient updates based on user feedback to continuously optimize the interactive response strategy.

Images