A personalized adaptive interaction method, apparatus and system

Abstract

The application discloses a kind of personalized adaptive interaction methods, by collecting user voice, touch trajectory data and portrait information;Dialect enhancement and the text of recognition are carried out to voice data;Touch trajectory feature is extracted and is fused into joint feature vector with portrait;Analysis touch mode;Based on portrait and touch mode, adjust speech rate, UI font size and touch feedback strength using transfer learning;According to text, joint feature vector and the dynamic adjustment of instruction response of adjusted parameter interface;Evaluate feedback effect, trigger adaptive process to realize dynamic learning adjustment.The method improves dialect recognition accuracy by end-side distillation Whisper model combined with multi-modal dialect corpus, accurately identifies touch mode by touch trajectory processing combined with DBSCAN clustering, dynamically adjusts interaction parameters based on transfer learning and meta-learning, continuously optimizes through feedback evaluation mechanism, and realizes lightweight deployment by end-side distillation technology.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human-computer interaction, and particularly relates to a personalized adaptive interaction method, device and system. BACKGROUND

[0002] With the rapid development of information technology, human-computer interaction technology has become the core link connecting users and various intelligent devices, and is widely used in many fields such as smart phones, smart homes, intelligent vehicle systems, etc. The pros and cons of the user experience directly affect the popularity of the device and the user's use stickiness. Under this background, developing human-computer interaction systems that are more in line with user needs and more intelligent has become a key direction to improve product competitiveness and promote industry development, and relevant research and development projects have also emerged.

[0003] In the prior art, the human-computer interaction system usually adopts fixed interaction parameters and response modes, which is difficult to meet the personalized needs of different users. For example, dialect users often have low recognition accuracy in voice interaction due to insufficient dialect adaptation of the recognition model; the touch habits (such as touch force and trajectory mode) of different users are significantly different, and the traditional system lacks dynamic analysis capability for touch behavior; in addition, the parameters such as the speed of the interaction interface and the font size are often set uniformly, and cannot be adaptively adjusted according to user characteristics (such as age and visual ability).

[0004] These problems lead to obvious limitations in the user experience of existing interaction systems, and there is an urgent need for an interaction technology that can comprehensively process multi-modal data and realize dynamic adaptive adjustment. SUMMARY

[0005] The purpose of the present application is to provide a personalized adaptive interaction method, device and system to solve the problems of insufficient personalization of the interaction system, poor dialect recognition effect, and inflexible touch feedback in the prior art.

[0006] To solve the above technical problems, the technical solution adopted by the present application is:

[0007] A personalized adaptive interaction method, comprising the following steps:

[0008] Step S1, collecting voice data and touch trajectory data of a user, and obtaining user portrait information;

[0009] Step S2, performing dialect voice enhancement processing and voice recognition on the voice data to obtain text information;

[0010] Step S3, performing feature extraction on the touch trajectory data, and fusing the user portrait information to form a joint feature vector;

[0011] Step S4, performing touch mode analysis based on the touch trajectory data;

[0012] Step S5, based on the user portrait information and the touch mode, using the transfer learning technology to adjust the speech speed, UI font size and touch feedback strength of the voice interaction;

[0013] Step S6, according to the text information, the joint feature vector and the adjusted interaction parameters, dynamically adjusting the interaction interface and the instruction response to realize the personalized adaptive interaction;

[0014] Step S7, the feedback effect of the personalized adaptive interaction is evaluated, and the adaptive process is triggered according to the evaluation result, and the above steps are repeated to dynamically learn and adjust.

[0015] According to the above technical scheme, the speech data is processed by dialect speech enhancement and speech recognition, which is realized by the end-side distilled Whisper model, the teacher-student network structure is used for knowledge distillation, the original Whisper-large pre-training model in mixed mandarin and dialect corpus is used as the teacher model, the lightweight Whisper-tiny architecture is used as the student model, and the attention distillation loss function is introduced, and the function is specifically:

[0016]

[0017] In the formula, The attention distillation loss is represented by ; It is a hyperparameter, and the value range is usually between 0 and 1, which is used to balance the weight of two different loss terms and determine their contribution proportion to the final loss; The cross-entropy loss is represented by ; The attention matrix of the teacher model is represented by ; The attention matrix of the student model is represented by, The distance measure between the attention distribution of the teacher model and the attention distribution of the student model is represented by.

[0018] According to the above technical scheme, the dialect speech enhancement processing also includes constructing a multi-modal corpus containing 10 dialects, expanding the data amount by 5 times through speech synthesis technology, and training the end-side distilled Whisper model using the dialect variant generated by the adversarial network.

[0019] According to the above technical scheme, the feature extraction of the touch trajectory data is as follows: the spatial distribution feature is extracted by using the convolutional neural network as the spatial feature; the LSTM network is used to capture the time sequence mode as the time sequence feature; the Kalman filter and normalization processing are performed on the touch pressure sensor data to obtain the pressure value and pressure fluctuation amplitude as the pressure feature.

[0020] According to the technical solution, the DBSCAN density clustering algorithm is used to analyze the touch mode, the fused joint feature vector is unsupervised grouped, and the touch mode of the user is recognized.

[0021] According to the technical solution, the transfer learning technology is used to adjust the interaction parameter, the DANN deep domain adaptation network is used to combine the user portrait label to perform domain adaptation training, the task loss and the adversarial loss are minimized to obtain the interaction parameter of the user portrait information and the touch mode, wherein the expected negative form is represented by ; and the classifier is represented by, which classifies the features extracted from the feature image, and is used to convert the feature output of the model into a prediction probability distribution of each category. The feature extraction result of the model on the input x is represented by ; the classifier is represented by , which is used to classify the features extracted from the feature image, and the DANN adversarial loss weight is represented by .

[0022] According to the technical solution, the transfer learning technology is used to adjust the interaction parameter, and the MAML model-independent meta-learning algorithm is used to initialize the model parameter, the gradient accumulation technology is used to update the model once every 50 data, and the prototype samples of each group are retained to perform playback training regularly.

[0023] According to the technical solution, the speech speed of the speech interaction is adjusted, the reinforcement learning is used to optimize the time length stretching coefficient of the speech synthesis, and the reward function is set to realize , wherein the reinforcement learning reward function weights are represented by respectively; the UI font size is adjusted by a linear regression model , wherein a and b represent the linear regression model coefficients respectively; the touch feedback strength is adjusted by combining the pressure feature and the touch mode, and the following formula is used to realize dynamic adjustment of the touch feedback threshold , wherein represents the touch feedback strength, that is, the feedback strength given by the device when the user performs a touch operation, represents the basic feedback strength, which is a fixed initial feedback strength value, and k is a coefficient for adjusting the influence degree of pressure fluctuation on the feedback strength.

[0024] An individualized adaptive interaction device comprises a data acquisition unit for acquiring voice data and touch trajectory data of a user and obtaining user portrait information.

[0025] The data acquisition unit is used to acquire voice data and touch trajectory data of a user and obtain user portrait information.

[0026] A voice processing unit is configured to perform dialect voice enhancement processing and voice recognition on the voice data to obtain text information.

[0027] A touch processing unit is configured to perform feature extraction on the touch trajectory data and fuse the extracted features with the user portrait information to form a joint feature vector.

[0028] A touch mode analysis unit is configured to perform touch mode analysis based on the touch trajectory data.

[0029] A parameter adjustment unit is configured to adjust the speech speed, UI font size and touch feedback strength of the voice interaction based on the user portrait information and the touch mode by using a transfer learning technique.

[0030] An interaction adjustment unit is configured to dynamically adjust the interaction interface and instruction response based on the text information, the joint feature vector and the adjusted interaction parameters, so as to realize personalized adaptive interaction.

[0031] A feedback evaluation unit is configured to evaluate the feedback effect of the personalized adaptive interaction, and trigger an adaptive process based on the evaluation result to control the data acquisition unit to collect new user voice data and touch trajectory data.

[0032] A personalized adaptive interaction system includes a data acquisition module configured to collect user voice data and touch trajectory data and obtain user portrait information.

[0033] The voice processing module is configured to perform dialect voice enhancement processing and voice recognition on the voice data to obtain text information.

[0034] The touch processing module is configured to perform feature extraction on the touch trajectory data and fuse the extracted features with the user portrait information to form a joint feature vector.

[0035] The touch mode analysis module is configured to perform touch mode analysis based on the touch trajectory data.

[0036] The parameter adjustment module is configured to adjust the speech speed, UI font size and touch feedback strength of the voice interaction based on the user portrait information and the touch mode by using a transfer learning technique.

[0037] The interaction adjustment module is configured to dynamically adjust the interaction interface and instruction response based on the text information, the joint feature vector and the adjusted interaction parameters, so as to realize personalized adaptive interaction.

[0038] The feedback evaluation module is configured to evaluate the feedback effect of the personalized adaptive interaction, and trigger an adaptive process based on the evaluation result to control the data acquisition module to collect new user voice data and touch trajectory data.

[0039] Compared with the prior art, the present application has the following advantages:

[0040] In the present application, the Whisper model is combined with multi-modal dialect corpus expansion through end-side distillation, which significantly improves the accuracy of dialect speech recognition and solves the dialect user interaction barrier; through the touch trajectory processing method of fusing space, time sequence and pressure features, combined with DBSCAN clustering, the user touch mode is accurately identified to provide data support for personalized feedback. Based on the transfer learning and meta-learning technology, the dynamic adjustment of parameters such as speech speed, font size, and touch feedback strength is realized to meet the interaction habits of different users. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The adaptive interaction method flowchart of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0043] Embodiment one

[0044] As shown in the figure, a personalized adaptive interaction method includes the following steps: Figure 1

[0045] Step S1, collect the user's voice data, touch trajectory data, and obtain the user portrait information;

[0046] Step S2, perform dialect speech enhancement processing and speech recognition on the voice data to obtain text information;

[0047] Step S3, extract features from the touch trajectory data and fuse them with the user portrait information to form a joint feature vector;

[0048] Step S4, analyze the touch mode based on the touch trajectory data;

[0049] Step S5, based on the user portrait information and the touch mode, adjust the speech speed, UI font size, and touch feedback strength of the voice interaction using the transfer learning technology;

[0050] Step S6, dynamically adjust the interactive interface and instruction response according to the text information, joint feature vector, and adjusted interaction parameters to realize personalized adaptive interaction;

[0051] Step S7, evaluate the feedback effect of personalized adaptive interaction, and trigger the adaptive process according to the evaluation result to repeat the above steps for dynamic learning and adjustment. ​

[0052] In the present application, the Whisper model is distilled on the end side, combined with multi-modal dialect corpus expansion, which significantly improves the accuracy of dialect speech recognition and solves the dialect user interaction barrier; through the touch trajectory processing method of fusing space, time sequence and pressure features, combined with DBSCAN clustering, the user touch mode is accurately recognized to provide data support for personalized feedback. Based on transfer learning and meta-learning technology, dynamic adjustment of parameters such as speech speed, font size, and touch feedback strength is realized to meet the interaction habits of different users.

[0053] Continuous optimization capability: Through the feedback evaluation mechanism to trigger the adaptive iterative process, the system can continuously learn new user features to maintain the continuous optimization of the interactive experience.

[0054] Lightweight deployment: End-side distillation technology realizes model lightweight, suitable for deployment in resource-constrained scenarios such as mobile devices, while ensuring computing efficiency and recognition accuracy.

[0055] Embodiment two

[0056] This embodiment is a further refinement of embodiment one. This embodiment takes the interactive system of a smart terminal (such as a smartphone) as an example to explain the execution process of the method in detail:

[0057] The data acquisition unit collects user speech data through a microphone with a sampling rate of 16 kHz, and collects touch trajectory data (including coordinates, pressure values, and timestamps) through a touch screen sensor with a sampling frequency of 100 Hz.

[0058] User portrait information is obtained through registration information (such as age, dialect type), and combined with historical interaction data to supplement user preferences (such as commonly used font size, touch force habit).

[0059] Dialect speech enhancement processing uses the Whisper-tiny model after end-side distillation, and the teacher model is a pre-trained Whisper-large (mixed Mandarin and 10 dialect corpora). In the attention distillation loss function, The value is 0.7, The cross-entropy loss, The mean square error of the attention matrix of the teacher and student models.

[0060] The multi-modal corpus contains Mandarin and 10 dialects such as Cantonese, Sichuanese, and Minnan speech and text pairs, which are expanded 5 times through TTS speech synthesis technology, and dialect variants (such as different accents of Cantonese) are generated using adversarial networks for model training, with 100 iterations.

[0061] Touch trajectory data extracts spatial features (such as trajectory shape, area distribution) through CNN (such as ResNet-18), and LSTM network (hidden layer dimension 128) captures time sequence features (such as sliding speed, click interval). After Kalman filtering (state transition matrix F = 1, observation matrix H = 1) and normalization processing, the pressure data get the pressure value and fluctuation amplitude in the range of 0-1.

[0062] The embodiment provides a mathematical model for touch trajectory feature extraction, in particular:

[0063] First, assume that the original sampling data time series of the touch trajectory is:

[0064]

[0065] Wherein, n is the number of sampling points; is the two-dimensional coordinate of the nth point; is the pressure value of the ith point; is the time stamp of the ith point (unit: s).

[0066] Step 1, spatial feature extraction (CNN model). Spatial features reflect the shape, position distribution and other spatial attributes of the trajectory, which are encoded by CNN.

[0067] Step 101, construct a spatial heat map: map the trajectory coordinates to a two-dimensional grid (spatial resolution M) of MxM, and construct a heat map :

[0068] The value of grid cell (u, v) is the proportion of the number of sampling points touched in the cell to the total number of points:

[0069]

[0070] Wherein, represents the (u, v)th grid cell, is an indicator function (1 if the condition is met, otherwise 0).

[0071] Step 102: CNN feature extraction: use a k-layer convolutional neural network to extract features from the heat map S, with the following structure:

[0072] The first convolutional layer: input , use convolution kernels with size , output feature map:

[0073]

[0074] Wherein,​ for a two-dimensional convolution operation, for a bias term, for an activation function (e.g., ReLU).

[0075] Pooling layer: max-pooling on F1 (window size 2x2, stride 2), output .

[0076] Subsequent layers: repeat the convolution and pooling operations, and finally obtain the spatial feature vector through global average pooling: ; where is the spatial feature dimension (e.g., 128).

[0077] Step 2, temporal feature extraction (LSTM model); temporal features reflect the dynamic changes of trajectories over time (e.g., speed, direction, rhythm), which are captured by LSTM to capture temporal patterns.

[0078] Step 201: Temporal input construction; define the temporal input sequence as the derived features of coordinates and time differences:

[0079]

[0080] where, , is the sampling interval.

[0081] The temporal input matrix is (n is the sequence length, and 5 is the feature dimension).

[0082] Step 202, LSTM feature extraction; LSTM captures long-term temporal dependencies through a gating mechanism, and the core formula is as follows:

[0083] Forget gate: controls the proportion of historical states to be retained:

[0084]

[0085] Input gate: controls the update proportion of new states:

[0086]

[0087] Candidate state: new state candidate value:

[0088]

[0089] Cell state update:

[0090]

[0091] Output gate: controls the output proportion of the current state:

[0092]

[0093] Hidden state:

[0094]

[0095] where, is the weight matrix, is the bias term, is the sigmoid function, is the element-wise product, is the initial state (usually set to 0).

[0096] The final hidden state at the last time step is taken as the time series feature vector:

[0097]

[0098] where is the time series feature dimension (e.g., 128).

[0099] Step 3, pressure feature extraction (Kalman filtering + normalization); pressure features reflect the size and stability of touch intensity, and the normalized value and fluctuation amplitude are calculated after Kalman filtering denoising.

[0100] Step 301: Kalman filter smoothing pressure data; original pressure contains noise, and Kalman filter is used to estimate the true pressure:

[0101] State definition: ( is the filtered pressure, is the pressure rate of change).

[0102] State equation (assuming uniform change):

[0103]

[0104] where ( is the sampling interval), Q is the process noise covariance (e.g. ).

[0105] Observation equation:

[0106]

[0107] where, (observed value), , R is the observation noise covariance (e.g., R = 0.02).

[0108] Filtering steps:

[0109] Prediction: (P is state covariance).

[0110] Update: (Kalman gain); .

[0111] The filtered pressure value is: .

[0112] Step 302: Normalization and fluctuation amplitude calculation; normalization: map the filtered pressure to the interval:

[0113]

[0114] where is the global minimum and maximum value of the pressure (statistical through historical data, such as ).

[0115] Fluctuation amplitude: measure the stability of the pressure, defined as the mean of the adjacent difference of the normalized pressure: .

[0116] Pressure feature vector:

[0117]

[0118] where is the average normalized pressure.

[0119] Step 4, feature fusion. Finally, the spatial, temporal, and pressure features are concatenated into a joint feature vector:

[0120]

[0121] In the present application, the shape and area distribution of the trajectory are preserved by the spatial feature through the heat map + CNN; the dynamic change rule is captured by the LSTM; the stability and consistency are ensured by the Kalman filter denoising + normalization of the pressure feature; in the present application, the feature dimensions can be flexibly adjusted to adapt to the computing power of different devices.

[0122] In the DBSCAN clustering algorithm, the neighborhood radius is set to 0.5, and the minimum sample number minPts is set to 5. The joint feature vector is divided into "fast operation", "accurate operation", "moderate operation", etc. 5 kinds of touch modes.

[0123] In the DANN network, the feature extractor adopts the CNN+LSTM architecture, the domain classifier is a 2-layer fully connected network (activation function ReLU), and the task classifier is a softmax layer. The value is 0.5, the training batch size is 32, and the iteration number is 200 rounds.

[0124] In MAML meta-learning, the meta-learning rate is 0.01, the task learning rate is 0.1, the model parameters are updated once every 50 data through gradient accumulation, and 100 prototype samples of each touch mode are reserved for regular playback training.

[0125] When adjusting the speech speed, the reward function of the reinforcement learning is , to predict the speech speed, is the user's preferred speech speed, S is the fluency score of the speech, and E is the error rate. The UI font size is adjusted through a linear regression model (x is the user's age and y is the font size). In the formula of the touch feedback force, is the reference force, k = 0.2, is the normalized pressure feature.

[0126] The embodiment provides a method for dynamically adjusting the touch feedback threshold. To realize the combination of dynamically adjusting the touch feedback threshold, the quantitative indicators of the pressure features and the classification results of the touch modes are integrated into the feedback force calculation formula to form a complete mathematical model.

[0127] Step 1: integrate the pressure features into a comprehensive pressure factor ; the and are weighted and fused to reflect the comprehensive influence of the pressure features on the feedback:

[0128]

[0129] Step 2: adjust the coefficient k in combination with the touch mode. Different touch modes are dynamically adjusted through the weight of the pressure sensitivity coefficient k:

[0130]

[0131] For example: , when the “gentle operation” mode , k = 0.3 * 1.5 = 0.45; when the “fast operation” mode , k = 0.3 * 0.8 = 0.24.

[0132] Step 3: the final touch feedback force formula. Integrate the above results into the basic formula to obtain the dynamic feedback formula combined with the pressure features and the touch mode:

[0133]

[0134] Substitute the expressions of k and :

[0135]

[0136] In this invention, the model achieves dynamic adaptation in the following way:

[0137] when The bigger (the stronger), The larger the value (the more volatile the fluctuation). The larger the size, the stronger the feedback; different modes Adjust the k value to adapt the feedback strategy to the user's operating habits (e.g., reduce sensitivity in "precise operation" mode to avoid accidental feedback).

[0138] Ultimately, the pressure characteristics determine the basic trend of feedback intensity, and the touch mode determines the sensitivity of this trend, thus achieving a personalized and adaptive touch feedback effect.

[0139] Furthermore, the interaction adjustment unit dynamically adjusts the interface layout and command response speed based on text information (such as instructions after speech recognition), joint feature vectors (such as the touch mode being "quick operation"), and adjusted parameters (such as a 10% increase in speech speed, a 16px increase in font size, and a 20% increase in feedback intensity).

[0140] Furthermore, the feedback evaluation unit evaluates user operation success rate (e.g., number of times the command was executed correctly / total number of times) and satisfaction rating (1-5 stars). When the success rate is below 80% or the satisfaction rating is below 3 stars, an adaptive process is triggered to retrain the model.

[0141] Example 3

[0142] This embodiment is a further refinement of Embodiment 2, focusing on the specific technical details of dialect speech enhancement processing:

[0143] The specific optimization of end-side distillation is as follows: In the teacher-student network structure, the number of parameters of the student model Whisper-tiny is 1 / 20 of that of the teacher model Whisper-large. Through knowledge distillation, while maintaining a recognition accuracy of over 85%, the computational complexity is reduced by 50%.

[0144] In the attention distillation loss function, the teacher model's attention matrix has a dimension of (L×H) (L is the sequence length, H is the hidden layer dimension), and the student model's corresponding dimension is (L×H / 2). After interpolation alignment, the mean squared error is calculated, enabling the student model to learn the teacher model's attention distribution. Both attention matrices are generated based on the same input speech data. By calculating the distance metric between the two (such as mean squared error) and combining it with cross-entropy loss, the attention distillation loss function is constructed. Ultimately, this allows the student model to learn the teacher model's attention pattern, improving dialect recognition performance.

[0145] The multi-modal corpus construction specifically includes: the corpus contains pure speech, noisy speech (signal-to-noise ratio of 5-15 dB), and speech with different emotions (happy, angry, calm) of 10 dialects, and the original data amount of each dialect is 100 hours, and after expansion, reaches 500 hours.

[0146] When the adversarial network generates dialect variants, the generator adopts the WaveNet architecture, the discriminator is a CNN classifier, and the dialect speech samples closer to the real scene are generated through adversarial training to improve the generalization ability of the model.

[0147] Embodiment Four

[0148] This embodiment is a further refinement of embodiment three, focusing on the optimization scheme of touch processing and parameter adjustment:

[0149] The touch feature fusion strategy specifically includes: the spatial feature, the time sequence feature, and the pressure feature are extracted through CNN, LSTM, and a full connection layer respectively, then are fused into a 1024-dimensional joint feature vector in a series manner, and are normalized through LayerNorm to improve the feature representation ability.

[0150] The attention mechanism is introduced, the weight coefficient of the pressure feature is dynamically adjusted according to the visual preference weight in the user portrait (such as the elderly paying more attention to touch feedback), and the influence of the key feature on the pattern analysis is enhanced.

[0151] The combination of transfer learning and meta-learning specifically includes: when the DANN network is trained, the user portrait label (such as the dialect type and age) and the touch feature are input simultaneously, and the model is adapted to the distribution difference of different user groups through domain adversarial training.

[0152] When the MAML model is initialized, specific model parameter initialization values are generated for different touch modes (such as slow touch of the elderly and fast touch of the young), the convergence speed of the model in the new user scene is accelerated, and the first adaptive adjustment time is shortened from 10 minutes of the traditional method to 3 minutes.

[0153] It should be noted that, in this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0154] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that the technical solutions described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalent ones. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A personalized adaptive interaction method, characterized in that: Includes the following steps: Step S1: Collect the user's voice data and touch trajectory data, and obtain user profile information; Step S2: Perform dialect speech enhancement processing and speech recognition on the speech data to obtain text information; Step S3: Extract features from the touch trajectory data to obtain touch trajectory features; fuse the touch trajectory features with user profile information to form a joint feature vector; Step S4: Perform touch pattern analysis based on touch trajectory data; Step S5: Based on the user profile information and the touch mode, the interaction parameters are adjusted using transfer learning technology. The interaction parameters include the speech rate of voice interaction, the UI font size, and the touch feedback intensity. Step S6: Based on the text information, joint feature vector, and interaction parameters adjusted in step S5, dynamically adjust the interactive interface and command response to achieve personalized adaptive interaction. Step S7: Evaluate the feedback effect of personalized adaptive interaction, trigger the adaptive process based on the evaluation result, and return to step S1 for dynamic learning and adjustment.

2. The personalized adaptive interaction method according to claim 1, characterized in that: Dialectal speech enhancement and speech recognition are performed on the speech data using a Whisper model after edge distillation. A teacher-student network structure is adopted for knowledge distillation. The original Whisper-large pre-trained model on mixed Mandarin and dialect corpora serves as the teacher model, and the lightweight Whisper-tiny architecture serves as the student model. An attention distillation loss function is introduced, the specific function of which is: In the formula, This indicates attention distillation loss; It is a hyperparameter, typically ranging from 0 to 1, used to weigh the weights of two different loss terms and determine their respective contributions to the final loss. Represents cross-entropy loss; The attention matrix representing the teacher model; This represents the attention matrix of the student model. This represents the distance metric between the attention distribution of the teacher model and the attention distribution of the student model.

3. The personalized adaptive interaction method according to claim 2, characterized in that: Dialect speech enhancement processing also includes building a multimodal corpus containing multiple dialects, expanding the data volume through speech synthesis technology, and using adversarial networks to generate dialect variants to train the end-distilled Whisper model.

4. The personalized adaptive interaction method according to claim 3, characterized in that: Feature extraction is performed on the touch trajectory data, specifically by using a convolutional neural network to extract spatial distribution features as spatial features. An LSTM network is used to capture timing patterns as timing features; Kalman filtering and normalization are performed on the touch pressure sensor data to obtain pressure values ​​and pressure fluctuation amplitudes as pressure features.

5. The personalized adaptive interaction method according to claim 4, characterized in that: The touch pattern analysis specifically uses the DBSCAN density clustering algorithm to perform unsupervised grouping of the fused joint feature vectors to identify the user's touch patterns.

6. The personalized adaptive interaction method according to claim 5, characterized in that: The interaction parameters are adjusted using transfer learning techniques. Specifically, a DANN deep domain adaptive network is used in conjunction with user profile tags for domain adaptive training, minimizing the task loss. and the loss of combat sum This maps user profile information and touch patterns to group-specific interaction parameters, where -E represents the desired negative form. This represents a task classifier used to output the model's features. Convert to predicted probability distributions for each category; This represents the feature extraction result of the model on input x; Represents a domain classifier. This indicates that the DANN uses lossy weights.

7. A personalized adaptive interaction method according to claim 6, characterized in that: The interaction parameters were adjusted using transfer learning techniques, and the model parameters were initialized using the MAML model-independent meta-learning algorithm. The model was updated every 50 data points using gradient accumulation techniques, and prototype samples of each group were retained for periodic replay training.

8. A personalized adaptive interaction method according to claim 7, characterized in that: The speech rate of the voice interaction is adjusted, and reinforcement learning is used to optimize the duration stretching coefficient of speech synthesis. The reward function R for reinforcement learning is set as follows: , These represent the weights of the reinforcement learning reward function; Adjusting the UI font size is achieved using a linear regression model: , where a and b represent the coefficients of the linear regression model; Adjust the touch feedback intensity by combining pressure characteristics and touch mode, and dynamically adjust the touch feedback threshold using the following formula: In the formula, Indicates the tactile feedback intensity. This represents the basic feedback strength, and k is a coefficient used to adjust the degree to which pressure fluctuations affect the feedback strength.

9. A personalized adaptive interactive device, characterized in that: It includes a data acquisition unit for collecting users' voice data, touch trajectory data, and obtaining user profile information; The speech processing unit is used to perform dialect speech enhancement processing and speech recognition on the speech data to obtain text information; The touch processing unit is used to extract features from touch trajectory data and fuse them with user profile information to form a joint feature vector; The touch pattern analysis unit is used to perform touch pattern analysis based on touch trajectory data; The parameter adjustment unit is used to adjust the speech rate of voice interaction, UI font size and touch feedback intensity based on user profile information and touch mode using transfer learning technology. The interaction adjustment unit is used to dynamically adjust the interactive interface and command response based on text information, joint feature vectors and adjusted interaction parameters to achieve personalized adaptive interaction. The feedback evaluation unit is used to evaluate the feedback effect of personalized adaptive interaction. Based on the evaluation result, it triggers the adaptive process and controls the data acquisition unit to collect new user voice data and touch trajectory data.

10. A personalized adaptive interaction system, characterized in that: It includes a data acquisition module, which is used to collect users' voice data, touch trajectory data, and obtain user profile information; The speech processing module is used to perform dialect speech enhancement and speech recognition on the speech data to obtain text information. The touch processing module is used to extract features from touch trajectory data and fuse them with user profile information to form a joint feature vector; The touch pattern analysis module is used to perform touch pattern analysis based on touch trajectory data; The parameter adjustment module is used to adjust the speech rate of voice interaction, UI font size and touch feedback intensity based on user profile information and touch mode using transfer learning technology. The interaction adjustment module is used to dynamically adjust the interactive interface and command response based on text information, joint feature vectors and adjusted interaction parameters to achieve personalized adaptive interaction. The feedback evaluation module is used to evaluate the feedback effect of personalized adaptive interaction. Based on the evaluation results, it triggers the adaptive process and controls the data acquisition module to collect new user voice data and touch trajectory data.

Images