A Multimodal Approach to Designing the User Interface for Elderly Care Robots

Abstract

This invention discloses a multimodal interactive interface design method for elderly care robots, relating to the field of intelligent elderly care. The method includes: collecting user facial images, voice signals, and touch operation information to form a raw multimodal data stream; processing the raw multimodal data stream to extract standardized user real-time state feature vectors; inputting the user real-time state feature vectors and historical interaction data into a support vector regression model; obtaining the real-time cognitive load level based on the user's real-time state feature vectors in the current session; and analyzing the rate of change of user touch operation accuracy within a time window using historical interaction data, calculating the acceleration value of capability change and the trend of touch operation accuracy, and fusing them to derive the interaction abstraction level. This invention constructs a joint decision matrix sensitive to the acceleration of capability change, dynamically triggering an adaptive interaction strategy when accelerated capability decline is detected, driving the interface engine to adjust interface rendering and multimodal guidance in real time.

Description

Technical Field

[0001] This invention relates to the field of intelligent elderly care, and in particular to a design method for the interactive interface of elderly care robots based on multimodality. Background Technology

[0002] In the field of smart elderly care, multimodal interaction technology is the key to improving the interaction experience between elderly users and robots. Existing technologies usually adopt multimodal interaction methods based on static rules, which map the user's voice, gestures and other commands to the robot's fixed operations through predefined rules, realizing basic multimodal fusion and providing technical support for the initial application of elderly care robots.

[0003] However, the core limitation of existing methods lies in the static nature of their interaction strategies. They cannot adapt to the dynamic changes in the cognitive and physiological states of elderly users, lack the ability to perceive the real-time cognitive load of users, and are unable to predict and respond to the long-term decline in their interaction capabilities. When users experience temporary fatigue that leads to an increase in cognitive load, or when their hand control ability begins to decline slowly, the static interface cannot provide corresponding adaptive adjustments, resulting in a rigid interaction experience that is difficult to meet the dynamic needs of users. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a multimodal approach to designing an interactive interface for elderly care robots, which solves the problem that existing technologies, due to their static interaction strategies, cannot dynamically adapt to the real-time cognitive state and long-term ability changes of elderly users.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides a design method for an interactive interface of an elderly care robot based on multimodality, which includes collecting user facial images, voice signals and touch operation information and forming an original multimodal data stream;

[0008] The raw multimodal data stream is processed to extract standardized real-time user state feature vectors;

[0009] The real-time cognitive load level is obtained by inputting the user's real-time state feature vector and historical interaction data into the support vector regression model and obtaining the real-time cognitive load level based on the user's real-time state feature vector in the current session.

[0010] By analyzing the rate of change of user touch operation accuracy within a time window through historical interaction data, the acceleration value of capability change and the trend of touch operation accuracy are calculated and fused to obtain the interaction abstraction level. Statistical analysis is performed on user operation patterns in historical interaction data to set capability boundary levels. The normal fluctuation range of user capability stability period in historical interaction data is statistically analyzed to set capability degradation judgment threshold.

[0011] A joint decision matrix is ​​constructed based on real-time cognitive load level and interaction abstraction level. The dimension of the joint decision matrix is ​​configured to be sensitive to the acceleration value of capability change. When the acceleration value of capability change is negative and the absolute value exceeds the capability degradation judgment threshold, the interaction abstraction level is greater than the capability boundary level, and the joint decision matrix outputs an adaptive interaction strategy instruction set.

[0012] Based on the adaptive interaction strategy instruction set, the interaction interface engine of the elderly care robot is driven to dynamically render the interface and trigger multimodal guidance prompts, record the interaction results between the user and the rendered interface, and update the historical interaction data using the interaction results.

[0013] As a preferred embodiment of the multimodal-based elderly care robot interactive interface design method of the present invention, the following steps are included: Collecting user facial images, voice signals, and touch operation information to form an original multimodal data stream:

[0014] The robot acquires user facial images through its visual sensors, user voice signals through its auditory sensors, and user touch operation information through its tactile sensors.

[0015] The user's facial image, voice signal, and touch operation information are aggregated to form a raw multimodal data stream.

[0016] As a preferred embodiment of the multimodal elderly care robot interaction interface design method described in this invention, the following steps are included: processing the original multimodal data stream to extract standardized user real-time state feature vectors:

[0017] The user facial image sequence in the original multimodal data stream was analyzed to obtain the pupil diameter change rate as the first feature;

[0018] The user speech signal in the original multimodal data stream is analyzed, and the speech rate and pause frequency are extracted as the second feature;

[0019] The user touch operation information in the original multimodal data stream is analyzed to obtain operation smoothness and touch point positioning accuracy as the third feature;

[0020] The first, second, and third features are normalized and vectorized and then concatenated to form a standardized real-time user status feature vector.

[0021] As a preferred embodiment of the multimodal elderly care robot interaction interface design method described in this invention, the method includes the following steps: Based on the user's real-time state feature vector and historical interaction data, input into a support vector regression model; and based on the user's real-time state feature vector in the current session, obtain the real-time cognitive load level.

[0022] The cognitive load truth values ​​were obtained by using the standardized NASA-TLX scale when users performed interactive tasks. A large dataset of labeled user real-time state feature vectors and corresponding cognitive load truth values ​​was collected and standardized.

[0023] The radial basis function is used as the kernel function of the support vector regression model. The initial values ​​of the regularization parameter and the kernel function parameter are set. The support vector regression model is trained using the processed dataset. The support vector regression model parameters are optimized by minimizing the error between the actual cognitive load value and the predicted value of the support vector regression model, and a pre-trained support vector regression model is obtained.

[0024] The user's real-time status feature vector and historical interaction data are input together into a pre-trained support vector regression model;

[0025] The support vector regression model weights the pupil diameter change rate, blink frequency, speech rate, pause frequency, operation smoothness and touch point positioning accuracy features in the user's real-time state feature vector of the current session to obtain the real-time cognitive load level.

[0026] As a preferred embodiment of the multimodal elderly care robot interactive interface design method described in this invention, the following steps are included: analyzing the rate of change of user touch operation accuracy within a time window using historical interaction data, calculating the acceleration value of capability change and the trend of touch operation accuracy, and fusing them to derive the interaction abstraction level:

[0027] Extract the accuracy of user touch operations recorded in the past few days from the historical interaction database, and divide them into continuous and equal-length time windows according to time order. Divide the past day data into the first time window, the second time window and the current time window.

[0028] Based on the daily average value sequence of touch operation accuracy, the least squares method is used to perform univariate linear regression fitting to obtain the slope parameter of the regression line, and the trend of touch operation accuracy is extracted from the slope parameter of the regression line.

[0029] Based on the average value of the touch operation accuracy window of the first time window, the second time window and the current time window, the acceleration value of capability change is calculated using the central difference method.

[0030] The trend of touch operation accuracy and the acceleration value of capability change are fused using the dynamic weighted fusion method of the Sigmoid function to obtain a scalar value. Cluster analysis is then performed on the user's historical interaction data to obtain the level range.

[0031] Mapping scalar values ​​to level ranges yields interactive abstraction levels.

[0032] As a preferred embodiment of the multimodal elderly care robot interaction interface design method described in this invention, the method includes the following steps: statistically analyzing user operation patterns in historical interaction data, setting capability threshold levels, statistically analyzing the normal fluctuation range of user capability stability periods in historical interaction data, and setting capability decline judgment thresholds.

[0033] Extract all user interaction abstraction level records from the historical interaction database, and perform cluster analysis on the interaction abstraction level records using the K-means clustering algorithm to obtain clustering results. Based on the centroids of the clustering results, obtain candidate values ​​for capability boundary levels.

[0034] Based on the correlation between the cognitive characteristics of elderly users and the difficulty of interface operation, domain knowledge is obtained. Based on the domain knowledge, the median value is selected from the candidate values ​​of the ability boundary level to obtain the ability boundary level.

[0035] Filter out all capability change acceleration value records corresponding to the user's capability stability period from the historical interaction database, and calculate the mean and standard deviation of the capability change acceleration value records corresponding to the capability stability period;

[0036] The threshold for determining capability decline is set by adding twice the standard deviation to the mean of the recorded acceleration values ​​of capability change during the capability stabilization period.

[0037] As a preferred embodiment of the multimodal elderly care robot interaction interface design method described in this invention, the following steps are included: A joint decision matrix is ​​constructed based on real-time cognitive load level and interaction abstraction level. The dimension of the joint decision matrix is ​​configured to be sensitive to the acceleration value of capability change. When the acceleration value of capability change is negative and its absolute value exceeds the capability decline judgment threshold, the interaction abstraction level is greater than the capability boundary level. The joint decision matrix outputs an adaptive interaction strategy instruction set, including the following steps:

[0038] Based on real-time cognitive load level, interaction abstraction level, and capability change acceleration state, a joint decision matrix is ​​constructed for each combination of values.

[0039] In each decision, the capability change acceleration value and the capability decay judgment threshold are used to make a judgment according to the rules. When the capability change acceleration value is negative and the absolute value exceeds the capability decay judgment threshold, the capability change acceleration status is judged as a warning. When the capability change acceleration value is non-negative or its absolute value does not exceed the capability decay judgment threshold, the capability change acceleration status is judged as normal.

[0040] Using the acceleration state of capability change as a joint index, the joint decision matrix is ​​queried. When the acceleration state of capability change is an early warning, the joint decision matrix outputs an adaptive interaction strategy instruction set if the interaction abstraction level is greater than the capability boundary level or if the interaction abstraction level is not greater than the capability boundary level.

[0041] As a preferred embodiment of the multimodal-based interactive interface design method for elderly care robots described in this invention, the following steps are included: According to the adaptive interaction strategy instruction set, the interactive interface engine of the elderly care robot is driven to dynamically render the interface and trigger multimodal guidance prompts; the interaction results between the user and the rendered interactive interface are recorded; and the historical interaction data is updated using the interaction results.

[0042] The adaptive interaction strategy instruction set is sent to the interaction interface engine of the elderly care robot. The interaction interface engine parses the adaptive interaction strategy instruction set and performs dynamic rendering of the interface control layout and visual style.

[0043] The interactive interface engine synchronously triggers the speech synthesis module to play guidance prompts and control the screen to display visual animations according to the adaptive interaction strategy instruction set. The camera and touch screen record the interaction process between the user and the rendered interactive interface.

[0044] Extract the task completion status, number of operation errors, and user facial expression feedback from the interaction process as the result of this interaction, and write this result into the historical interaction database to update the historical interaction data.

[0045] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the multimodal elderly care robot interactive interface design method as described in the first aspect of the present invention.

[0046] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the multimodal-based elderly care robot interactive interface design method described in the first aspect of the present invention.

[0047] The beneficial effects of this invention are as follows: By collecting and fusing user facial images, voice signals, and touch operation information to form a multimodal data stream, after extracting standardized feature vectors, and combining them with historical interaction data, a support vector regression model is used to calculate the real-time cognitive load level. Based on the changing trend and acceleration value of touch operation accuracy, the interaction abstraction level is obtained. By constructing a joint decision matrix sensitive to the acceleration of capability changes, an adaptive interaction strategy is dynamically triggered when capability acceleration decline is detected. This drives the interface engine to adjust the interface rendering and multimodal guidance in real time, and records the interaction results to continuously optimize historical data, thereby achieving dynamic personalized adaptation and forward-looking intervention of the interactive interface. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a design method for the interactive interface of a multimodal elderly care robot.

[0050] Figure 2 Flowchart for extracting real-time user status feature vectors.

[0051] Figure 3 A flowchart for interactive abstraction of level calculation.

[0052] Figure 4 This is a schematic diagram of adaptive interaction strategy decision-making. Detailed Implementation

[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0054] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0055] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0056] Reference Figures 1-4 This is one embodiment of the present invention, which provides a method for designing an interactive interface for a multimodal elderly care robot, including the following steps:

[0057] S1. Collect user facial images, voice signals, and touch operation information to form a raw multimodal data stream.

[0058] S1.1 Acquire user facial images through the visual sensor of the elderly care robot, acquire user voice signals through the auditory sensor of the elderly care robot, and acquire user touch operation information through the tactile sensor of the elderly care robot.

[0059] Furthermore, the elderly care robot's visual sensor is activated and continuously captures video streams focused on the user's face, extracting a continuous sequence of facial images. The elderly care robot's auditory sensor, a four-microphone array, is activated and continuously collects sound signals from the environment, enhancing speech from the user's direction through beamforming technology to obtain clear user voice signals. The elderly care robot's tactile sensor, a capacitive touchscreen, continuously monitors changes in the capacitance of the screen surface, recording the coordinates, pressure values, and gesture trajectories formed by continuous touch points when the user's fingers touch the screen, which are then summarized into user touch operation information.

[0060] S1.2. Collect user facial images, user voice signals and user touch operation information to form a raw multimodal data stream.

[0061] Furthermore, the user facial image sequence acquired by the visual sensor, the user voice signal acquired by the auditory sensor, and the user touch operation information acquired by the tactile sensor are aligned and packaged according to a unified time base. The alignment operation ensures that all modal data have accurate timestamps. The user facial image, the user voice signal fragment within 500 milliseconds before and after the current moment, and the user touch operation information at the current moment are associated as data packets at the same time point. The time-aligned and associated data packets are aggregated together to form a raw multimodal data stream containing multimodal information and having time synchronization.

[0062] S2. Process the original multimodal data stream and extract standardized user real-time state feature vectors.

[0063] S2.1 Analyze the user facial image sequence in the original multimodal data stream and obtain the pupil diameter change rate as the first feature.

[0064] Furthermore, for the user facial image sequence in the original multimodal data stream, a face detection algorithm based on a Haar feature cascade classifier is used to locate the face region in each frame of the image. Then, within the face region, a facial key point detection model based on a shape prediction algorithm is used to locate the pupil center coordinates. By calculating the pixel distance change of the line connecting the pupil centers between consecutive frames, combined with the known camera focal length parameters for calibration, the actual pupil diameter is calculated. Within a time window, the ratio of the difference between the maximum and minimum pupil diameter values ​​to the average value is calculated, and the pupil diameter change rate is obtained as the first feature.

[0065] It should be noted that a large dataset of images with labeled facial key points (including pupil centers) was used. The parameters of the deep convolutional neural network were optimized by minimizing the Euclidean distance loss between the predicted key point coordinates and the true coordinates, using the gradient descent algorithm, until the model's prediction accuracy on the validation set reached the predetermined standard.

[0066] Specifically, the face detection algorithm based on the Haar feature cascade classifier scans each frame of the image to accurately locate the bounding box of the region containing the face. Within the face region defined by the bounding box, a facial key point detection model based on a shape prediction algorithm is further invoked to output the coordinates of a series of facial feature points, including the precise pixel positions of the left and right pupil centers. The pixel length of the line connecting the centers of both pupils is taken as the approximate pupil diameter of the current frame. By tracking the changes in pupil diameter values ​​in consecutive video frames and using the camera's factory-calibrated focal length parameters, the pixel distance is converted into the actual pupil diameter in millimeters in the physical world. Within a set time window, the maximum and minimum values ​​of the actual pupil diameter within the window are found, and the difference is calculated. This difference is divided by the average pupil diameter over the time period, and the resulting ratio is the pupil diameter change rate, which serves as the first feature characterizing the user's real-time state.

[0067] S2.2 Analyze the user speech signal in the original multimodal data stream and extract the speech rate and pause frequency as the second feature.

[0068] Furthermore, for the user speech signal in the original multimodal data stream, an endpoint detection algorithm based on short-time energy and zero-crossing rate is applied to distinguish speech segments from silent segments; for the detected speech segments, a speech recognition engine based on a hidden Markov model is used to convert the speech signal into text; the speech rate is obtained by counting the number of words in the text and dividing by the total duration of the speech segment; the pause frequency is obtained by counting the frequency of silent segments in the speech segment; the speech rate and pause frequency together serve as the second feature.

[0069] It should be noted that the speech recognition engine used employs a large dataset of speech segments and their corresponding text annotations. By maximizing the probability of the observed sequence (speech features) given the state sequence (phonemes, words), the forward-backward algorithm is used to estimate the parameters (state transition probabilities, observation probabilities) of the Hidden Markov Model.

[0070] Specifically, an endpoint detection algorithm based on short-time energy and zero-crossing rate is applied to continuous speech signals. By calculating whether the energy level of the signal exceeds a silence threshold and whether the zero-crossing rate is below a specific threshold within a short-time window, effective speech signal segments and silent segments are accurately distinguished. For each identified speech segment, it is input into a speech recognition engine based on a Hidden Markov Model. By matching acoustic features with a language model, the simulated speech signal is converted into a corresponding text sequence. The speech rate is calculated by counting the total number of words in the text sequence and then dividing it by the total duration of the speech segment from beginning to end. The result is the number of words per unit time, which serves as the quantification value of the speech rate. The pause frequency is calculated by counting the number of silent segments within the same speech segment whose duration exceeds a set threshold (e.g., 200 milliseconds) and dividing the number by the total duration of the speech segment to obtain the number of pauses per unit time, which serves as the pause frequency. The calculated speech rate value and pause frequency value together constitute the second feature characterizing the user's speech features.

[0071] S2.3 Analyze the user touch operation information in the original multimodal data stream to obtain operation smoothness and touch point positioning accuracy as the third feature.

[0072] Furthermore, the coordinate sequence of continuous touch points is extracted from the user touch operation information in the original multimodal data stream; the movement vector between adjacent touch points is calculated, and the standard deviation of the rate of change of the direction angle of the movement vector is calculated. The inverse ratio is defined as the operation smoothness; for pointing and clicking operations, the Euclidean distance between the final landing point of the touch point and the center point of the target space is calculated. The inverse ratio of the Euclidean distance is defined as the touch point positioning accuracy; operation smoothness and touch point positioning accuracy are used together as the third feature.

[0073] It should be noted that the screen coordinate sequence of consecutive touch points arranged in chronological order is extracted from the recorded touch events. The movement vectors between adjacent touch points are calculated, and then the direction angle of each movement vector is calculated. Then, the rate of change of the direction angle of the consecutive movement vectors is calculated, and the standard deviation of the rate of change is obtained. The magnitude of the standard deviation directly reflects the degree of jitter of the touch trajectory. The larger the value, the more unstable the operation. The reciprocal of the standard deviation is defined as the operation smoothness. The higher the value, the smoother the operation. The calculation of touch point positioning accuracy is for pointing and clicking operations: after the user completes a click, the final landing point coordinates of the touch point are recorded, and the Euclidean distance between the final landing point coordinates and the geometric center point of the target space is calculated. The distance intuitively reflects the accuracy of the click. The larger the distance, the lower the accuracy. The reciprocal of the Euclidean distance is defined as the touch point positioning accuracy. The calculated operation smoothness value and touch point positioning accuracy value together constitute the third feature characterizing the user's touch interaction capability.

[0074] S2.4 Normalize and vectorize the first feature, second feature and third feature and concatenate them to form a standardized user real-time state feature vector.

[0075] Furthermore, the first feature, pupil diameter change rate, the second feature, speech rate and pause frequency, and the third feature, operation smoothness and touch point positioning accuracy, are all Z-score normalized to make the mean of each feature 0 and the standard deviation 1. Then, all the normalized feature values ​​are concatenated into a one-dimensional array in a fixed order to form a standardized real-time user state feature vector.

[0076] S3. Input the user's real-time state feature vector and historical interaction data into the support vector regression model, and obtain the real-time cognitive load level based on the user's real-time state feature vector in the current session.

[0077] S3.1 Obtain the true value of cognitive load when users perform interactive tasks using the standardized NASA-TLX scale. Collect a large dataset of labeled user real-time state feature vectors and corresponding true values ​​of cognitive load, and perform standardization processing.

[0078] Furthermore, during the standard interactive tasks set by the elderly care robot, a standardized NASA-TLX scale is used to conduct a subjective questionnaire survey on users, collecting their psychological load scores for the current task as ground truth values ​​for cognitive load. A labeled dataset is formed by pairing the real-time state feature vectors of a large number of users in different task scenarios with the corresponding ground truth values ​​for cognitive load. The Z-score standardization is then applied to the values ​​of each dimension of the real-time state feature vectors in the labeled dataset, ensuring a mean of 0 and a standard deviation of 1, resulting in a standardized dataset.

[0079] S3.2. Use the radial basis function as the kernel function of the support vector regression model, set the initial values ​​of the regularization parameter and the kernel function parameter, train the support vector regression model using the processed dataset, optimize the support vector regression model parameters by minimizing the error between the true cognitive load value and the predicted value of the support vector regression model, and obtain the pre-trained support vector regression model.

[0080] Furthermore, the radial basis function is selected as the kernel function of the support vector regression model, and the initial values ​​of the regularization parameter and the radial basis function kernel parameter are set. The support vector regression model is trained using a standardized dataset. During the training process, the parameters of the support vector regression model, including the weight coefficients and bias terms of the support vectors, are optimized by minimizing the mean squared error between the true cognitive load value and the predicted value of the support vector regression model, until the prediction error of the model on the reserved validation set converges to a stable and small threshold below, thus completing the construction of the pre-trained support vector regression model.

[0081] S3.3 Input the user's real-time state feature vector and historical interaction data into the pre-trained support vector regression model.

[0082] Furthermore, the standardized real-time user status feature vector is combined with the user's historical interaction data extracted from the historical interaction database to form an extended feature vector, which is then input into a pre-trained support vector regression model.

[0083] S3.4 The support vector regression model weights the pupil diameter change rate, blink frequency, speech rate, pause frequency, operation smoothness and touch point positioning accuracy features in the user's real-time state feature vector of the current session to obtain the real-time cognitive load level.

[0084] Furthermore, the pre-trained support vector regression model, based on its internally learned weight distribution, performs a weighted summation of each feature in the input user real-time state feature vector (including pupil diameter change rate, blink frequency, speech rate, pause frequency, operation smoothness, and touch point positioning accuracy), and adds a bias term. The calculation process maps continuous feature values ​​to a real value output, and the output value is the obtained real-time cognitive load level.

[0085] It should be noted that after receiving the user's real-time state feature vector for the current session, the pre-trained support vector regression model calls the weight coefficients learned during its training phase, which correspond to the dimensions of the feature vector. These weight coefficients correspond to the importance of six features: pupil diameter change rate, blink frequency, speech rate, pause frequency, operation smoothness, and touch point positioning accuracy. Each feature value in the user's real-time state feature vector is multiplied by its corresponding weight coefficient, and all weighted results are summed. A bias term constant obtained during training is then added. The result of the weighted summation plus the bias term is a continuous real value, which is the real-time cognitive load level output by the support vector regression model.

[0086] S4. By analyzing the rate of change of user touch operation accuracy within a time window through historical interaction data, calculate the acceleration value of capability change and the trend of touch operation accuracy, and integrate them to obtain the interaction abstraction level.

[0087] S4.1 Extract the accuracy of user touch operations recorded in the past few days from the historical interaction database, and divide it into continuous and equal-length time windows according to time order. Divide the past day data into the first time window, the second time window and the current time window.

[0088] Furthermore, the accuracy values ​​of user touch operations recorded in the past ninety days are extracted from the historical interaction database. The accuracy value of each day is taken as a data point in chronological order. The past ninety days are divided into three consecutive and equal-length time windows, each containing thirty days of data. The earliest thirty days of data are divided into the second time window, the middle thirty days of data are divided into the first time window, and the most recent thirty days of data are divided into the current time window.

[0089] S4.2 Based on the daily average value sequence of touch operation accuracy, the least squares method is used to perform univariate linear regression fitting to obtain the slope parameter of the regression line, and the trend of touch operation accuracy is extracted from the slope parameter of the regression line.

[0090] Furthermore, for the thirty daily average touch operation accuracy data points within the current time window, using the time sequence number as the independent variable and the daily average accuracy as the dependent variable, a univariate linear regression fitting is performed using the least squares method to obtain an optimal fitting line. The slope parameter of this fitted line is then extracted as the trend of touch operation accuracy. The sign and magnitude of the slope parameter reflect the direction and rate of change of user interaction capability within the recent window.

[0091] S4.3. Based on the average value of the touch operation accuracy window of the first time window, the second time window and the current time window, calculate the acceleration value of capability change using the central difference method.

[0092] The expression for the acceleration value of energy change is:

[0093] ;

[0094] in, The acceleration value of the change in capability. The number of days between windows. This represents the average value of the user's touch operation accuracy within the current time window. This represents the average accuracy of touch operations performed by users on the window in real time. This represents the average value of the user touch operation accuracy for the second time window.

[0095] Furthermore, when calculating the acceleration value of capability change based on the window average of user touch operation accuracy data within the second time window, the first time window, and the current time window, the difference between the current time window average and the first time window average is calculated. This reflects the change in capability over a recent period. The difference between the mean of the first time window and the mean of the second time window is calculated. The difference reflects the change in ability over a slightly earlier period; then the difference between these two changes is calculated, i.e. The difference reflects the trend of the change in capacity itself. Dividing this difference by the number of days in the window interval... The square of the value is used to eliminate the influence of the time scale, thus obtaining a standardized value for the acceleration of capability change. .

[0096] It should be noted that existing technologies for assessing user capabilities typically rely on static analysis of operational performance at a single point in time or over a period of time, calculating average accuracy or a simple rate of change (first-order difference). This method can only describe whether capabilities are changing, but it cannot quantify whether such changes are accelerating, decelerating, or remaining stable.

[0097] This invention calculates the acceleration value of capability change using the central difference method (second-order difference), enabling a more dynamic and refined characterization of the capability change trend itself. It can identify earlier whether a user's capability has entered a rapid decline phase or is in a stable plateau phase. A user whose average accuracy is slowly declining but whose acceleration value is negative (accelerating decline) requires a more timely and proactive interface intervention strategy than a user whose average accuracy is also slowly declining but whose acceleration value is zero (uniform decline). Based on the early warning capability of acceleration, the interactive interface of the elderly care robot can shift from passive post-event adaptation to proactive and preventative pre-event adaptation, improving the timeliness and effectiveness of interactive support.

[0098] S4.4. The trend of touch operation accuracy and the acceleration value of capability change are fused using the dynamic weighted fusion method of the Sigmoid function to obtain a scalar value. Cluster analysis is then performed on the user's historical interaction data to obtain the level range.

[0099] The scalar value expression is:

[0100] ;

[0101] in, For scalar values, The dynamic weight of the force change acceleration value. For the trend of touch operation accuracy, This is the acceleration term after scaling and transformation.

[0102] Furthermore, the trend of touch operation accuracy, representing the long-term direction of change, and the acceleration value of capability change, representing the drastic degree of short-term change, are merged into a single scalar value. The fusion process employs a dynamic weighting method, focusing on the design of the weighting function, which is dynamically adjusted based on the sign of the acceleration value: when... When the value is negative (indicating that the ability is deteriorating at an accelerated rate), an acceleration term is assigned. Higher weights result in better fusion outcomes. More sensitive to the deteriorating trend; when When the value is non-negative, it is assigned a long-term trend. With high weights, the Sigmoid function is used to scale the acceleration values. Mapped to the interval (0,1) such that... Within a comparable order of magnitude, through The scalar value is obtained.

[0103] It should be noted that when integrating multi-dimensional capability indicators, existing technologies typically use fixed weights or simple linear weighted averages. This static integration method cannot distinguish the relative importance of long-term trends and short-term mutations.

[0104] The difference in this invention is based on a dynamic weighted fusion mechanism. Its weight allocation strategy is sensitive to the sign (i.e., the direction of change) of the acceleration of capability change, realizing a risk-first decision-making logic. When a high-risk signal of accelerated capability decline is detected, the weight of the long-term trend will be temporarily ignored or reduced, so that the scalar value of the fusion result will immediately be biased to reflect the acceleration of deterioration, triggering a more proactive intervention strategy. This is equivalent to giving the system a crisis perception capability. When faced with the risk of a sharp decline in user status, it can exceed its average performance (long-term trend) and make a rapid and protective response.

[0105] S4.5. Map scalar values ​​to level ranges to obtain interactive abstraction levels.

[0106] Furthermore, the obtained scalar values ​​are mapped to preset level ranges to derive the interaction abstraction level. The level range is determined by performing K-means clustering analysis on the long-term interaction abstraction level records of all users in the historical interaction database. For example, if the cluster centers are 0.1, 0.4, and 0.7, then scalar values ​​less than 0.25 are defined as semantic intent level, scalar values ​​between 0.25 and 0.55 are defined as instruction button level, and scalar values ​​greater than 0.55 are defined as precise operation level.

[0107] S5. Perform statistical analysis on user operation patterns in historical interaction data, set capability threshold levels, statistically analyze the normal fluctuation range of user capability stability period in historical interaction data, and set capability decline judgment threshold.

[0108] S5.1 Extract all user interaction abstraction level records from the historical interaction database, perform K-means clustering analysis on the interaction abstraction level records, obtain clustering results, and obtain candidate values ​​for capability boundary levels based on the centroids of the clustering results.

[0109] Furthermore, the abstracted interaction levels achieved by all users in all historical interaction sessions are extracted from the historical interaction database, forming a dataset containing a large number of data points. The K-means clustering algorithm is used to perform cluster analysis on this dataset, setting the number of clusters to three. Through iterative calculation, the data points are divided into three categories, and the coordinates of the cluster center point for each category are output. The values ​​of the cluster center points are extracted as candidate values ​​for capability boundary levels.

[0110] S5.2 Based on the correlation between the cognitive characteristics of elderly users and the difficulty of interface operation, domain knowledge is obtained. Based on the domain knowledge, the intermediate value is selected from the candidate values ​​of the ability boundary level to obtain the ability boundary level.

[0111] Furthermore, based on the experiential knowledge of human-computer interaction experts regarding the correlation between the cognitive characteristics of elderly users and the difficulty of interface operation—that is, domain knowledge—the candidate values ​​for the low, medium, and high ability boundary levels were analyzed. Domain knowledge indicates that user interaction ability typically presents three natural clusters: low, medium, and high. The candidate value with the middle value among the three candidate values ​​was selected as the ability boundary level, which is used in subsequent decision-making to distinguish whether the user is at a level that requires simplified interaction support or can maintain a relatively complex interaction level.

[0112] S5.3. Filter out all capability change acceleration value records corresponding to the user's capability stability period from the historical interaction database, and calculate the mean and standard deviation of the capability change acceleration value records corresponding to the capability stability period.

[0113] Furthermore, based on the stability of users' long-term interaction performance, the time periods in which users are in a stable ability period are filtered out from the historical interaction database. All capability change acceleration value records corresponding to these time periods are extracted, the arithmetic mean of capability change acceleration value records corresponding to the filtered capability stable period is obtained, and the standard deviation of the records is calculated.

[0114] It should be noted that the process of filtering all capability change acceleration value records corresponding to a user's capability stability period from the historical interaction database is as follows: First, filter conditions are set based on the stability of the user's long-term interaction performance. For example, select the time period in the past 90 consecutive days in which the absolute value of the long-term touch operation accuracy trend is lower than a preset threshold (e.g., 0.01). These time periods are determined to be the user's capability stability period. Extract all capability change acceleration values ​​recorded in these time periods determined to be capability stability periods from the historical interaction database. Calculate the arithmetic mean of the filtered capability change acceleration value records, that is, the sum of all values ​​divided by the total number of records. At the same time, calculate the standard deviation of the records, that is, the square root of the average of the squares of the differences between each value and the mean.

[0115] S5.4. Set the threshold for judging capability decline by adding twice the standard deviation to the mean value of the capability change acceleration value recorded during the capability stabilization period.

[0116] Furthermore, the mean of the recorded acceleration values ​​of capability change during the capability stabilization period is added to twice the standard deviation of the record, and the resulting value is set as the threshold for capability decline judgment. When the newly calculated acceleration value of capability change exceeds the upper limit of the normal fluctuation range during the stabilization period, it is judged as an abnormality or a sign of decline.

[0117] S6. Construct a joint decision matrix based on real-time cognitive load level and interaction abstraction level. Configure the dimension of the joint decision matrix to be sensitive to the acceleration value of capability change. When the acceleration value of capability change is negative and the absolute value exceeds the capability decay judgment threshold, the interaction abstraction level is greater than the capability boundary level, and the joint decision matrix outputs an adaptive interaction strategy instruction set.

[0118] S6.1 Based on the real-time cognitive load level, interaction abstraction level, and capability change acceleration state, construct a joint decision matrix for each combination of values.

[0119] Furthermore, a joint decision matrix is ​​constructed based on all possible combinations of three dimensions: real-time cognitive load level, interaction abstraction level, and acceleration of ability change. This achieves systematization and foresight in decision-making logic, and integrates the user's current instantaneous cognitive pressure, long-term interaction ability level, and acceleration trend of ability change in a three-dimensional way. This makes decision-making no longer dependent on static judgments of single or bivariate variables. When the acceleration of ability change indicates that the user's ability is entering a period of rapid decline, even if the user's long-term interaction ability level is still at a high level, the matrix can still force the output of a protective simplification strategy. This allows intervention to be initiated before the user actually encounters operational difficulties. The mechanism upgrades the response mode from passive adaptation to proactive protection, enhancing the robustness and security of the interactive interface when the user's state is unstable, and providing elderly users with a more flexible and caring interactive experience.

[0120] S6.2. In each decision, based on the capability change acceleration value and the capability decay judgment threshold, a judgment is made according to the rules. When the capability change acceleration value is negative and the absolute value exceeds the capability decay judgment threshold, the capability change acceleration state is judged as a warning. When the capability change acceleration value is non-negative or its absolute value does not exceed the capability decay judgment threshold, the capability change acceleration state is judged as normal.

[0121] Furthermore, at each decision-making moment, the acceleration value of capability change and the preset capability decay judgment threshold are obtained, and a judgment is made according to the preset judgment rules: if the current capability change acceleration value is negative and its absolute value exceeds the capability decay judgment threshold (for example, the capability change acceleration value is -0.2 and the capability decay judgment threshold is 0.15), then the current capability change acceleration state is judged as a warning; if the current capability change acceleration value is not negative or its absolute value does not exceed the capability decay judgment threshold, then the current capability change acceleration state is judged as normal.

[0122] S6.3. Using the capability change acceleration state as a joint index, query the joint decision matrix. When the capability change acceleration state is a warning, if the interaction abstraction level is greater than the capability boundary level or the interaction abstraction level is not greater than the capability boundary level, the joint decision matrix will output an adaptive interaction strategy instruction set.

[0123] Furthermore, the determined acceleration state of capability change, along with the current real-time cognitive load level and interaction abstraction level, are used as a joint index to query the constructed joint decision matrix. The joint decision matrix is ​​configured to provide an adaptive interaction strategy instruction set for the warning state when the acceleration state of capability change is in the warning state, regardless of whether the interaction abstraction level is greater than or less than the capability boundary level.

[0124] S7. Based on the adaptive interaction strategy instruction set, drive the interaction interface engine of the elderly care robot to dynamically render the interface and trigger multimodal guidance prompts, record the interaction results between the user and the rendered interaction interface, and use the interaction results to update the historical interaction data.

[0125] S7.1 The adaptive interaction strategy instruction set is sent to the interaction interface engine of the elderly care robot. The interaction interface engine parses the adaptive interaction strategy instruction set and performs dynamic rendering of the interface control layout and visual style.

[0126] Furthermore, while rendering the interface, the interactive interface engine simultaneously triggers the speech synthesis module based on the guidance prompts contained in the adaptive interaction strategy instruction set. This converts the text instructions into speech signals and plays the guidance prompts through the speaker. The interactive interface engine controls the screen to display preset visual animation effects around specific controls, such as highlighting or pulsed apertures. The camera continuously captures the user's facial video stream, and the touchscreen continuously records the user's touch operation event sequence, together recording the complete interaction process between the user and the rendered interactive interface.

[0127] S7.2 The interactive interface engine synchronously triggers the speech synthesis module to play guidance prompts and control the screen to display visual animations according to the adaptive interaction strategy instruction set. The camera and touch screen record the interaction process between the user and the rendered interactive interface.

[0128] Furthermore, the recorded interaction process is analyzed to extract key indicators: determining whether the user successfully completed the interaction task within the preset time, obtaining the task completion status (success / failure / timeout); counting the number of clicks or erroneous swipes that deviated from the target control during the operation, obtaining the number of operation errors; applying an expression recognition algorithm to the user's facial video stream captured by the camera to identify the user's main expression categories after completing the task (e.g., satisfaction, confusion, frustration), obtaining user expression feedback; and combining the task completion status, the number of operation errors, and the user expression feedback of this interaction as the result of this interaction, writing the interaction result into the corresponding record in the historical interaction database in a structured data format, thus updating the historical interaction data.

[0129] S7.3 Extract the task completion status, number of operation errors, and user facial expression feedback from the interaction process as the result of this interaction, and write the result of this interaction into the historical interaction database to update the historical interaction data.

[0130] Furthermore, the facial expression recognition algorithm used collects a large dataset of face images labeled with facial expression categories, trains it using a convolutional neural network architecture, and optimizes the network parameters by minimizing the cross-entropy loss between the network's predicted facial expression categories and the true labels.

[0131] This embodiment also provides a computer device applicable to the design method of the interactive interface of the elderly care robot based on multimodality, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the design method of the interactive interface of the elderly care robot based on multimodality as proposed in the above embodiment.

[0132] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0133] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the method for designing an interactive interface for a multimodal elderly care robot as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0134] In summary, this invention collects and fuses user facial images, voice signals, and touch operation information to form a multimodal data stream. After extracting standardized feature vectors, it combines historical interaction data and uses a support vector regression model to calculate the real-time cognitive load level. Based on the changing trend and acceleration value of touch operation accuracy, it derives the interaction abstraction level. By constructing a joint decision matrix sensitive to the acceleration of capability changes, it dynamically triggers an adaptive interaction strategy when capability acceleration decline is detected. This drives the interface engine to adjust interface rendering and multimodal guidance in real time and records interaction results to continuously optimize historical data, thereby achieving dynamic personalized adaptation and proactive intervention of the interactive interface.

[0135] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A design method for the interactive interface of a multimodal elderly care robot, characterized by: include, It collects user facial images, voice signals, and touch operation information and forms a raw multimodal data stream; The raw multimodal data stream is processed to extract standardized real-time user state feature vectors; The real-time cognitive load level is obtained by inputting the user's real-time state feature vector and historical interaction data into the support vector regression model and obtaining the real-time cognitive load level based on the user's real-time state feature vector in the current session. By analyzing historical interaction data, the rate of change of user touch operation accuracy within a time window is analyzed. The acceleration value of capability change and the trend of touch operation accuracy are calculated and fused to derive the interaction abstraction level, including the following steps: Extract the accuracy of user touch operations recorded in the past few days from the historical interaction database, and divide them into continuous and equal-length time windows according to time order. Divide the past day data into the first time window, the second time window and the current time window. Based on the daily average value sequence of touch operation accuracy, the least squares method is used to perform univariate linear regression fitting to obtain the slope parameter of the regression line, and the trend of touch operation accuracy is extracted from the slope parameter of the regression line. Based on the average value of the touch operation accuracy window of the first time window, the second time window and the current time window, the acceleration value of capability change is calculated using the central difference method. The expression for the acceleration value of energy change is: ; in, The acceleration value of the change in capability. The number of days between windows. This represents the average value of the user's touch operation accuracy within the current time window. This represents the average accuracy of touch operations performed by users on the window in real time. This represents the average value of the accuracy of user touch operations in the second time window. The trend of touch operation accuracy and the acceleration value of capability change are fused using the dynamic weighted fusion method of the Sigmoid function to obtain a scalar value. Cluster analysis is then performed on the user's historical interaction data to obtain the level range. The scalar value expression is: ； in, For scalar values, The dynamic weight of the force change acceleration value. For the trend of touch operation accuracy, This is the acceleration term after scaling and transformation; Mapping scalar values ​​to level ranges yields interactive abstraction levels; Statistical analysis of user operation patterns in historical interaction data is performed to set capability threshold levels. The normal fluctuation range of user capability stability period in historical interaction data is statistically analyzed, and a capability decline judgment threshold is set. A joint decision matrix is ​​constructed based on real-time cognitive load level and interaction abstraction level. The dimension of the joint decision matrix is ​​configured to be sensitive to the acceleration value of capability change. When the acceleration value of capability change is negative and the absolute value exceeds the capability degradation judgment threshold, the interaction abstraction level is greater than the capability boundary level, and the joint decision matrix outputs an adaptive interaction strategy instruction set. Based on the adaptive interaction strategy instruction set, the interaction interface engine of the elderly care robot is driven to dynamically render the interface and trigger multimodal guidance prompts, record the interaction results between the user and the rendered interface, and update the historical interaction data using the interaction results.

2. The method for designing the interactive interface of a multimodal elderly care robot as described in claim 1, characterized in that: Acquiring user facial images, voice signals, and touch operation information to form a raw multimodal data stream includes the following steps: The robot acquires user facial images through its visual sensors, user voice signals through its auditory sensors, and user touch operation information through its tactile sensors. The user's facial image, voice signal, and touch operation information are aggregated to form a raw multimodal data stream.

3. The method for designing the interactive interface of a multimodal elderly care robot as described in claim 2, characterized in that: The raw multimodal data stream is processed to extract standardized real-time user state feature vectors, including the following steps: The user facial image sequence in the original multimodal data stream was analyzed to obtain the pupil diameter change rate as the first feature; The user speech signal in the original multimodal data stream is analyzed, and the speech rate and pause frequency are extracted as the second feature; The user touch operation information in the original multimodal data stream is analyzed to obtain operation smoothness and touch point positioning accuracy as the third feature; The first, second, and third features are normalized and vectorized and then concatenated to form a standardized real-time user status feature vector.

4. The method for designing the interactive interface of a multimodal elderly care robot as described in claim 3, characterized in that: The real-time cognitive load level is obtained by inputting the user's real-time state feature vector and historical interaction data into a support vector regression model, and then using the user's real-time state feature vector in the current session. This includes the following steps: The cognitive load truth values ​​were obtained by using the standardized NASA-TLX scale when users performed interactive tasks. A large dataset of labeled user real-time state feature vectors and corresponding cognitive load truth values ​​was collected and standardized. The radial basis function is used as the kernel function of the support vector regression model. The initial values ​​of the regularization parameter and the kernel function parameter are set. The support vector regression model is trained using the processed dataset. The support vector regression model parameters are optimized by minimizing the error between the actual cognitive load value and the predicted value of the support vector regression model, and a pre-trained support vector regression model is obtained. The user's real-time status feature vector and historical interaction data are input together into a pre-trained support vector regression model; The support vector regression model weights the pupil diameter change rate, blink frequency, speech rate, pause frequency, operation smoothness and touch point positioning accuracy features in the user's real-time state feature vector of the current session to obtain the real-time cognitive load level.

5. The method for designing the interactive interface of a multimodal elderly care robot as described in claim 1, characterized in that: Statistical analysis of user operation patterns in historical interaction data is performed to set capability threshold levels. The normal fluctuation range of user capabilities during stable periods in historical interaction data is statistically analyzed, and a capability decline judgment threshold is set. This includes the following steps: Extract all user interaction abstraction level records from the historical interaction database, and perform cluster analysis on the interaction abstraction level records using the K-means clustering algorithm to obtain clustering results. Based on the centroids of the clustering results, obtain candidate values ​​for capability boundary levels. Based on the correlation between the cognitive characteristics of elderly users and the difficulty of interface operation, domain knowledge is obtained. Based on the domain knowledge, the median value is selected from the candidate values ​​of the ability boundary level to obtain the ability boundary level. Filter out all capability change acceleration value records corresponding to the user's capability stability period from the historical interaction database, and calculate the mean and standard deviation of the capability change acceleration value records corresponding to the capability stability period; The threshold for determining capability decline is set by adding twice the standard deviation to the mean of the recorded acceleration values ​​of capability change during the capability stabilization period.

6. The method for designing an interactive interface for a multimodal elderly care robot as described in claim 5, characterized in that: A joint decision matrix is ​​constructed based on real-time cognitive load level and interaction abstraction level. The dimensions of the joint decision matrix are configured to be sensitive to the acceleration value of capability change. When the acceleration value of capability change is negative and its absolute value exceeds the capability degradation judgment threshold, the interaction abstraction level is greater than the capability boundary level. The joint decision matrix outputs an adaptive interaction strategy instruction set, including the following steps: Based on real-time cognitive load level, interaction abstraction level, and capability change acceleration state, a joint decision matrix is ​​constructed for each combination of values. In each decision, the capability change acceleration value and the capability decay judgment threshold are used to make a judgment according to the rules. When the capability change acceleration value is negative and the absolute value exceeds the capability decay judgment threshold, the capability change acceleration status is judged as a warning. When the capability change acceleration value is non-negative or its absolute value does not exceed the capability decay judgment threshold, the capability change acceleration status is judged as normal. Using the acceleration state of capability change as a joint index, the joint decision matrix is ​​queried. When the acceleration state of capability change is an early warning, the joint decision matrix outputs an adaptive interaction strategy instruction set if the interaction abstraction level is greater than the capability boundary level or if the interaction abstraction level is not greater than the capability boundary level.

7. The multimodal-based elderly care robot interaction interface design method as described in claim 6, characterized in that, Based on the adaptive interaction strategy instruction set, the interaction interface engine of the elderly care robot dynamically renders the interface and triggers multimodal guidance prompts, records the interaction results between the user and the rendered interface, and updates historical interaction data using the interaction results, including the following steps: The adaptive interaction strategy instruction set is sent to the interaction interface engine of the elderly care robot. The interaction interface engine parses the adaptive interaction strategy instruction set and performs dynamic rendering of the interface control layout and visual style. The interactive interface engine synchronously triggers the speech synthesis module to play guidance prompts and control the screen to display visual animations according to the adaptive interaction strategy instruction set. The camera and touch screen record the interaction process between the user and the rendered interactive interface. Extract the task completion status, number of operation errors, and user facial expression feedback from the interaction process as the result of this interaction, and write this result into the historical interaction database to update the historical interaction data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the multimodal elderly care robot interactive interface design method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the multimodal elderly care robot interactive interface design method according to any one of claims 1 to 7.

Images