An interaction mode recommendation method fusing user preference and ergonomics

Abstract

The application discloses a kind of interactive mode recommendation methods of fusing user preference and ergonomics, by introducing a large language model as intelligent reasoning agent (Intelligent Agent), construct a real two-way cognitive framework.The framework is translated into understandable, about user preference symbolic memory by preference prediction module to the physiological signal of user on one hand, realize the depth cognition of subjective state of people by robot;On the other hand, collect real-time ergonomics parameters.Finally, by the large language model intelligent agent, the two kinds of heterogeneous information are balanced and reasoned by thought chain, dynamically balance the subjective preference of user and physical ergonomics, so as to recommend the optimal interactive mode to user, reduce the number of interactive operation and failure, shorten task completion time, improve the natural degree of interaction, comfort degree perceived by user.

Description

Technical Field

[0001] This invention belongs to the field of human-robot interaction technology, and more specifically, relates to a method for recommending interaction methods that integrates user preferences and ergonomics. Background Technology

[0002] In recent years, augmented reality (AR) technology has been increasingly integrated with human-computer interaction (HRI), aiming to improve the efficiency and naturalness of interaction. Existing technologies are mainly developing along two directions:

[0003] 1. Using augmented reality for one-way visualization of robot intent

[0004] In human-robot collaborative tasks, to help users clearly understand the robot's intentions and avoid misunderstandings and collisions, researchers use AR technology to overlay the robot's internal state or future plans onto the user's field of vision as virtual information. For example, Newbury et al. used the HoloLens 2 head-mounted display to show the robot's target position and trajectory as a virtual path to the user. This approach enhances human-robot cognition (Robot→Human), effectively improving task fluency and safety, especially when the robot makes a mistake, allowing users to understand and correct it more quickly. However, this information flow is unidirectional; the system only transmits robot information to the human but cannot perceive or understand the user's subjective feelings and state during the interaction.

[0005] 2. Using physiological signals for one-way perception of user status

[0006] To enable robots to understand human states, some studies use physiological signals (such as skin conductance response (GSR) and photoplethysmography (PPG)) to monitor users' psychological states, including stress, emotions, and feelings of security. For example, in human-robot interaction tasks, analyzing a user's physiological signals can determine whether the user feels insecure, and adjust the robot's behavior accordingly, such as slowing down or increasing distance. This approach enhances the robot's understanding of humans (Human → Robot), attempting to create a safer and more comfortable interactive environment. However, these systems typically rely on pre-set rules (e.g., performing an action when the sense of security falls below a certain threshold), lacking a comprehensive understanding and dynamic balancing ability of complex situations, and failing to effectively integrate with the rich contextual information provided by AR (Augmented Reality).

[0007] While the aforementioned existing technologies have made some progress in their respective directions, they suffer from the following core, objective drawbacks:

[0008] 1. Lack of genuine two-way cognition and collaboration

[0009] Existing technologies either involve robots transmitting information unidirectionally to humans (AR visualization) or humans transmitting information unidirectionally to robots (physiological signal perception). These two processes are disconnected and fail to form a complete information interaction loop. This prevents the system from establishing a connection between "understanding the robot's intentions" and "perceiving the user's state," thus hindering true two-way cognition and collaboration.

[0010] 2. It cannot effectively resolve the inherent conflict between subjective user preferences and objective ergonomics.

[0011] During interaction, user-preferred interaction methods (e.g., gestures that feel more intuitive and fun) may be ergonomically inefficient or tiring; while ergonomically sound and least physically demanding interaction methods (e.g., voice or gaze) may not be preferred by users or feel unnatural in certain situations. Existing technologies typically optimize only one aspect, either satisfying user preferences or optimizing ergonomics, but there is no mechanism that can dynamically and intelligently balance and resolve the conflict between these two competing factors, resulting in a poor overall user experience.

[0012] 3. There is a lack of a central inference engine capable of handling multimodal heterogeneous information.

[0013] Existing systems typically use rule-based logic or simple machine learning models to process single types of data. For example, triggering fixed robot behaviors based on numerical values ​​of physiological signals. However, user preferences (derived from complex physiological and psychological activities) and ergonomic constraints (derived from the geometric relationships of physical space) are two entirely different types of information (heterogeneous). The former is subjective and dynamically changing, while the latter is objective and based on physical rules. Current technologies lack a sufficiently powerful central inference engine capable of simultaneously understanding and fusing these two types of information—symbolic preference descriptions (e.g., "the user dislikes gestures in area A") and structured physical data (e.g., "bending over is required")—and making complex, context-aware decisions based on this. Summary of the Invention

[0014] The purpose of this invention is to overcome the shortcomings of the prior art and provide an interaction method recommendation method that integrates user preferences and ergonomics. This method dynamically balances subjective user preferences with objective ergonomic data to recommend the optimal interaction method, thereby reducing the number of interaction operations and failures, shortening task completion time, and improving the perceived naturalness and comfort of the interaction.

[0015] To achieve the above-mentioned objectives, the present invention provides a method for recommending interaction methods that integrates user preferences and ergonomics, characterized by comprising the following steps:

[0016] (1) Quantify and symbolize user preferences

[0017] The preference prediction module transforms the user's original, continuous physiological signals into discrete, large-scale language model-understandable symbolic language, i.e., quantified and symbolized user preferences: the valence index changes of each interaction method in each interaction area. The interaction methods include voice, gesture, gaze, and gaze + voice. The interaction areas include unreadable areas, readable areas, and controllable areas. The area behind the user is an unreadable area, the area in front of the user that the arm can reach is a controllable area, and the area in front of the user that the arm cannot reach is a readable area. The valence index changes include positive valence (increased valence), ineffective valence (no change in valence), and negative valence (decreased valence).

[0018] (2) Quantitative Ergonomics

[0019] The data acquisition module collects physical environment data in real time and calculates the arm length and the angle between the user's line of sight and the target, generating ergonomic parameters, i.e., Boolean type flags.

[0020] Whether to bend over: If the arm length is insufficient to reach the interactive target, the flag is True; otherwise, the flag is False.

[0021] Whether to turn head: If the angle between the user's line of sight and the target exceeds half of the AR headset's field of view, the flag is True; otherwise, the flag is False.

[0022] (3) Intelligent decision-making agent based on large language model

[0023] 3.1) Construction of the reasoning module

[0024] An inference module is employed to receive user preferences from the preference prediction module and ergonomic parameters from the data acquisition module, and performs intelligent trade-offs. The inference module uses a large language model as an intelligent inference agent, programming and guiding its behavior through thought chain prompts, enabling it to act as a decision-making agent. These thought chain prompts consist of three parts: instructions, few-shot examples, and the current context. The instructions include:

[0025] Interaction usability rules: Voice is not available in the controllable or readable area; gestures are not available in the readable area; gestures, gazes, and gazes + voice are not available in the unreadable area.

[0026] User preferences, flexibility, and physical effort levels: User preferences: positive valence > negative valence > negative valence; Flexibility: gaze + speech > gesture > gaze > speech; Physical effort: gesture + bending over > gesture + no bending over (i.e., only hand movement) > gaze and gaze + speech with head movement > gaze and gaze + speech with no head movement > speech (i.e., mouth movement);

[0027] Decision-making level: User preferences are considered first, with flexibility and physical effort used as secondary criteria;

[0028] The few-sample example provides a complete example of input-thought chain-output. The input part covers different interaction objects / interaction areas, different user preferences, and different ergonomic parameters. The thought chain part lists the intermediate reasoning steps in detail, guiding the large language model to generate comprehensive and prioritized recommendations based on the balance of user preferences and ergonomic parameters.

[0029] The current scenario is: a query consisting of user preferences obtained by the current preference prediction module, ergonomic parameters output by the data acquisition module, interactive objects, and interactive areas. Among them, user preferences include at least the change in the valence index of the interactive area where the current interactive object is located.

[0030] 3.2) Recommended Interaction Methods

[0031] The current context is input into the reasoning module. Under the guidance of the thought chain prompts, the large language model analyzes the current context step by step. First, it selects the top N interaction methods that meet the interaction usability rules based on user preferences. Then, it sorts the top N interaction methods a second time based on flexibility and physical effort. Finally, it outputs a priority-based recommendation list of interaction methods.

[0032] (4) Interactive guidance

[0033] This interactive guidance module is responsible for presenting the decision results of the large language model, i.e., the recommended list of interaction methods, to the user: In the AR environment, the recommended list of interaction methods output by the large language model is displayed in the form of a virtual UI near the corresponding interaction target to provide clear guidance, but the user still retains the final choice, thus achieving a balance between system initiative and user autonomy.

[0034] The objective of this invention is achieved as follows.

[0035] This invention integrates user preferences and ergonomics in its interaction recommendation method, introducing a large-scale language model as an intelligent agent to construct a truly bidirectional cognitive framework. This framework, on the one hand, uses a preference prediction module to "translate" the user's physiological signals into understandable, symbolic memories of user preferences, enabling the robot to deeply understand the user's subjective state; on the other hand, it collects real-time ergonomic parameters. Finally, the large-scale language model intelligent agent weighs these two heterogeneous pieces of information through thought processes, dynamically balancing the user's subjective preferences and physical ergonomics, thereby recommending the optimal interaction method to the user. This reduces the number of interaction operations and failures, shortens task completion time, and improves the perceived naturalness and comfort of the interaction. Attached Figure Description

[0036] Figure 1 This is a flowchart of a specific implementation of the interaction method recommendation method that integrates user preferences and ergonomics of the present invention;

[0037] Figure 2 This is a schematic diagram illustrating the principle of a specific implementation of the interaction method recommendation method that integrates user preferences and ergonomics of the present invention;

[0038] Figure 3 This is a schematic diagram of the interactive area;

[0039] Figure 4 This is a schematic diagram illustrating the principle of a specific implementation method for quantifying and symbolizing user preferences;

[0040] Figure 5 It is a subjective and objective data evaluation chart, where (a) represents objective behavior and (b) represents subjective evaluation. Detailed Implementation

[0041] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0042] In this embodiment, as Figure 1 , 2 As shown, the interaction recommendation method of this invention, which integrates user preferences and ergonomics, includes the following steps:

[0043] Step S1: Quantify and symbolize user preferences

[0044] The preference prediction module transforms the user's original, continuous physiological signals into discrete, large-scale language model-understandable symbolic language—that is, quantified and symbolized user preferences: the valence index changes of each interaction method in each interaction region. Interaction methods include voice, gesture, gaze, and gaze + voice; interaction regions include unreadable regions, readable regions, and controllable regions, such as... Figure 3 As shown, the area behind the user is an unreadable area, the area in front of the user that the arm can reach is a controllable area, and the area in front of the user that the arm cannot reach is a readable area. Changes in valence index include positive valence (increased valence), ineffective valence (no change in valence index), and negative valence (decreased valence index).

[0045] In this embodiment, as Figure 4 As shown, the specific steps include:

[0046] Step S1.1: Physiological signal acquisition

[0047] The user's skin conductance response (GSR) and photoplethysmography (PPG) signals are collected in real time and synchronously over a period of time using wearable devices (such as Shimmer3+).

[0048] Step S1.2: Data Preprocessing

[0049] The acquired GSR and PPG signals undergo a series of processing steps to ensure data quality:

[0050] Segmentation: A 7-second sliding window and a 0.5-second step size are used to simultaneously slice the GSR and PPG signals to balance the continuity of the interaction and the robustness of the model. Each pair of GSR and PPG slice signals at the same time constitutes a physiological signal sample, which is labeled as positive or negative emotional valence based on the user's response. The GSR slice signal is denoted as S. GSR The PPG slice signal is denoted as S. PPG ;

[0051] Resampling: The physiological signal samples of positive and negative sentiment valence are balanced to ensure that their numbers are equal, so as to avoid bias towards one category when training the sentiment valence model. The training set consists of the same number of physiological signal samples of positive and negative sentiment valence.

[0052] Filtering: A third-order Butterworth filter is applied to the PPG signal of the physiological signal samples in the training set, and a medium-level smoothing filter is applied to the GSR signal to reduce noise.

[0053] Step S1.3: Emotional Valence Modeling

[0054] Construct a deep learning model, namely the sentiment valence model, to predict whether a user's sentiment valence is positive or negative. The structure of this sentiment valence model is as follows: Figure 4 As shown, specifically:

[0055] Step S1.3.1: Calculation of self-attention mechanism

[0056] Self-attention mechanism calculations were performed on GSR slice signals and PPG slice signals from physiological signal samples respectively to capture GSR slice signal S. GSR PPG slice signal S PPG The internal temporal dependencies of each segment are used to obtain the self-attention signal SA of the GSR slice. GSR PPG slice self-attention signal SA PPG Using the GSR slice self-attention signal SA GSR For example, for the i-th point SA GSR_i ,have:

[0057]

[0058] Among them, W Q W K W V dk is the learnable parameter matrix, and dk is the dimension of the key vector;

[0059] Step S1.3.2: Cross-attention calculation and feature fusion

[0060] First, the self-attention signal SA of the GSR slice. GSR PPG slice self-attention signal SA PPG Cross-attention calculation is performed to effectively fuse signals from two different modalities, GSR and PPG, to resolve the delay problem between signals and to complement each other's information, resulting in the GSR slice cross-attention signal CA. GSR PPG slice cross-attention signal CA PPG Using GSR slice cross-attention signal CA GSR For example, for the i-th point CA GSR_i ,have:

[0061]

[0062] Among them, W Q W K W V dk is the learnable parameter matrix, and dk is the dimension of the key vector. In the specific implementation, the above matrix is ​​different in the calculation of each self-attention mechanism and cross-attention mechanism.

[0063] This cross-attention mechanism helps the model network focus on important single-modal time segments and potential alignment relationships between the two modalities by exchanging information between features of the two modalities.

[0064] Then, the GSR slice cross-attention signal CA GSR PPG slice cross-attention signal CA PPG The features are then spliced ​​together to obtain the fused characteristics.

[0065] Step S1.3.3: Valence Classification Prediction

[0066] A 5-layer CNN + fully connected layer is used to extract higher-order features from the fused features and perform the final valence classification prediction: positive or negative sentiment valence.

[0067] Step S1.3.4: Train the emotional valence model

[0068] The emotional valence model is trained using physiological signal samples from the training set to obtain a well-trained emotional valence model.

[0069] In its implementation, the sentiment valence model is trained using the PyTorch framework and employs the Adam optimizer. To ensure robustness of the evaluation, 5-fold cross-validation is used. The complete model proposed in this invention ultimately achieves an accuracy of 93.68% and an F1 score of 93.54%. This performance significantly outperforms ablation models using only partial mechanisms and various baseline models, including LSTM, KNN, and SVM, demonstrating the superiority of the proposed model structure. Furthermore, the single-layer cross-attention mechanism ensures the system's real-time response capability, with a latency between 25 and 100 milliseconds, thus meeting the requirement for the preference prediction module to be updated at least once per second in real-time interactions.

[0070] Step S1.4: Valence Index and User Preference Generation

[0071] For the current user, GSR and PPG signals are collected simultaneously. Then, a 7-second sliding window and a 1-second step size are used to slice the signal, generating physiological signal samples at 1 second intervals. The PPG signal of the physiological signal sample is filtered using a third-order Butterworth filter, and the GSR signal is filtered using a medium-range smoothing filter. The samples are then fed into the emotional valence model to obtain positive or negative emotional valence.

[0072] The binary prediction result of the affective valence model, i.e., positive or negative affective valence, is transformed into a continuously changing valence index (VI) to track users' ongoing affective feedback to a certain interaction method. Its update rule is as follows:

[0073]

[0074] Where Val(t) is the valence index at the current time t, and Val(t-1) is the valence index at the previous time t-1.

[0075] The most crucial step is to correlate changes in the valence index with the current interaction context (i.e., interaction method and interaction area) to generate an LLM-readable, symbolic user preference. For example: "Using gestures in a controllable area elicits a negative valence." This step completes the critical transformation from continuous physiological signals to discrete linguistic symbols.

[0076] Step S2: Quantifying Ergonomics

[0077] The data acquisition module collects physical environment data in real time and calculates the arm length and the angle between the user's line of sight and the target, generating ergonomic parameters, i.e., Boolean type flags.

[0078] Whether to bend over: If the arm length is insufficient to reach the interactive target, the flag is True; otherwise, the flag is False.

[0079] Whether to turn head: If the angle between the user's line of sight and the target exceeds half of the AR headset's field of view (e.g., more than 21.5 degrees in the horizontal direction), the flag is True; otherwise, the flag is False.

[0080] This data acquisition module is responsible for acquiring objective data from the physical world in real time, serving as another input for LLM inference. In this embodiment, it specifically includes:

[0081] Step S2.1: Physical Context Data Acquisition

[0082] Using the sensors of AR devices (such as HoloLens 2), information such as the user's height, the type and three-dimensional coordinates of the interactive target, and the interaction area (controllable area, readable area, unreadable area) between the user and the interactive target can be obtained in real time.

[0083] Step S2.2: Ergonomic parameter calculation

[0084] Arm length estimation: Based on the user's height, the arm length L is estimated using the Jarzem linear formula. arm :

[0085]

[0086] Where H is the user's height, and m and b are the preset slope and intercept parameters.

[0087] The angle between the user's line of sight and the target is calculated based on the three-dimensional coordinates of the interactive target (with the user's location as the origin).

[0088] Ergonomic parameters are generated according to the above rules.

[0089] Step S3: Intelligent Decision Agent Based on Large Language Model

[0090] The inference module receives "user preferences" (subjective information) from the preference prediction module and "ergonomic parameters" (objective information) from the data acquisition module, and makes an intelligent trade-off.

[0091] Step S3.1: Construction of the inference module

[0092] An inference module is employed, receiving user preferences from the preference prediction module and ergonomic parameters from the data acquisition module, and performing intelligent trade-offs. This inference module uses a large language model (such as LLAMA 3.3) as the intelligent inference agent, i.e., the central inference engine. It programs and guides the behavior of the large language model through chain-of-thought (CoT) prompts, enabling it to act as a decision-making agent. The chain-of-thought prompts consist of three parts: instructions, few-shot examples, and the current context. Instructions define the task objective, rules for all available interaction methods (gestures, gaze, voice, gaze + voice) (such as available area, flexibility, physical burden), and explicitly specify the decision-making hierarchy: user preferences > flexibility and physical ergonomics. This sets the basic framework and priorities for LLM inference. Specifically, it includes:

[0093] Interaction usability rules: Voice is not available in the controllable or readable area; gestures are not available in the readable area; gestures, gazes, and gazes + voice are not available in the unreadable area.

[0094] User preferences, flexibility, and physical effort levels: User preferences: positive valence > negative valence > negative valence; Flexibility: gaze + speech > gesture > gaze > speech; Physical effort: gesture + bending over > gesture + no bending over (i.e., only hand movement) > gaze and gaze + speech with head movement > gaze and gaze + speech with no head movement > speech (i.e., mouth movement);

[0095] Decision-making level: Based on the following user preference memory and ergonomic factors (interaction flexibility and physical effort), the recommended interaction methods are determined and ranked according to the following principles: user preferences are considered first, and flexibility and physical effort are used as secondary criteria.

[0096] Few-shot examples provide complete examples of input-thought chain-output. The input covers different interactive objects / areas (such as robot dogs / controllable areas, drones / readable areas, humanoid robots / unreadable areas), different user preferences, and different ergonomic parameters. The thought chain details the intermediate reasoning steps, guiding the large language model to generate comprehensive and prioritized recommendations, i.e., outputs, based on a balance between user preferences and ergonomic parameters.

[0097] The current context is a query consisting of user preferences obtained by the current preference prediction module, ergonomic parameters output by the data acquisition module, the interaction object, and the interaction area. User preferences at least include changes in the valence index of the interaction area where the current interaction object is located. For example, "Question: 1. User preference memory: Gesture interaction elicits negative valence in the controllable area, gaze interaction elicits positive valence in the readable area, voice interaction elicits positive valence in the unreadable area, gaze + voice interaction elicits positive valence in the controllable area; 2. Interaction target: Humanoid robot; 3. Current area: Controllable area; 4. Should I bend over: Yes; 5. Is head movement required: Yes."

[0098] Step S3.2: Recommended Interaction Methods

[0099] The current context is input into the reasoning module. Guided by the thought chain prompts, the large language model analyzes the current context step by step. First, it selects the top N interaction methods that meet the interaction usability rules based on user preferences. Then, it sorts the top N interaction methods a second time based on flexibility and physical effort, and finally outputs a priority-based recommendation list of interaction methods.

[0100] In this embodiment, the above instructions, few-shot examples, and the current context are combined into the final input, and the suffix "A: Let's think step by step" is added to trigger the model's thought process reasoning. The model's output includes not only the final recommendation result but also detailed reasoning steps. To ensure consistency in the output format, the system requires the model to follow a specific structure, such as: "Please start answering after all the questions (Q) with 'A: Let's think step by step. A1: target= XXX (Interaction Target), region= XXX (CurrentInteraction Area of ​​Target), Available Interaction: … recommended\textunderscore interaction=XXX. A2: … . [ANSWER COMPLETED]'."

[0101] Step S4: Interactive Guidance

[0102] This interactive guidance module is responsible for presenting the decision results of the large language model, i.e., the recommended list of interaction methods, to the user: In the AR environment, the recommended list of interaction methods output by the large language model is displayed in the form of a virtual UI near the corresponding interaction target to provide clear guidance, but the user still retains the final choice, thus achieving a balance between system initiative and user autonomy.

[0103] Results Display

[0104] The study validated the findings through a user study involving 29 participants. The experiment was structured into four groups: Group A (baseline group without any guidance), Group B (considering only user preferences), Group C (considering only ergonomics), and Group D (using the complete system of this invention, balancing preferences and ergonomics).

[0105] Experimental results are as follows Figure 5 As shown, the asterisk indicates statistical significance. Compared with the other three groups, group D, which adopts the complete solution of this invention, shows significant advantages in multiple dimensions:

[0106] Objective performance: The number of interactive operations and failures is significantly reduced, and the task completion time is shorter.

[0107] Subjective evaluation: Users perceived significantly higher levels of naturalness and comfort in the interaction, while the mental workload, physical workload, and effort level measured by the NASA-TLX scale were significantly lower.

[0108] These results strongly demonstrate the technical advantages of the present invention: by introducing an LLM intelligent agent to dynamically balance the user's subjective preferences and physical ergonomics, the overall efficiency of human-computer interaction and user experience can be effectively and significantly improved, solving the core contradiction that existing technologies cannot address.

[0109] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A method for recommending interaction methods that integrates user preferences and ergonomics, characterized in that, Includes the following steps: (1) Quantify and symbolize user preferences The preference prediction module transforms the user's original, continuous physiological signals into discrete, large-scale language model-understandable symbolic language, i.e., quantified and symbolized user preferences: the valence index changes of each interaction method in each interaction area. The interaction methods include voice, gesture, gaze, and gaze + voice. The interaction areas include unreadable areas, readable areas, and controllable areas. The area behind the user is an unreadable area, the area in front of the user that the arm can reach is a controllable area, and the area in front of the user that the arm cannot reach is a readable area. The valence index changes include positive valence (increased valence), ineffective valence (no change in valence), and negative valence (decreased valence). (2) Quantitative Ergonomics The data acquisition module collects physical environment data in real time and calculates the arm length and the angle between the user's line of sight and the target, generating ergonomic parameters, i.e., Boolean type flags. Whether to bend over: If the arm length is insufficient to reach the interactive target, the flag is True; otherwise, the flag is False. Whether to turn head: If the angle between the user's line of sight and the target exceeds half of the AR headset's field of view, the flag is True; otherwise, the flag is False. (3) Intelligent decision-making agent based on large language model 3.1) Construction of the reasoning module An inference module is employed to receive user preferences from the preference prediction module and ergonomic parameters from the data acquisition module, and performs intelligent trade-offs. The inference module uses a large language model as an intelligent inference agent, programming and guiding its behavior through thought chain prompts, enabling it to act as a decision-making agent. These thought chain prompts consist of three parts: instructions, few-shot examples, and the current context. The instructions include: Interaction usability rules: Voice is not available in the controllable or readable area; gestures are not available in the readable area; gestures, gazes, and gazes + voice are not available in the unreadable area. User preferences, flexibility, and physical effort levels: User preferences: positive valence > ineffective valence > negative valence; Flexibility: gaze + speech > gesture > gaze > speech; Physical effort: gesture + bending over > gesture + no bending over (i.e., only hand movement) > gaze and gaze + speech with head movement > gaze and gaze + speech with no head movement > speech (i.e., mouth movement); Decision-making level: User preferences are considered first, with flexibility and physical effort used as secondary criteria; The few-sample example provides a complete example of input-thought chain-output. The input part covers different interaction objects / interaction areas, different user preferences, and different ergonomic parameters. The thought chain part lists the intermediate reasoning steps in detail, guiding the large language model to generate comprehensive and prioritized recommendations based on the balance of user preferences and ergonomic parameters. The current scenario is: a query consisting of user preferences obtained by the current preference prediction module, ergonomic parameters output by the data acquisition module, interactive objects, and interactive areas. Among them, user preferences include at least the change in the valence index of the interactive area where the current interactive object is located. 3.2) Recommended Interaction Methods The current context is input into the reasoning module. Under the guidance of the thought chain prompts, the large language model analyzes the current context step by step. First, it selects the top N interaction methods that meet the interaction usability rules based on user preferences. Then, it sorts the top N interaction methods a second time based on flexibility and physical effort. Finally, it outputs a priority-based recommendation list of interaction methods. (4) Interactive guidance This interactive guidance module is responsible for presenting the decision results of the large language model, i.e., the recommended list of interaction methods, to the user: In the AR environment, the recommended list of interaction methods output by the large language model is displayed in the form of a virtual UI near the corresponding interaction target to provide clear guidance, but the user still retains the final choice, thus achieving a balance between system initiative and user autonomy.

2. The interaction method recommendation method integrating user preferences and ergonomics according to claim 1, characterized in that, The quantification and symbolization of user preferences Includes the following steps: 1.1) Physiological signal acquisition The wearable device collects the user's skin conductance response (GSR) signal and photoplethysmography (PPG) signal in real time over a period of time. 1.2) Data Preprocessing The acquired GSR and PPG signals undergo a series of processing steps: Segmentation: A 7-second sliding window with a 0.5-second step size was used to simultaneously slice the GSR and PPG signals. Each pair of GSR and PPG slice signals at the same time constituted a physiological signal sample, which was labeled as positive or negative emotional valence based on the user's response. The GSR slice signal was denoted as S. GSR The PPG slice signal is denoted as S. PPG ; Resampling: The physiological signal samples of positive and negative emotional valence are balanced to make their numbers equal. The same number of physiological signal samples of positive and negative emotional valence constitute the training set. Filtering: A third-order Butterworth filter is applied to the PPG signal of the physiological signal samples in the training set, and a medium-range smoothing filter is applied to the GSR signal to reduce noise. 1.3) Emotional Valence Modeling Construct a deep learning model, namely the sentiment valence model, to predict whether a user's sentiment valence is positive or negative. 1.4 Valence Index and User Preference Generation For the current user, GSR and PPG signals are collected simultaneously. Then, a 7-second sliding window and a 1-second step size are used to slice the signal, generating physiological signal samples at 1 second intervals. The PPG signal of the physiological signal sample is filtered using a third-order Butterworth filter, and the GSR signal is filtered using a medium-range smoothing filter. The samples are then fed into the emotional valence model to obtain positive or negative emotional valence. The binary prediction result of the emotional valence model, i.e., positive or negative emotional valence, is transformed into a continuously changing valence index to track users' ongoing emotional feedback to a certain interaction method. Its update rule is as follows: ； Where Val(t) is the valence index at the current time t, and Val(t-1) is the valence index at the previous time t-1; By associating changes in the valence index with the current interaction context, namely the interaction method and interaction area, a readable and symbolic user preference can be generated from a large language model.

3. The interaction method recommendation method integrating user preferences and ergonomics according to claim 2, characterized in that, The structure of the emotional valence model is as follows: 1.3.1) Calculation of self-attention mechanism Self-attention mechanism calculations were performed on GSR slice signals and PPG slice signals from physiological signal samples to obtain the GSR slice self-attention signal SA. GSR PPG slice self-attention signal SA PPG ; 1.3.2) Cross-attention calculation and feature fusion First, the self-attention signal SA of the GSR slice. GSR PPG slice self-attention signal SA PPG Cross-attention calculation is performed to obtain the GSR slice cross-attention signal CA. GSR PPG slice cross-attention signal CA PPG ; Then, the GSR slice cross-attention signal CA GSR PPG slice cross-attention signal CA PPG The components are then spliced ​​together to obtain the fused features; 1.3.3) Valence Classification Prediction A 5-layer CNN + fully connected layer is used to extract high-order features from the fused features and perform the final valence classification prediction: positive or negative sentiment valence; 1.3.4) Training the Emotional Valence Model The emotional valence model is trained using physiological signal samples from the training set to obtain a well-trained emotional valence model.

Images