Embodied intelligent device human-computer interaction method and device based on human state perception, edge computing device and storage medium

Abstract

This disclosure provides a method, apparatus, edge computing device, and storage medium for human-computer interaction in embodied intelligent devices based on human condition perception, belonging to the field of embodied intelligence technology. It includes: acquiring human factor data of a user interacting with the embodied intelligent device; identifying the user's human condition category based on the human factor data, wherein the human condition category represents the user's physical and mental state; acquiring user commands and fusing the user commands and human condition categories to obtain fused perception information; and generating a response strategy corresponding to the user commands based on the fused perception information, so that the embodied intelligent device executes the interaction based on the response strategy. The embodiments of this disclosure can improve the accuracy of decision-making in embodied intelligent devices.

Description

Technical Field

[0001] This disclosure belongs to the field of embodied intelligence technology, and in particular relates to a human-computer interaction method, device, edge computing device and storage medium for embodied intelligence devices based on human state perception. Background Technology

[0002] Embossed smart devices are intelligent systems that achieve deep interaction with the physical environment through modules such as sensors and actuators. They can make decisions and execute tasks autonomously and are widely used in fields such as home services, industrial production, medical care, and smart education.

[0003] In related technologies, embodied intelligent devices typically plan tasks and make behavioral decisions based on user instructions. They complete corresponding interactive responses by parsing language instructions, gesture signals, or click operations. This can easily lead to mechanical decisions, resulting in decisions that do not meet the user's actual needs. Summary of the Invention

[0004] This disclosure aims to at least address the problem in existing technologies where embodied intelligent devices tend to make mechanical decisions, failing to consider the user's physical and mental state, resulting in decisions that do not meet the user's actual needs. To this end, this disclosure proposes a human-computer interaction method, apparatus, edge computing device, and storage medium for embodied intelligent devices based on human state perception, in order to improve the accuracy of decision-making by embodied intelligent devices.

[0005] In a first aspect, this disclosure provides a human-computer interaction method for embodied intelligent devices based on human state perception, including: Acquire human factors data of users who are interacting with embodied smart devices; The user's human factors status category is identified based on the aforementioned human factors data, wherein the human factors status category is used to characterize the user's physical and mental state. The user command is acquired, and the user command and the human condition category are fused to obtain fused perception information; Based on the fused perception information, a response strategy corresponding to the user command is generated, so that the embodied intelligent device can perform interaction based on the response strategy.

[0006] In some embodiments, identifying the user's human factors status category based on the human factors data includes: The process of identifying user's human factors status category based on the human factors data includes: When the human factors data is unimodal data, the human factors data is input into the corresponding human factors state recognition model to obtain the user's human factors state category; When the human factors data is multimodal data, the human factors data is input into the corresponding multimodal human factors state recognition model to obtain the user's human factors state category; Alternatively, when the human factors data is multimodal data, the human factors data is input into the human factors state recognition model corresponding to each modality to obtain the human factors state categories of multiple users; the human factors state categories of the multiple users are analyzed and decided to obtain the human factors state category of the final user.

[0007] In some embodiments, the user instructions include voice instructions and / or action instructions; The step of fusing the user command and the human condition category to obtain fused perception information includes: The user command is parsed and converted to obtain the first text information; The human condition category is converted to obtain the second text information; The first text information and the second text information are fused to obtain fused perception information.

[0008] In some embodiments, fusing the first text information and the second text information to obtain fused perceptual information includes: The first text information and the second text information are concatenated to obtain the fused perception information.

[0009] In some embodiments, generating a response strategy corresponding to the user instruction based on the fused sensing information includes: The fused perception information is input into a pre-trained language model so that the language model outputs a response strategy that matches the human condition category based on the user instructions in the fused perception information.

[0010] In some embodiments, the response strategy includes an action response; the method further includes: The language model retrieves a preset action library based on the fused perceptual information to obtain at least one action that matches the human condition category, wherein the actions in the preset action library are associated with the human condition category. The action response is based on at least one of the matched actions.

[0011] In some embodiments, the response strategy includes a voice response; the method further includes: The language model retrieves a preset timbre library based on the fused perceptual information to obtain a target timbre corresponding to the human condition category, wherein the timbres in the preset timbre library correspond to the human condition category; The speech response is based on the target timbre.

[0012] In some embodiments, generating a response strategy corresponding to the user instruction based on the fused sensing information includes: Obtain the environmental data of the user; The response strategy corresponding to the user command is generated based on the environmental data and the fused perception information.

[0013] In some embodiments, the method further includes: During the interaction process performed by the embodied intelligent device based on the response strategy, the user's human factors data is collected to identify the human factors state after the interaction. The response strategy is updated based on the human condition after the interaction.

[0014] Secondly, this disclosure provides a human-computer interaction device for embodied intelligent devices based on human state perception, comprising: The user data acquisition module is used to acquire human factors data of users who are interacting with the embodied smart device. The identification module is used to identify the user's human factors status category based on the human factors data, wherein the human factors status category is used to characterize the user's physical and mental state. The fusion module is used to acquire user commands and fuse the user commands with the human condition category to obtain fused perception information; The response module is used to generate a response strategy corresponding to the user command based on the fused perception information, so that the embodied intelligent device can perform interaction based on the response strategy.

[0015] Thirdly, this disclosure provides an edge computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the human-computer interaction method of embodied intelligent device based on human state perception as described in the first aspect above.

[0016] Fourthly, this disclosure provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the human-computer interaction method for embodied intelligent devices based on human state perception as described in the first aspect above.

[0017] Through the above technical solution, the human-computer interaction method for embodied intelligent devices based on human condition perception provided in this disclosure acquires human condition data of users in the interaction state with the embodied intelligent device, identifies the user's human condition category based on the human condition data, and integrates user commands with human condition categories to generate response strategies. This enables the embodied intelligent device to not only consider the user's surface interaction information when performing interactions, but also incorporate human condition information, thereby deeply understanding the user's real needs and emotional state, adaptively adjusting and optimizing the interaction logic, adapting to the real needs of different users in different situations, achieving intelligent interaction that is more in line with user experience, and thus improving the accuracy of the embodied intelligent device's decision-making.

[0018] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this disclosure and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the human-computer interaction method for embodied intelligent devices based on human state perception provided in this embodiment of the disclosure. Figure 2 This is a schematic diagram illustrating a scenario example provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram of the structure of the embodied intelligent device human-computer interaction device based on human condition perception provided in the embodiments of this disclosure; Figure 4 This is a schematic diagram of the structure of the embodied intelligent device human-computer interaction system based on human state perception provided in this embodiment of the disclosure; Figure 5 This is a schematic diagram of the structure of the edge computing device provided in the embodiments of this disclosure. Detailed Implementation

[0021] The technical solutions of the embodiments of this disclosure will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure are within the scope of protection of this disclosure.

[0022] The terms "first," "second," etc., used in this disclosure and in the claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] It should be noted that in the optional embodiments of this disclosure, the user information and related data involved require user permission or consent when applied to specific products or technologies. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if user-related data is involved in the embodiments of this disclosure, it must be obtained with the user's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the user's authorization and consent.

[0024] In this embodiment, the embodied intelligent device can integrate multimodal sensors, decision control modules, and artificial intelligence algorithms to achieve intelligent mapping from digital space to physical or virtual space. The embodied intelligent device can be a physical entity equipped with actuators such as robotic arms, mobile chassis, and dexterous hands, such as humanoid robots, service robots, autonomous vehicles, and wearable intelligent devices; it can also be an intelligent agent with a virtual avatar capable of interacting with users through a display interface or voice channel, such as a virtual digital human, an intelligent voice assistant, and an immersive interactive interface.

[0025] Embossed intelligent devices can be applied to fields such as home services, industrial production, medical care, and smart education. Taking smart education as an example, educational robots can serve as intelligent learning companions for students, answering questions through natural language interaction, assisting in completing experiments, and dynamically adjusting teaching content according to learning progress; virtual digital teachers can simulate real teaching scenarios, providing immersive language practice and contextualized knowledge explanations. Of course, embodied intelligent devices can also be applied to other fields, and this disclosure does not limit their applications.

[0026] In related technologies, embodied intelligent devices typically plan tasks and make behavioral decisions based on user instructions, completing corresponding interactive responses by parsing language commands, gesture signals, or click operations. However, due to a lack of perception and analysis of user emotions, they often make mechanical decisions, resulting in outcomes that do not meet the user's true needs.

[0027] For example, in smart education scenarios, when a student, due to difficulty understanding a concept, expresses anxiety (such as speaking faster, trembling, or a racing heart) and asks "explain it again," the embody intelligent teaching assistant will still repeat the explanation at the normal pace and in the normal procedure. It cannot perceive the student's anxiety and slow down its speech or break down the difficult points, leading to the student becoming even more agitated and unable to understand the concept. In home service scenarios, when a user, due to fatigue and irritability, urgently says "turn on the lights," the device merely mechanically performs the operation of turning on the lights. It cannot perceive the user's fatigue, nor can it proactively adjust the light brightness to a softer mode or reduce environmental noise, failing to meet the user's true comfort needs.

[0028] Based on this, this disclosure proposes a human-computer interaction method, apparatus, edge computing device, and storage medium based on human-state perception for embodied intelligent devices. The following, in conjunction with the accompanying drawings, provides a detailed description of the human-computer interaction method, apparatus, edge computing device, and storage medium based on human-state perception for embodied intelligent devices provided by this disclosure through specific embodiments and application scenarios.

[0029] Among them, the human-computer interaction method of embodied intelligent devices based on human state perception can be applied to the terminal, and can be executed by the hardware or software in the terminal.

[0030] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0031] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may also be one or more other physical user interface devices, such as a physical keyboard, mouse, and joystick.

[0032] The human-computer interaction method based on human state perception of embodied intelligent devices provided in this disclosure can be executed by an edge computing device or a functional module or entity within an edge computing device that can implement the human-computer interaction method based on human state perception of embodied intelligent devices. The edge computing device mentioned in this disclosure can be a wearable device (such as a smartwatch or smart bracelet), an eye tracker, an AR (Augmented Reality) / VR (Virtual Reality) / XR (Extended Reality) / MR (Mixed Reality) device, an edge server, an edge gateway, a cloud computing device, etc. The execution subject of the human-computer interaction method based on human state perception of embodied intelligent devices can also be an embodied intelligent device. The following description uses an edge computing device as the execution subject to illustrate the human-computer interaction method based on human state perception of embodied intelligent devices provided in this disclosure.

[0033] like Figure 1 As shown, the human-computer interaction method of the embodied intelligent device based on human state perception includes steps 110, 120, 130 and 140.

[0034] Step 110: Obtain human factors data of users who are interacting with the embodied smart device.

[0035] In this embodiment of the disclosure, human factors data refers to data that can characterize a user's physiological state, emotional state, and behavioral state, including but not limited to electroencephalogram (EEG) data, near-infrared brain imaging data, physiological data, eye-tracking data, and facial image data. Physiological data includes, but is not limited to, electrocardiogram (ECG) data, electrodermal activity (EDA) data, electromyography (EMG) data, photoplethysmogram (PPG) data, skin temperature (SKT) data, and heart rate variability (HRV) data.

[0036] In this embodiment of the disclosure, the interaction state refers to the state in which the user and the embodied intelligent device are exchanging information or performing physical actions. For example, the user is interacting with the embodied intelligent device through voice commands, gesture operations, haptic feedback, or visual gaze, or the embodied intelligent device is providing feedback to the user through voice, actions, etc. In the interaction state, the user's human factors data can reflect the user's physiological state, emotional state, and behavioral state during the interaction process.

[0037] In this embodiment of the disclosure, human factors data of a user interacting with a embodied smart device can be acquired. For example, human factors data of a user interacting with a embodied smart device can be acquired through a wearable device. Wearable devices may include terminal devices with data acquisition and data transmission functions, such as smart bracelets, smart glasses, eye trackers, EEG caps, and adhesive physiological sensors. For example, a smart bracelet with heart rate and blood oxygen detection functions can be worn on the user's wrist to collect heart rate and blood oxygen saturation data; smart glasses with an integrated eye-tracking module can be worn on the user's eyes to acquire eye movement data such as pupil diameter changes and fixation point coordinates; and an EEG cap can be worn on the user's head to acquire EEG data. The wearable device can transmit the collected human factors data to an edge computing device via communication methods such as Bluetooth, Wi-Fi, or wired connection.

[0038] In some embodiments, human factors data of users interacting with the embody smart device can also be obtained in other ways, such as by obtaining human factors data such as voice and facial images of users through cameras, microphones, etc.; human factors data of users transmitted from other devices can also be received. This disclosure does not limit the method of obtaining human factors data of users interacting with the embody smart device.

[0039] Step 120: Identify the user's human factors status category based on human factors data, whereby the human factors status category is used to characterize the user's physical and mental state.

[0040] In this embodiment of the disclosure, the human condition category is a classification of the user's current physical and mental state. The human condition category may include, but is not limited to, emotional state category, cognitive load state category, fatigue state category, stress state category, and pressure state category.

[0041] The emotional state category describes the user's emotional experience tendencies, and can include positive emotional states (such as happiness and relaxation), negative emotional states (such as sadness and frustration), and neutral emotional states. The cognitive load category describes the user's information processing resource consumption, and can include low cognitive load, medium cognitive load, and high cognitive load. The fatigue state category describes the user's accumulated physical and mental fatigue, and can include a conscious state, a mild fatigue state, and a severe fatigue state. The stress state category describes the user's emergency response pattern when facing sudden or abnormal situations, and can include stressed and non-stressed states. A stressed state is characterized by high activation of the sympathetic nervous system, accompanied by physiological characteristics such as a sudden increase in heart rate, a sharp increase in skin conductance, and high concentration of attention. The stress state category describes the user's subjective experience of tension under continuous demand and environmental resource imbalance, and can include low stress, moderate stress, and high stress.

[0042] In some embodiments, before identifying the user's human factors status category, the human factors data can be preprocessed, such as noise filtering, baseline correction, feature extraction, and data normalization. The preprocessed human factors data is then input into the trained human factors status recognition model to obtain the human factors status category.

[0043] The human condition recognition model is a classification model built based on machine learning or deep learning algorithms. It is pre-trained with a large number of labeled samples, which include human factor data of different users under various interaction intentions and corresponding human factor condition category labels. Human factor condition category labels can include emotional state category, cognitive load category, fatigue state category, stress state category, and pressure state category. During training, the human factor condition recognition model can learn the correlation between different human factor data and human factor condition categories, so that when new human factor data is input, it can output the user's current human factor condition category and confidence level.

[0044] Step 130: Obtain user instructions and fuse user instructions with human factor state categories to obtain fused perception information.

[0045] In this embodiment, user commands are surface-level interaction signals output during the interaction between the user and the embodied intelligent device. User commands can include, but are not limited to, various human-computer interaction commands such as voice commands, touch commands, gesture commands, and visual interaction commands. Since a single user command can only reflect the user's surface-level operational needs and cannot reflect the user's current physical state, emotional preferences, and scene adaptation needs, it is easy to lead to inaccurate interaction responses. Therefore, it is necessary to combine information fusion based on the user's human condition category.

[0046] In some embodiments, user instructions can be semantically parsed to extract elements such as task objectives, operation objects, and constraints, resulting in a semantic feature vector. The identified human state categories are then transformed into structured state feature vectors, where each dimension corresponds to the confidence level of the emotional state category, cognitive load category, fatigue state category, stress state category, and pressure state category, respectively. The instruction semantic features and state feature vectors can be jointly encoded, such as through dimension concatenation and weight matching, to form fused perceptual information.

[0047] Step 140: Generate a response strategy corresponding to the user command based on the fused perception information, so that the embodied intelligent device can perform interaction based on the response strategy.

[0048] In this embodiment of the disclosure, the response strategy is an interaction rule formulated by the embodied intelligent device for the fusion of perception information. After the response strategy is generated, the embodied intelligent device can call the interaction execution logic corresponding to the response strategy and drive the execution module to complete the corresponding human-computer interaction actions, such as voice feedback, interface interaction, function execution, body interaction feedback and other types of interactive behaviors.

[0049] In some embodiments, fused sensing information can be input into a policy generation model to obtain the response policy output by the policy generation model.

[0050] Policy generation models are built upon machine learning or deep learning algorithms. For example, they can employ policy networks based on the Transformer architecture or a hybrid architecture combining graph neural networks and rule engines. Extensive historical interaction data can be collected to train the policy generation model. This historical data includes fused perceptual information under different intentions and corresponding labeled response policies. During training, the policy generation model learns the correlation between user intention, human state category, and response policy within the fused perceptual information, enabling it to generate response policies that match the user's true intentions when new fused perceptual information is input.

[0051] In a specific example, the fused perceptual information includes the user command "play music" and the human condition category "high stress state". The policy generation model can associate the user command with the human condition category to generate a "relaxing" response strategy, such as selecting soothing audio content, reducing the number of interaction confirmation steps, and controlling the playback volume to a moderate level.

[0052] For the same user command, the response strategy will differ depending on the user's state category. For example, if the user's emotion is "sad," the response strategy will be a concise and reassuring one to reduce redundant information and add empathetic expression; if the user's emotion is "happy," the response strategy will be a lively and detailed one to increase interactivity and expand relevant content; if the user's emotion is "neutral," the response strategy will be a neutral and clear one to maintain logical rigor.

[0053] The human-computer interaction method for embodied intelligent devices based on human condition perception disclosed herein acquires human condition data of users interacting with the embodied intelligent device, identifies the user's human condition category based on the human condition data, and integrates user commands with human condition categories to generate response strategies. This enables the embodied intelligent device to not only consider the user's surface interaction information when performing interactions, but also incorporate human condition information, thereby gaining a deeper understanding of the user's real needs and emotional state, adaptively adjusting and optimizing the interaction logic, adapting to the real needs of different users in different situations, achieving more user-friendly intelligent interaction, and thus improving the accuracy of the embodied intelligent device's decision-making.

[0054] In some embodiments, identifying a user's human factors status category based on human factors data includes: When the human factors data is unimodal, the human factors data is input into the corresponding human factors status recognition model to obtain the user's human factors status category; When the human factors data is multimodal, the human factors data is input into the corresponding multimodal human factors state recognition model to obtain the user's human factors state category; Alternatively, when the human factors data is multimodal, the human factors data is input into the human factors state recognition model corresponding to each modality to obtain the human factors state categories of multiple users; the human factors state categories of multiple users are analyzed and decisions are made to obtain the human factors state category of the final user.

[0055] In this embodiment, single-modal data refers to human factors data collected by a single type of sensor, such as human factors data containing only electroencephalogram (EEG) data or only skin conductance data.

[0056] Human state recognition models are classification models built based on machine learning or deep learning algorithms. They learn the mapping relationship between single-modal human factors data and corresponding human state categories. For example, when the input data is EEG data, the human state recognition model can extract features such as alpha wave inhibition rate and amplitude. After feature transformation through fully connected layers or recurrent neural network layers, the output layer outputs the human state category and confidence score. The human state recognition model is pre-trained with a large number of single-modal labeled samples, which include single-modal human factors data and corresponding human state category labels from different users under various intentions.

[0057] In this embodiment, multimodal data refers to human factors data collected by two or more different types of sensors, such as simultaneous EEG data and heart rate data, or simultaneous EEG data, skin conductance data, and eye movement data.

[0058] Multimodal human factor state recognition models are built upon a multimodal fusion deep learning architecture, capable of receiving human factor data from different modalities. For example, a multimodal human factor state recognition model can employ a multimodal Transformer architecture based on a cross-attention mechanism. In this architecture, each modality's data undergoes independent modal encoders to extract modality-specific features. Intermodal information interaction and fusion are then achieved through a cross-modal attention layer, and finally, a joint classification layer outputs the human factor state category and confidence level. Furthermore, the multimodal human factor state recognition model is pre-trained end-to-end using a large number of labeled multimodal samples.

[0059] The human condition recognition model corresponding to each modality is a classification model with a structure similar to that of the single-modality model. For example, the heart rate modality corresponds to the heart rate human condition recognition model, the EEG modality corresponds to the EEG human condition recognition model, and the eye movement modality corresponds to the eye movement human condition recognition model.

[0060] After obtaining the output results of the human condition recognition model corresponding to each modality, the recognition results of each modality can be combined to obtain the human condition category.

[0061] In some embodiments, a weighted voting method can be used, which assigns weights based on the historical recognition accuracy of each modality and performs weighted voting on the human state category output by each modality.

[0062] In some embodiments, a probabilistic fusion method can be used to weighted average the probability distributions of human factor state categories output by each modality, generate a comprehensive probability distribution, and then take the category corresponding to the highest probability as the human factor state category. Of course, other methods can also be used to fuse the output results of the human factor state recognition models corresponding to each modality to obtain the human factor state category, and this disclosure does not limit this approach.

[0063] In this embodiment, corresponding human factor status recognition methods are matched for single-modal and multi-modal human factor data formats. For single-modal human factor data, a single corresponding human factor status recognition model is used for recognition. For multi-modal human factor data, recognition can be performed through a unified multi-modal model or by recognizing each modality independently. This approach can adapt to different human factor data types in different collection scenarios and improve the flexibility of user human factor status category recognition.

[0064] In some embodiments, user instructions include voice instructions and / or action instructions; By fusing user commands and human-related state categories, fused perception information is obtained, including: The user command is parsed and converted to obtain the first text information; The human condition category is transformed to obtain the second text information; By fusing the first text information and the second text information, fused perception information is obtained.

[0065] In this embodiment, user instructions may include one or more of voice instructions and action instructions. Voice instructions are interactive intentions expressed by the user through voice input to the embodied smart device, such as the user saying "Please get me a glass of water"; action instructions are interactive intentions expressed by the user through body movements or gestures to the embodied smart device, such as the user waving or making a stop gesture.

[0066] If the user command is a voice command, the voice signal can be parsed and converted into text using a speech recognition model to obtain the first text information. If the user command is a gesture command, the user's body movements can be parsed using a gesture recognition model, and the recognized gesture semantic labels can be converted into descriptive text to obtain the first text information. For example, if a user makes a "waving" gesture, the gesture recognition model outputs the gesture category label "waving to indicate approach," which, after text conversion, yields the first text information "user waving to indicate approach." If the user command includes both voice and gesture commands, the voice and gesture commands are converted into text respectively, and then concatenated and combined according to temporal sequence or semantic association to form the first text information.

[0067] In this embodiment, the human state category and confidence level can be converted into structured descriptive text to obtain the second text information. For example, when the identification result is the emotional state "anxiety" (confidence level 0.85), the cognitive load state "high" (confidence level 0.90), and the stress state "high pressure" (confidence level 0.80), the converted second text information is "The user's current emotional state is anxiety, cognitive load is high, and the user is in a high-pressure state".

[0068] In this embodiment, the first text information and the second text information can be fused by text splicing or text embedding fusion to obtain fused perception information.

[0069] In this embodiment, by converting user commands into first text information and human condition categories into second text information, and then fusing the first text information and the second text information to obtain fused perception information, it is possible to adapt to user commands of different modalities such as voice, gestures, and postures, and to fuse user commands of different modalities with human condition categories in the same semantic space, thereby improving the quality of multimodal information fusion.

[0070] Furthermore, by integrating the first and second text information, it not only considers the user's current human condition category but also incorporates user commands. This allows it to provide feedback on user commands while understanding the user's true state. For example, when a user is identified as being in an anxious state, if a regular query command is received, a gentle and soothing response strategy can be generated to calm the user's emotions.

[0071] In some embodiments, the first text information and the second text information are fused to obtain fused perceptual information, including: By concatenating the first and second text information, fused perception information is obtained.

[0072] In this embodiment, the first text information and the second text information can be combined according to a preset template, such as combining the first text information and the second text information according to the template "User instruction: [first text information]; User status: [second text information]", to form fused perception information. In a specific example, the obtained fused perception information can be: User status: sad; User instruction: I'm a little tired today and want to finish the task quickly.

[0073] In some embodiments, the first text information and the second text information can be directly concatenated, with the first text information placed before or after the second text information, thereby obtaining fused perception information.

[0074] In this embodiment, the fused perception information is obtained by concatenating the first text information and the second text information, which simplifies the fusion process of user instructions and human condition categories and improves the fusion efficiency of fused perception information.

[0075] In some embodiments, a response strategy corresponding to a user command is generated based on fused sensing information, including: The fused perception information is input into a pre-trained language model so that the language model can output a response strategy that matches the human cause state category based on the user instructions in the fused perception information.

[0076] The language model is a generative model built on the Large Language Model (LLM) architecture. It can receive fused perceptual information in the form of natural language as input and generate response strategy text adapted to the current intent based on the constraints of user command semantics and human state categories. The language model is pre-trained on large-scale general text corpora and professional corpora in the interaction domain, and has powerful semantic understanding, contextual reasoning and text generation capabilities.

[0077] In this embodiment, prompts can be used to standardize the expected output of the language model, making the language model clearly define the task objectives and output constraints. Based on user instructions fused with perceptual information, the language model can output response strategies that match the user's state category. Through emotion-response mapping rules, the language model associates and constrains the user's state category with response style, content depth, and interaction form, ensuring that the generated response strategy not only executes user instructions but also adapts to the user's physical and mental state.

[0078] For example, if the emotional state in the human condition category is "sad" or the stress state is "high-pressure", the language model can output concise and reassuring voice response text as a response strategy to reduce redundant information load and add empathetic expressions and emotional support statements; if the emotional state is "happy" or the stress state is "non-stressful", the language model outputs lively and detailed voice response text as a response strategy to increase interactive questions and related content expansion; if the emotional state is "neutral", the language model outputs neutral and clear voice response text as a response strategy to maintain logical rigor and information accuracy.

[0079] In a specific example, given the fused perception information: user state: sad; user instruction: I'm a little tired today, the language model can generate a voice response text: "You sound like you really have a hard day! Shall we take a break before continuing to study?" It can also generate action response code to trigger the embodied smart device to perform a gentle nodding action.

[0080] In this embodiment, by inputting fused perception information into the language model, the powerful semantic understanding and reasoning capabilities of the language model can be fully utilized to conduct in-depth correlation analysis between user command content and human state categories. This allows the language model to adjust the output content based on the user's state, generating more personalized responses that better meet the user's real needs, thereby improving the user experience.

[0081] In some embodiments, the response strategy includes an action response; the method further includes: The language model retrieves a pre-defined action library based on fused perceptual information and obtains at least one action that matches the human state category, wherein the actions in the pre-defined action library are associated with the human state category. The action response is based on at least one matched action.

[0082] In this embodiment, motion response is the interactive feedback output by the embodied smart device to the user through limb movements or posture changes, such as performing a nod, a dance move, or a stop gesture.

[0083] In this embodiment, the action library is a pre-built set of structured actions that defines multiple actions that the embodied smart device can perform. Each action may include fields such as action identifier, action name, action description, applicable intent, corresponding emotion, and action parameter template. For example, the action library includes "gentle nod," with the corresponding fields: gentle nod, applicable intent: soothing, listening, corresponding emotion: sadness, anxiety, and action parameters: amplitude ±5 degrees, frequency 0.5Hz, and duration 2-5 seconds.

[0084] In some embodiments, the language model can parse and fuse perceptual information to extract human state categories, such as emotional state, stress state, and anxiety state labels. Based on these human state labels, it can retrieve actions with corresponding emotions from an action library to form a first candidate set. For example, if the human state category indicates that the user is in a "sad" state, the language model can retrieve actions from the action library that match the emotion "sad" to form a first candidate set.

[0085] Language models can parse and fuse user command semantics from perceived information to determine the user's current interaction intent type, such as requesting reassurance, requesting interaction, requesting space or task collaboration, etc. Based on the interaction intent type, actions matching the interaction intent type are selected from the first candidate set to form a second candidate set. For example, if the user command is "Don't come any closer," the language model can select actions with applicable intents including "avoidance," such as the "turn around" action, from the first candidate set to form a second candidate set.

[0086] The language model can determine at least one action as the response strategy output from the second-level candidate set. For example, if the user's instruction is clear and the human condition is stable, a single action can be output. If the user's instruction is ambiguous or the human condition is complex, the language model can output a combination of action sequences, such as first performing "gentle nod" to express respect, and then performing "turn around" to respond to the request.

[0087] It should be noted that the response strategy output by the language model can include structured action instructions, which contain information such as action identifiers, action parameter configurations, and execution timing. These structured action instructions can be executable code from the motion mechanism of the embodied intelligent device, enabling the device to drive actuators such as servo motors and joint modules to complete corresponding physical actions.

[0088] In this embodiment, by inputting fused perception information into a language model, the language model matches at least one action from a predefined action library based on the fused perception information to generate an action response strategy. This enables the language model to perform joint reasoning of user instructions and human state in a high-level semantic space, and to map the reasoning results to actions or action combinations with clear semantic labels in the action library. This reduces the complexity of the language model in generating physically executable strategies and the risk of hallucinations, further improving the accuracy of decision-making by embodied intelligent devices.

[0089] In some embodiments, the response strategy includes a voice response; the method further includes: The language model retrieves a pre-set timbre library based on fused perceptual information and obtains the target timbre corresponding to the human condition category. The timbres in the pre-set timbre library correspond to the human condition category. Speech response based on target timbre.

[0090] In this embodiment, voice response is the interactive feedback output by the embodied smart device to the user via voice, such as playing soothing words or prompts through a speaker.

[0091] The timbre library defines a variety of timbres, each corresponding to different timbre configurations, such as fundamental frequency range, speech rate, volume dynamics, formant structure, and emotional prosody. For example, the timbre library includes "mild timbre" (corresponding to a low fundamental frequency, slow speech rate, soft volume, and rounded formants, suitable for negative emotional states such as sadness and anxiety), "lively timbre" (corresponding to a high fundamental frequency, brisk speech rate, full volume, and bright formants, suitable for positive emotional states such as happiness and excitement), and "steady timbre" (corresponding to a moderate fundamental frequency, stable speech rate, constant volume, and neutral formants, suitable for neutral emotions and normal cognitive load).

[0092] The timbre library can also store the mapping relationship between different timbres and different human state categories. For example, if the emotional state in the human state category is "sad" or "depressed", it corresponds to "gentle timbre"; if the emotional state in the human state category is "happy", it corresponds to "lively timbre", and so on.

[0093] The language model can parse the human state category from the fused perceptual information and retrieve the corresponding target timbre from the timbre library based on this mapping relationship. After determining the target timbre, the target timbre can be used to generate a speech response through a speaker, playing the corresponding speech content in the response strategy.

[0094] Those skilled in the art should understand that the aforementioned action response and voice response are not intended to limit the response strategy, and other methods for providing feedback to users are also within the scope of this disclosure, such as text response and image / cartoon response.

[0095] In this embodiment, by matching the target timbre corresponding to the human condition category and playing the voice response based on the target timbre, the acoustic features of the voice output can be adjusted according to the user's state, which improves the matching degree between the emotional expression of the embodied intelligent device and the user's needs, and further improves the accuracy of the embodied intelligent device's decision-making.

[0096] In some embodiments, a response strategy corresponding to a user command is generated based on fused sensing information, including: Obtain data about the user's environment; Response strategies are generated based on environmental data and fused sensing information to correspond to user commands.

[0097] In this embodiment, during the acquisition of human factors data, environmental sensing devices can also be used to collect environmental data about the user's environment, such as ambient light intensity, ambient noise levels (decibels), ambient temperature, and humidity. This environmental data can be collected through sensors integrated into the smart device itself, or through independent IoT sensor nodes.

[0098] Response strategies corresponding to user commands can be generated based on fused perception information and environmental data. For example, fused perception information and environmental data can be input into a language model or policy generation model to obtain the response strategy output by the model.

[0099] In this embodiment, the generation of the response strategy not only uses fused perception information but also takes into account environmental data, reducing the interference of environmental factors on human perception. This makes the response strategy more adaptable to the current environment and more in line with the user's real needs, further improving the accuracy of decision-making by the embodied intelligent device.

[0100] In some embodiments, the human-computer interaction method for embodied intelligent devices based on human state perception further includes: During the interaction process performed by the embodied intelligent device based on the response strategy, the user's human factors data is collected to identify the human factors state after the interaction; Response strategy based on human-centric state updates after interaction.

[0101] During interactions performed by embodied intelligent devices based on response strategies, wearable devices can collect users' human factors data and identify human factors state categories. After interaction with embodied intelligent devices, the user's human factors state category may change. For example, if the response strategy is a soothing voice and a gentle nodding motion, the human factors state category after the interaction may change to a decrease in user anxiety and a drop in stress level, or it may remain at a high-pressure or high-stress level.

[0102] After identifying the human condition state after the interaction, the response strategy can be updated based on the post-interaction human condition state. In some embodiments, a difference vector of the human condition state before and after the interaction can be calculated to determine the magnitude and direction of change in the human condition state category. For example, if the emotional state before the interaction is "anxious" (confidence 0.85) and the emotional state after the interaction is "mild anxiety" (confidence 0.60), then the improvement in emotional state is positive, and the magnitude of the change is from anxiety to mild anxiety, indicating that the current response strategy is effective.

[0103] The response strategy can be updated based on the difference vector. For example, the difference vector can be converted into text, concatenated with the human condition state category after the interaction, and then input into the language model to update the response strategy.

[0104] In some embodiments, user instructions, the human-cause state category before interaction, the executed response strategy, and the human-cause state category after interaction can be input into the language model to generate an updated response strategy.

[0105] In this embodiment, by continuously collecting user human factors data during the interaction process performed by the embodied intelligent device based on the response strategy to identify the human factors state after the interaction, and updating the response strategy based on the human factors state after the interaction, the interaction process can adapt to changes in the user's state, thereby further improving the accuracy of the embodied intelligent device's decision-making.

[0106] Figure 2 Examples of scenarios provided in this disclosure's embodiments, such as Figure 2 As shown, taking smart education as an example, students, or users, can wake up the embodied intelligent device, i.e., the teaching robot, to initiate the interactive process. The wake-up operation can be achieved through voice wake-up words, touch buttons, or near-field sensing. After the robot is woken up, it acquires the user's voice commands, such as "Start today's lesson," "How do you pronounce this word?" or "I'm a little tired today and want to finish the task quickly"; and it also acquires human factors data of the user interacting with the embodied intelligent device through wearable devices or embedded sensors. Figure 2 As shown, human factors data includes multiple modalities such as electroencephalogram (EEG) data, near-infrared brain imaging data, physiological data, eye movement data, and facial image data.

[0107] After acquiring human factors data, the modal data can be input into the human factors state recognition model to identify the user's human factors state category. For example... Figure 2 As shown, the human condition recognition model includes models such as fatigue state assessment model, load state assessment model, stress state assessment model, pressure state assessment model and emotional state assessment model, each model corresponding to different dimensions of human condition recognition tasks.

[0108] The outputs of various models can be merged to obtain human condition categories, and these categories can be converted into structured text, i.e., second text information. In a specific example, the second text information could be "User fatigue state: mild fatigue; Load state: high cognitive load".

[0109] In this scenario example, the user's voice input command can be converted into first text information, which is then fused with second text information to obtain fused perceptual information. The fusion process can be carried out by text concatenation to form joint text that includes the semantics of the user command and the category of human condition, such as "Human condition: sad; User command: I'm a little tired today and want to finish the task quickly".

[0110] After obtaining the fused perception information, the fused perception information can be transmitted to the language model. For example, the fused perception information can be transmitted to a server that has deployed the language model, or to a server with a language model interface, so that the language model can generate a response strategy corresponding to the user command based on the fused perception information.

[0111] Language models can analyze and fuse perceptual information to determine whether the current user is in a suitable mental and physical state to continue learning. For example... Figure 2 As shown, if the human condition category indicates that the user is in an unsuitable state such as fatigue, high pressure, or negative emotions, the response strategy output by the language model can include a non-teaching response, such as "tell a joke in Chinese and let the user rest for one minute" to alleviate the user's emotions; if the human condition category indicates that the user is in an acceptable learning state, such as a neutral or positive emotion, the language model outputs response content related to the teaching task, such as generating a word teaching sentence for the instruction "how to pronounce this word".

[0112] After determining the output content, the language model further integrates word teaching statements with action instructions to output a structured response strategy. The action instructions are generated by the language model based on fused perceptual information, matching at least one action from an action library.

[0113] like Figure 2As shown, the response strategy can be encapsulated in JSON format, including "action" and "response" fields. The "action" field defines the action code executed by the robot, and the "response" field stores the text of the voice response. For example, the response strategy could be {"action": [point_to_word, gentle_nod], "response": "This word is pronounced 'apple', repeat after me: apple"}, where "point_to_word" indicates the action of pointing to the word card, and "gentle_nod" indicates an encouraging nod action.

[0114] After parsing the response strategy, the robot transmits the text in the "response" field to the speech synthesis module, which selects the appropriate timbre parameters based on the human state category for playback. It also generates control commands based on the action code in the "action" field and transmits them to the motion control module to drive the robotic arm, head servo motor, chassis and other actuators to perform the corresponding actions.

[0115] like Figure 2 As shown, the robot's interactive feedback is a combination of "voice response + adapted actions", such as simultaneously broadcasting the pronunciation of words and performing pointing and nodding actions.

[0116] The human-computer interaction method for embodied intelligent devices based on human-cause state perception provided in this disclosure can be executed by an embodied intelligent device human-computer interaction device based on human-cause state perception. This disclosure uses the execution of the human-computer interaction method based on human-cause state perception by an embodied intelligent device human-computer interaction device as an example to illustrate the embodied intelligent device human-computer interaction device based on human-cause state perception provided in this disclosure.

[0117] This disclosure also provides a human-computer interaction device for embodied intelligent devices based on human state perception.

[0118] like Figure 3 As shown, the embodied intelligent device human-computer interaction device based on human state perception includes: User acquisition module 310 is used to acquire human factors data of users who are interacting with the embodied smart device; The identification module 320 is used to identify the user's human factors status category based on human factors data, wherein the human factors status category is used to characterize the user's physical and mental state. The fusion module 330 is used to acquire user commands and fuse user commands with human factor state categories to obtain fused perception information; The response module 340 is used to generate a response strategy corresponding to the user command based on the fused sensing information, so that the embodied intelligent device can perform interaction based on the response strategy.

[0119] According to the embodied intelligent device human-computer interaction device based on human condition perception disclosed herein, by acquiring human condition data of users in the interaction state with the embodied intelligent device, and identifying the user's human condition category based on the human condition data, the user's instructions and human condition category are fused to generate a response strategy. This enables the embodied intelligent device to not only consider the user's surface interaction information when performing interaction, but also incorporate human condition information, thereby deeply understanding the user's real needs and emotional state, adaptively adjusting and optimizing the interaction logic, adapting to the real needs of different users in different situations, achieving intelligent interaction that is more in line with user experience, and thus improving the accuracy of the embodied intelligent device's decision-making.

[0120] In some embodiments, the identification module 320 is further configured to: When the human factors data is unimodal, the human factors data is input into the corresponding human factors status recognition model to obtain the user's human factors status category; When the human factors data is multimodal, the human factors data is input into the corresponding multimodal human factors state recognition model to obtain the user's human factors state category; Alternatively, when the human factors data is multimodal, the human factors data is input into the human factors state recognition model corresponding to each modality to obtain the human factors state categories of multiple users; the human factors state categories of multiple users are analyzed and decisions are made to obtain the human factors state category of the final user.

[0121] In some embodiments, the fusion module 330 is further configured to: By fusing user commands and human-related state categories, fused perception information is obtained, including: The user command is parsed and converted to obtain the first text information; The human condition category is transformed to obtain the second text information; By fusing the first text information and the second text information, fused perception information is obtained.

[0122] In some embodiments, the fusion module 330 is further configured to: By concatenating the first and second text information, fused perception information is obtained.

[0123] In some embodiments, the response module 340 is further configured to: The fused perception information is input into a pre-trained language model so that the language model can output a response strategy that matches the human cause state category based on the user instructions in the fused perception information.

[0124] In some embodiments, the response module 340 is further configured to: The language model retrieves a pre-defined action library based on fused perceptual information and obtains at least one action that matches the human state category, wherein the actions in the pre-defined action library are associated with the human state category. The action response is based on at least one matched action.

[0125] In some embodiments, the response module 340 is further configured to: The language model retrieves a pre-set timbre library based on fused perceptual information and obtains the target timbre corresponding to the human condition category. The timbres in the pre-set timbre library correspond to the human condition category. Speech response based on target timbre.

[0126] In some embodiments, the response module 340 is further configured to: Obtain data about the user's environment; Response strategies are generated based on environmental data and fused sensing information to correspond to user commands.

[0127] In some embodiments, the response module 340 is further configured to: During the interaction process performed by the embodied intelligent device based on the response strategy, the user's human factors data is collected to identify the human factors state after the interaction; Response strategy based on human-centric state updates after interaction.

[0128] The human-computer interaction device based on human state perception in this disclosure can be an edge computing device or a component within an edge computing device, such as an integrated circuit or a chip. The edge computing device can be a terminal or other devices besides a terminal. For example, the edge computing device can be a wearable device (such as a smartwatch or smart bracelet), an eye tracker, an AR (Augmented Reality) / VR (Virtual Reality) / XR (Extended Reality) / MR (Mixed Reality) device, an edge server, an edge gateway, or a cloud computing device; this disclosure does not impose specific limitations.

[0129] In an optional example, deploying an embodied intelligent device based on human state perception as an edge computing device or component on the edge deployment side can migrate at least some computing tasks to the edge device, leveraging the advantages of edge computing to improve data processing efficiency, reduce latency and network resource consumption and dependence, and also benefit data security and privacy protection.

[0130] The human-computer interaction device based on human state perception in this embodiment can be a device with an operating system. This operating system can be Microsoft Windows, Android, iOS, or other possible operating systems; this embodiment does not impose specific limitations.

[0131] In some embodiments, such as Figure 4 As shown, this disclosure also provides a human-computer interaction system for embodied intelligent devices based on human state perception, including: Data acquisition device 401 is used to collect users' human factors data; Edge computing device 402 is used to identify the user's human factors status category based on human factors data; The embodied intelligent device 403 is used to fuse user commands and human condition categories to obtain fused perception information; generate response strategies corresponding to user commands based on the fused perception information; and execute interactions based on the response strategies.

[0132] In some embodiments, the data acquisition device 401 may be a wearable device, a camera, a microphone, or other devices. For example, a user may wear a wearable device that can collect the user's human factors data and transmit the human factors data to an edge computing device 402. The edge computing device 402 may identify the human factors state category through a human factors state recognition model and transmit the identification result of the human factors state category to an embodied smart device 403.

[0133] The embodied intelligent device 403 can acquire user commands, recognition results of human condition categories, and environmental data related to the user interaction scenario, such as ambient light intensity, ambient noise level, ambient temperature, and humidity. The embodied intelligent device 403 can fuse the acquired data, generate a response strategy through a language model, and execute the interaction according to the response strategy.

[0134] In some embodiments, such as Figure 5 As shown, this disclosure also provides an edge computing device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, it implements the various processes of the above-described embodiment of the human-computer interaction method for embodied intelligent devices based on human state perception, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0135] This disclosure also provides a computer-readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements the various processes of the above-described embodiments of the human-computer interaction method for embodied intelligent devices based on human state perception, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0136] The processor is the processor in the edge computing device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0137] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described human-computer interaction method for embodied intelligent devices based on human state perception.

[0138] The processor is the processor in the edge computing device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0139] This disclosure also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the human-computer interaction method for embodied intelligent devices based on human state perception, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0140] It should be understood that the chip mentioned in the embodiments of this disclosure may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0141] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this disclosure is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0143] The embodiments of this disclosure have been described above with reference to the accompanying drawings. However, this disclosure is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this disclosure without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this disclosure.

[0144] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0145] Although embodiments of this disclosure have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A human-computer interaction method for embodied intelligent devices based on human state perception, characterized in that, include: Acquire human factors data of users who are interacting with embodied smart devices; The user's human factors status category is identified based on the aforementioned human factors data, wherein the human factors status category is used to characterize the user's physical and mental state. The user command is acquired, and the user command and the human condition category are fused to obtain fused perception information; Based on the fused perception information, a response strategy corresponding to the user command is generated, so that the embodied intelligent device can perform interaction based on the response strategy.

2. The method according to claim 1, characterized in that, The process of identifying user's human factors status category based on the human factors data includes: When the human factors data is unimodal data, the human factors data is input into the corresponding human factors state recognition model to obtain the user's human factors state category; When the human factors data is multimodal data, the human factors data is input into the corresponding multimodal human factors state recognition model to obtain the user's human factors state category; Alternatively, when the human factors data is multimodal data, the human factors data is input into the human factors state recognition model corresponding to each modality to obtain the human factors state categories of multiple users; the human factors state categories of the multiple users are analyzed and decided to obtain the human factors state category of the final user.

3. The method according to claim 1, characterized in that, The user instructions include voice instructions and / or action instructions; The step of fusing the user command and the human condition category to obtain fused perception information includes: The user command is parsed and converted to obtain the first text information; The human condition category is converted to obtain the second text information; The first text information and the second text information are fused to obtain fused perception information.

4. The method according to claim 3, characterized in that, The process of fusing the first text information and the second text information to obtain fused perception information includes: The first text information and the second text information are concatenated to obtain the fused perception information.

5. The method according to claim 1, characterized in that, The step of generating the response strategy corresponding to the user command based on the fused sensing information includes: The fused perception information is input into a pre-trained language model so that the language model outputs a response strategy that matches the human condition category based on the user instructions in the fused perception information.

6. The method according to claim 5, characterized in that, The response strategy includes an action response; the method further includes: The language model retrieves a preset action library based on the fused perceptual information to obtain at least one action that matches the human condition category, wherein the actions in the preset action library are associated with the human condition category. The action response is based on at least one of the matched actions.

7. The method according to claim 5, characterized in that, The response strategy includes a voice response; the method further includes: The language model retrieves a preset timbre library based on the fused perceptual information to obtain a target timbre corresponding to the human condition category, wherein the timbres in the preset timbre library correspond to the human condition category; The speech response is based on the target timbre.

8. The method according to claim 1, characterized in that, The step of generating the response strategy corresponding to the user command based on the fused sensing information includes: Obtain the environmental data of the user; The response strategy corresponding to the user command is generated based on the environmental data and the fused perception information.

9. The method according to claim 1, characterized in that, The method further includes: During the interaction process performed by the embodied intelligent device based on the response strategy, the user's human factors data is collected to identify the human factors state after the interaction. The response strategy is updated based on the human condition after the interaction.

10. A human-computer interaction device for embodied intelligent devices based on human state perception, characterized in that, include: The user data acquisition module is used to acquire human factors data of users who are interacting with the embodied smart device. The identification module is used to identify the user's human factors status category based on the human factors data, wherein the human factors status category is used to characterize the user's physical and mental state. The fusion module is used to acquire user commands and fuse the user commands with the human condition category to obtain fused perception information; The response module is used to generate a response strategy corresponding to the user command based on the fused perception information, so that the embodied intelligent device can perform interaction based on the response strategy.

11. An edge computing device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 9.

Images