Robot interaction method, system, device and medium based on expression recognition

Abstract

The present application relates to the technical field of robot interaction, and provides a robot interaction method, system, device and medium based on expression recognition, comprising: acquiring facial images, physiological signs and voice information of an interaction object, collecting historical interaction data and corresponding object signs, feedback results and scene interference levels; performing personalized screening on the historical interaction data according to the physiological signs and the object signs to obtain target interaction data; constructing a preset expression interaction model of multi-modal fusion and performing iterative training; cooperatively processing the currently collected information to obtain real-time expression states, emotional intensity levels and interaction intention data of the interaction object; inputting into the expression interaction model for matching recognition, dynamically adapting and adjusting the interaction mode in combination with historical interaction preferences and the current scene interference level to obtain an adapted interaction response scheme; and controlling the robot to perform corresponding interaction actions. The present application improves the accuracy of emotion recognition and the adaptation degree of interaction response.

Description

Technical Field

[0001] This application relates to the technical field of robot interaction, and in particular to robot interaction methods, systems, devices and media based on facial expression recognition. Background Technology

[0002] With the rapid development of artificial intelligence technology, service robots are being used more and more widely in fields such as home, healthcare, and education. In the process of human-computer interaction, accurately understanding the user's emotional state and interaction intent is crucial for improving the user experience.

[0003] Existing robot interaction methods typically respond based on single voice commands or simple facial expression recognition. However, in real-world applications, users' facial expressions are easily affected by factors such as ambient lighting and occlusion, and relying solely on visual information is insufficient to accurately determine the user's true emotional intensity and physiological state. Furthermore, current technologies often lack in-depth analysis of users' personalized characteristics and historical interaction habits, resulting in low adaptability of interaction responses and an inability to meet users' personalized interaction needs in complex scenarios. Therefore, improving the accuracy of robots in recognizing users' emotional states and enabling adaptive interaction responses based on user characteristics and scene interference are pressing technical challenges that need to be addressed. Summary of the Invention

[0004] To address at least one of the aforementioned technical problems, this application provides a robot interaction method, system, device, and medium based on facial expression recognition.

[0005] Firstly, this application provides a robot interaction method based on facial expression recognition, which adopts the following technical solution:

[0006] A robot interaction method based on facial expression recognition, comprising:

[0007] Acquire facial image information, physiological sign information, and voice interaction information of the interactive object;

[0008] Collect historical interaction data, as well as the object characteristics information of the interactive objects, interaction feedback results, and scene interference levels in the historical interaction data;

[0009] Based on the physiological signs information and the object's signs information, the historical interaction data is screened in a personalized manner to obtain target interaction data that is compatible with the interaction object;

[0010] Based on the interaction feedback results, the target interaction data is improved, and multi-dimensional features are extracted from the improved target interaction data to obtain scene features and corresponding facial expression recognition features, voice features, physiological features and interaction response features.

[0011] A multimodal fusion preset facial expression interaction model is constructed. The preset facial expression interaction model includes an facial expression recognition branch, a speech parsing branch, and a physiological feature analysis branch. Based on the facial expression recognition features, the speech features, the physiological features, and the interaction response features, the preset facial expression interaction model is iteratively trained to obtain the trained facial expression interaction model.

[0012] Based on the facial expression recognition features, the voice features, the physiological features, and the interaction response features, the facial image information, the physiological sign information, and the voice interaction information are processed collaboratively to obtain the real-time facial expression state, emotion intensity level, and interaction intent data of the interactive object.

[0013] The real-time facial expression status, emotional intensity level, and interaction intent data are input into the facial expression interaction model for multi-dimensional matching and recognition to obtain the interaction matching result. The interaction matching result includes at least one candidate interaction response scheme and a fit score corresponding to at least one candidate interaction response scheme.

[0014] Determine whether there is an optimal interaction response scheme in the interaction matching results whose adaptation score is higher than the preset adaptation threshold. If there is, determine the interaction mode corresponding to the optimal interaction response scheme. Combine the historical interaction preferences of the interaction object and the interference level of the current scene, dynamically adapt and adjust the interaction mode to obtain the adapted interaction response scheme.

[0015] Based on the adapted interaction response scheme, the robot is controlled to perform the corresponding interactive actions.

[0016] By adopting the above technical solutions, a precise user profile is constructed by acquiring facial images, physiological characteristics, and voice information of the interactive object and combining them with historical interaction data for personalized screening. A multimodal fusion expression interaction model is used to comprehensively process multi-source information, effectively solving the problem of single-modal recognition being susceptible to environmental interference and occlusion, significantly improving the accuracy and robustness of emotion recognition. Through collaborative processing of real-time data, the user's real-time expression state, emotion intensity, and interaction intent are accurately obtained. Furthermore, the interaction mode is dynamically adapted and adjusted based on historical interaction preferences and scene interference levels, making the interaction response scheme more in line with the user's personalized needs and the characteristics of the current scene, greatly improving the naturalness of human-computer interaction and user satisfaction.

[0017] In a preferred embodiment, this application can be further configured as follows: the step of iteratively training the preset facial expression interaction model based on the facial expression recognition features, the speech features, the physiological features, and the interaction response features to obtain the trained facial expression interaction model includes:

[0018] Based on the interactive scene environment information and the scene interference level in the scene features, the historical interaction data is divided into different scene categories, and a set of facial expression recognition features, voice features, physiological features and interactive response features are obtained for each scene category.

[0019] Calculate the correlation strength between the facial expression recognition features, the voice features, and the physiological features; determine the collaborative correlation relationship corresponding to each scene category; and the corresponding mapping relationship between the collaborative correlation relationship and the interaction response features.

[0020] The facial expression recognition features of each scene category are finely classified and labeled to obtain facial expression category labels and emotion intensity labels;

[0021] Based on the aforementioned collaborative association and the corresponding mapping relationship, the facial expression recognition features, speech features, physiological features, facial expression category labels, emotion intensity labels, and interactive response features under each scene category are constructed into a training sample set for feature input and interactive response output;

[0022] The training sample set is input into the preset facial expression interaction model for iterative training, and the weights of the three feature branches of facial expression, voice and physiology are dynamically adjusted according to the interference level of the scene to obtain the trained preset facial expression interaction model.

[0023] The trained preset facial expression interaction model is validated by calculating the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate of the preset facial expression interaction model. Training is stopped when the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate all reach preset thresholds, thus obtaining the trained facial expression interaction model.

[0024] In a preferred embodiment, this application can be further configured as follows: the step of collaboratively processing the facial image information, the physiological sign information, and the voice interaction information based on the expression recognition features, the voice features, the physiological features, and the interaction response features to obtain the real-time expression state, emotion intensity level, and interaction intent data of the interactive object includes:

[0025] The acquired facial image information, physiological sign information, and voice interaction information are preprocessed to extract real-time expression features, real-time physiological features, and real-time voice features.

[0026] Based on the aforementioned collaborative association, real-time facial expression features are matched with facial expression recognition features, and cross-validation is performed using the aforementioned real-time physiological features to determine the real-time facial expression category and emotion intensity level.

[0027] The real-time expression category and the emotion intensity level are processed to track feature changes and obtain the real-time expression state.

[0028] Based on the real-time facial expressions and real-time physiological characteristics, semantic parsing is performed on the real-time speech features to obtain the interaction intent data of the interactive object.

[0029] In a preferred embodiment, this application can be further configured as follows: the step of tracking feature changes in the real-time expression category and the emotion intensity level to obtain the real-time expression state includes:

[0030] The similarity between the real-time facial expression features and the facial expression recognition features corresponding to each facial expression category label is calculated, and cross-validation is performed using real-time physiological features to determine the preliminary facial expression category and emotion intensity level.

[0031] The system tracks real-time facial expression features and real-time physiological feature changes within a preset time interval, judges the stability of the preliminary facial expression category and the emotional intensity level, and determines the real-time facial expression state of the interactive object.

[0032] In a preferred embodiment, this application can be further configured as follows: determining the interaction mode corresponding to the optimal interaction response scheme, and dynamically adapting and adjusting the interaction mode based on the historical interaction preferences of the interaction object and the current scene interference level to obtain an adapted interaction response scheme, including:

[0033] The historical interaction preference data of the interactive object is obtained, and the preference patterns in the historical interaction preference data are mined based on the collaborative filtering algorithm to generate a personalized preference model;

[0034] Adjust the presentation of the interactive response based on the current interference level to obtain the initial presentation format;

[0035] Based on the personalized preference model, collaborative relationships and corresponding mapping relationships, the content details, initial presentation form and interaction rhythm of the interaction response scheme are adjusted to obtain the adjusted interaction response scheme;

[0036] The adjusted interaction response scheme is validated for its rationality and suitability to obtain the adapted interaction response scheme.

[0037] In a preferred embodiment, this application can be further configured as follows: the step of verifying the rationality and fit of the adjusted interaction response scheme to obtain the adapted interaction response scheme includes:

[0038] Extract the core content of the adjusted interaction response scheme and perform semantic matching with the interaction intent data to determine whether the priority of the core content is reasonable. If the priority of the core content is reasonable and the semantic matching degree is greater than or equal to the preset matching threshold, then determine whether the tone, response speed, body movements and real-time facial expression status and emotional intensity level of the interaction response scheme match.

[0039] If the tone, response speed, body movements and real-time facial expression status and emotional intensity level of the interaction response scheme are matched, then the degree of fit between the presentation form and content details of the interaction response scheme and the current scene interference level and historical interaction preferences is verified to obtain the degree of fit corresponding to the adjusted interaction response scheme.

[0040] Determine whether the fit meets the preset fit range. If it does, the interaction response scheme is deemed reasonable, and an adapted interaction response scheme is obtained. If any item fails to meet the standard, the content, tone, and presentation of the interaction response scheme are readjusted until all verification items pass.

[0041] Secondly, this application provides a robot interaction system based on facial expression recognition, employing the following technical solution:

[0042] A robot interaction system based on facial expression recognition, comprising:

[0043] The information acquisition module is used to acquire facial image information, physiological sign information, and voice interaction information of the interactive object;

[0044] The information collection module is used to collect historical interaction data, as well as the object characteristics information of the interactive objects, interaction feedback results, and scene interference levels in the historical interaction data;

[0045] The data screening module is used to perform personalized screening of the historical interaction data based on the physiological signs information and the object's signs information to obtain target interaction data that is compatible with the interaction object.

[0046] The feature extraction module is used to perform data improvement processing on the target interaction data based on the interaction feedback results, and to perform multi-dimensional feature extraction on the improved target interaction data to obtain scene features and corresponding facial expression recognition features, voice features, physiological features and interaction response features.

[0047] The model building module is used to build a multimodal fusion preset facial expression interaction model. The preset facial expression interaction model includes a facial expression recognition branch, a speech parsing branch, and a physiological feature analysis branch. Based on the facial expression recognition features, the speech features, the physiological features, and the interaction response features, the preset facial expression interaction model is iteratively trained to obtain the trained facial expression interaction model.

[0048] The collaborative processing module is used to collaboratively process the facial image information, the physiological sign information and the voice interaction information based on the facial expression recognition features, the voice features, the physiological features and the interaction response features, to obtain the real-time facial expression status, emotion intensity level and interaction intent data of the interactive object.

[0049] The matching and recognition module is used to input the real-time facial expression state, emotion intensity level and interaction intent data into the facial expression interaction model for multi-dimensional matching and recognition, and to obtain the interaction matching result. The interaction matching result includes at least one candidate interaction response scheme and a fit score corresponding to at least one candidate interaction response scheme.

[0050] The adaptation adjustment module is used to determine whether there is an optimal interaction response scheme with an adaptation score higher than a preset adaptation threshold in the interaction matching results. If there is, the interaction mode corresponding to the optimal interaction response scheme is determined. The interaction mode is dynamically adapted and adjusted in combination with the historical interaction preferences of the interaction object and the interference level of the current scene to obtain the adapted interaction response scheme.

[0051] The control execution module is used to control the robot to perform corresponding interactive actions according to the adapted interaction response scheme.

[0052] In one possible implementation, when the model building module iteratively trains the preset facial expression interaction model based on the facial expression recognition features, the speech features, the physiological features, and the interaction response features to obtain the trained facial expression interaction model, it is specifically used for:

[0053] Based on the interactive scene environment information and the scene interference level in the scene features, the historical interaction data is divided into different scene categories, and a set of facial expression recognition features, voice features, physiological features and interactive response features are obtained for each scene category.

[0054] Calculate the correlation strength between the facial expression recognition features, the voice features, and the physiological features; determine the collaborative correlation relationship corresponding to each scene category; and the corresponding mapping relationship between the collaborative correlation relationship and the interaction response features.

[0055] The facial expression recognition features of each scene category are finely classified and labeled to obtain facial expression category labels and emotion intensity labels;

[0056] Based on the aforementioned collaborative association and the corresponding mapping relationship, the facial expression recognition features, speech features, physiological features, facial expression category labels, emotion intensity labels, and interactive response features under each scene category are constructed into a training sample set for feature input and interactive response output;

[0057] The training sample set is input into the preset facial expression interaction model for iterative training, and the weights of the three feature branches of facial expression, voice and physiology are dynamically adjusted according to the interference level of the scene to obtain the trained preset facial expression interaction model.

[0058] The trained preset facial expression interaction model is validated by calculating the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate of the preset facial expression interaction model. Training is stopped when the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate all reach preset thresholds, thus obtaining the trained facial expression interaction model.

[0059] Thirdly, this application provides an electronic device that adopts the following technical solution:

[0060] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described robot interaction method based on facial expression recognition.

[0061] Fourthly, this application provides a computer storage medium, as follows:

[0062] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described robot interaction method based on facial expression recognition.

[0063] In summary, this application has the following beneficial technical effects:

[0064] By acquiring facial images, physiological characteristics, and voice information of the interactive object and combining them with historical interaction data for personalized screening, an accurate user profile was constructed. A multimodal fusion facial expression interaction model was used to comprehensively process multi-source information, effectively solving the problem of single-modal recognition being susceptible to environmental interference and occlusion, significantly improving the accuracy and robustness of emotion recognition. Through collaborative processing of real-time data, the user's real-time facial expression state, emotional intensity, and interaction intent were accurately obtained. Furthermore, by dynamically adapting and adjusting the interaction mode based on historical interaction preferences and scene interference levels, the interaction response scheme became more in line with the user's personalized needs and the characteristics of the current scene, greatly improving the naturalness of human-computer interaction and user satisfaction. Attached Figure Description

[0065] Figure 1 This is a flowchart of a robot interaction method based on facial expression recognition in one embodiment of this application.

[0066] Figure 2This is another flowchart of a robot interaction method based on facial expression recognition, as described in one embodiment of this application.

[0067] Figure 3 This is a schematic diagram of the structure of a robot interaction system based on facial expression recognition, according to one embodiment of this application.

[0068] Figure 4 This is a schematic block diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0069] The following is in conjunction with the appendix Figure 1 To be continued Figure 4 This application will be described in further detail.

[0070] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0073] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0074] This application provides a robot interaction method based on facial expression recognition, executed by an electronic device. This electronic device can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication. This application does not impose any limitations on this. Figure 1 and Figure 2 As shown, the method includes:

[0075] Step S10: Obtain facial image information, physiological signs information, and voice interaction information of the interactive object.

[0076] Specifically, this embodiment uses visual sensors (such as RGB cameras and depth cameras) mounted on the robot to collect facial image information of the interactive object, and uses audio sensors (such as microphone arrays) to collect voice interaction information. In particular, this embodiment introduces the collection of physiological vital signs information, which can be obtained through non-contact sensors (such as millimeter-wave radar) or contact wearable devices (such as smart bracelets), including but not limited to data such as the interactive object's heart rate, heart rate variability, skin conductance, body temperature, or respiratory rate. It should be understood that traditional single-visual expression recognition is easily affected by changes in ambient lighting, facial occlusion (such as wearing a mask), or deliberate facial expressions by the user, leading to a decrease in recognition accuracy. This embodiment, by simultaneously acquiring physiological vital signs information, utilizes the characteristics of physiological indicators that are difficult to control by subjective consciousness and have strong anti-interference capabilities, providing an objective verification basis for subsequent expression recognition, thereby constructing a multi-dimensional perception foundation that is consistent with the actual expression.

[0077] Step S11: Collect historical interaction data and the object characteristics information of the interactive objects in the historical interaction data, the interaction feedback results, and the scene interference level.

[0078] Specifically, this application aims to address the problem of poor adaptability of general models to different individuals. The system compares the currently acquired physiological characteristics information with historically stored object characteristics information, filtering out historical interaction data that highly matches the physiological characteristics of the current interaction object. For example, it filters out historical interaction records of the user in similar heart rate zones or similar emotional states as target interaction data, thereby eliminating irrelevant data and improving the targeting and efficiency of model training.

[0079] Step S12: Based on physiological and object characteristics information, perform personalized screening of historical interaction data to obtain target interaction data that is compatible with the interaction object.

[0080] Step S13: Based on the interaction feedback results, perform data improvement processing on the target interaction data, and extract multi-dimensional features from the improved target interaction data to obtain scene features and corresponding facial expression recognition features, voice features, physiological features, and interaction response features.

[0081] In this embodiment of the application, during the data improvement process, the system interpolates or labels missing data to ensure data integrity. Subsequently, key feature vectors are extracted from the target interaction data using feature extraction algorithms (such as convolutional neural networks for image features, recurrent neural networks for speech features, and time-frequency analysis for physiological features). These feature vectors not only contain information about a single modality but also implicitly contain intermodal correlation features, laying the foundation for constructing a multimodal fusion model.

[0082] Step S14: Construct a multimodal fusion preset facial expression interaction model. Based on facial expression recognition features, speech features, physiological features, and interaction response features, iteratively train the preset facial expression interaction model to obtain the trained facial expression interaction model.

[0083] The preset facial expression interaction model includes facial expression recognition, speech parsing, and physiological feature analysis branches.

[0084] The model constructed in this embodiment adopts a multi-branch parallel processing architecture. The facial expression recognition branch processes facial image features, the speech parsing branch processes speech intonation features, and the physiological feature analysis branch specifically processes physiological data. During training, the system inputs the extracted features into the model and continuously adjusts the model parameters through the backpropagation algorithm, making the model output approximate realistic interactive response results. The advantage of this multimodal fusion architecture is that when the features of one branch are disturbed (e.g., the visual branch's features become blurred due to insufficient lighting), other branches (such as the physiological feature branch) can still provide high-confidence judgment criteria, thereby ensuring the overall robustness of the model.

[0085] Step S15: Based on facial expression recognition features, voice features, physiological features, and interactive response features, facial image information, physiological sign information, and voice interaction information are processed collaboratively to obtain the real-time facial expression status, emotion intensity level, and interactive intent data of the interactive object.

[0086] In this embodiment, the system does not simply process the three types of information in parallel, but rather performs cross-validation based on the correlation between features. For example, when the facial image shows a user with a "smiling" expression, but physiological feature analysis shows a rapid heart rate and strong skin conductance, the system will determine, based on collaborative processing logic, that the "smile" may be masking the true emotion of "tension" or "anxiety," thereby outputting a more accurate real-time facial expression state and emotion intensity level. Simultaneously, by combining semantic parsing of the voice content, the system further clarifies the user's interaction intent, such as "requesting help" or "indicating refusal."

[0087] Step S16: Input the real-time facial expression status, emotion intensity level, and interaction intent data into the facial expression interaction model for multi-dimensional matching and recognition to obtain the interaction matching result.

[0088] The interaction matching result includes at least one candidate interaction response scheme and a fit score corresponding to the at least one candidate interaction response scheme.

[0089] Specifically, the model searches a pre-stored knowledge base or a pre-trained parameter space based on the input real-time data and outputs several possible response solutions. Each solution is accompanied by a suitability score, which reflects the degree to which the solution matches the current user state and intent.

[0090] Step S17: Determine whether there is an optimal interaction response scheme with an adaptation score higher than the preset adaptation threshold in the interaction matching results. If so, determine the interaction mode corresponding to the optimal interaction response scheme. Combine the historical interaction preferences of the interaction object and the interference level of the current scene, dynamically adapt and adjust the interaction mode to obtain the adapted interaction response scheme.

[0091] Specifically, the system has a set adaptation threshold; only solutions with scores exceeding this threshold are considered. After selecting the optimal solution, this embodiment does not execute it directly. Instead, it dynamically adjusts the system based on historical interaction preferences (e.g., users prefer concise dialogue styles) and the current scene's interference level (e.g., noisy environment). For example, if the current scene has a high interference level (loud noise), the system automatically adjusts the interaction mode, changing the original voice broadcast to a higher volume, slower speech rate, or adding gesture prompts to ensure effective information delivery. This dynamic adaptation mechanism guarantees the flexibility and effectiveness of the interaction.

[0092] Step S18: Control the robot to perform the corresponding interactive actions according to the adapted interactive response scheme.

[0093] Ultimately, the robot's main control system generates control commands based on the adapted solution, driving the speaker to play voice, the display screen to show images, or the robotic arm to perform limb movements to complete the interaction.

[0094] In this embodiment, facial images, physiological characteristics, and voice information of the interactive object are acquired, and personalized screening is performed in conjunction with historical interaction data to construct an accurate user profile. A multimodal fusion facial expression interaction model is used to comprehensively process multi-source information, effectively solving the problem that single-modal recognition is susceptible to environmental interference and occlusion, significantly improving the accuracy and robustness of emotion recognition. Through collaborative processing of real-time data, the user's real-time facial expression state, emotional intensity, and interaction intent are accurately obtained. Furthermore, the interaction mode is dynamically adapted and adjusted based on historical interaction preferences and scene interference levels, making the interaction response scheme more in line with the user's personalized needs and current scene characteristics, greatly improving the naturalness of human-computer interaction and user satisfaction.

[0095] Furthermore, during the iterative training of the preset facial expression interaction model, based on facial expression recognition features, speech features, physiological features, and interaction response features, the preset facial expression interaction model is iteratively trained to obtain the trained facial expression interaction model. This includes: dividing historical interaction data into different scene categories according to the interaction scene environment information and scene interference level in the scene features, obtaining a set of facial expression recognition features, speech features, physiological features, and interaction response features corresponding to each scene category; calculating the correlation strength between facial expression recognition features, speech features, and physiological features, determining the collaborative correlation relationship corresponding to each scene category, and the corresponding mapping relationship between the collaborative correlation relationship and the interaction response features; and performing refined classification and labeling of the facial expression recognition features of each scene category to obtain facial expression category labels and emotion strength. The system employs a multi-level labeling approach. Based on collaborative relationships and corresponding mappings, it constructs a training sample set for each scene category, comprising facial expression recognition features, speech features, physiological features, facial expression category labels, emotion intensity labels, and interactive response features. This training sample set is then input into a pre-defined facial expression interaction model for iterative training. The weights of the three feature branches (facial expression, speech, and physiological) are dynamically adjusted according to the scene interference level to obtain the trained pre-defined facial expression interaction model. The trained model is then validated by calculating the facial expression recognition accuracy, interactive intent matching accuracy, and response adaptation rate. Training stops when all three rates reach a preset threshold, resulting in the completed facial expression interaction model.

[0096] In this embodiment, the scene interference level is a quantitative assessment of the quality of the interactive environment. For example, when the ambient noise level exceeds a preset noise threshold (e.g., 60 decibels), it is determined to be a high noise interference level; when the ambient light intensity is lower than a preset light threshold, it is determined to be a low light interference level. The system classifies historical data into different categories such as "high-noise scene," "low-light scene," and "standard scene" based on these interference levels, ensuring that the model can learn the differences in feature distribution under different environments during training.

[0097] The calculation of association strength aims to quantify the degree of interdependence between different modal features in a specific scenario. For example, in a standard scenario, facial expressions are clear and speech signals are pure, resulting in a high association strength between facial expression features and speech features. However, in a high-noise scenario, the credibility of speech features decreases, and their association strength with facial expression features weakens accordingly. In this case, physiological features (such as heart rate variability) are unaffected by acoustic noise, leading to a relatively stronger association strength with facial expression features. The system determines the cooperative association relationships between features by calculating indicators such as Pearson correlation coefficients or mutual information, and establishes a mapping network between this relationship and interactive response features (such as user satisfaction feedback). This mechanism enables the model to understand which modal features should be relied upon more under what kind of interference, thereby making accurate judgments.

[0098] The annotations not only include basic "joy, anger, sorrow, and happiness" categories, but also introduce emotion intensity labels (such as "mild joy" and "extreme anger"), providing the model with more granular training supervision signals. Based on collaborative relationships and corresponding mapping relationships, the facial expression recognition features, speech features, physiological features, facial expression category labels, emotion intensity labels, and interaction response features under each scene category are constructed into a training sample set of feature inputs and interaction response outputs.

[0099] Traditional multimodal fusion typically employs fixed weights, while this embodiment introduces a dynamic weight adjustment mechanism based on scene interference levels. Specifically, during training, the model adaptively adjusts the weight coefficients of each branch in the loss function according to the scene interference level of the current input sample. For example, when the input sample belongs to a "high-noise scene," the system automatically reduces the weight coefficient of the speech parsing branch (e.g., from 0.4 to 0.1) while increasing the weight coefficient of the physiological feature analysis branch (e.g., from 0.2 to 0.5) to compensate for the recognition loss caused by the lack of speech information. This dynamic adjustment mechanism forces the model to learn to rely more on physiological features with strong anti-interference capabilities in harsh environments during training, thereby significantly improving the model's robustness in complex real-world scenarios.

[0100] Furthermore, based on facial expression recognition features, voice features, physiological features, and interactive response features, facial image information, physiological sign information, and voice interaction information are collaboratively processed to obtain the real-time facial expression state, emotion intensity level, and interactive intent data of the interactive object. This includes: preprocessing the acquired facial image information, physiological sign information, and voice interaction information to extract real-time facial expression features, real-time physiological features, and real-time voice features; matching real-time facial expression features with facial expression recognition features based on collaborative correlation, and cross-validating with real-time physiological features to determine the real-time facial expression category and emotion intensity level; tracking feature changes in the real-time facial expression category and emotion intensity level to obtain the real-time facial expression state; and performing semantic parsing of the real-time voice features based on the real-time facial expression state and real-time physiological features to obtain the interactive intent data of the interactive object.

[0101] In this application, the preprocessing includes denoising, face alignment, and illumination normalization of the facial image to eliminate the impact of ambient light changes on image quality; filtering of physiological characteristic information to remove artifacts caused by poor sensor contact or motion; and denoising and endpoint detection of voice interaction information to separate effective human voice segments. Through preprocessing, high-quality real-time feature vectors are extracted, laying the foundation for subsequent accurate recognition.

[0102] Traditional single-visual recognition is highly susceptible to user spoofing or environmental interference. This embodiment introduces a physiological feature cross-validation mechanism, utilizing the characteristic that physiological indicators (such as heart rate and skin conductance) are difficult to control by subjective consciousness to verify the visual recognition results. The specific logic is as follows:

[0103] The system first calculates the similarity between real-time facial expression features and the facial expression recognition features corresponding to pre-stored facial expression category labels, initially identifying one or more candidate facial expression categories. Then, it retrieves the standard range of physiological characteristics corresponding to that candidate category in the collaborative association relationship (e.g., a "happy" state typically corresponds to a stable or slightly elevated heart rate and normal skin conductance). The real-time collected physiological characteristics are compared with this standard range. If the real-time physiological characteristics fall within the standard range, the preliminary matching result is confirmed; if the real-time physiological characteristics significantly deviate from the standard range, a correction mechanism is triggered. For example, if the real-time facial expression features show the user as "smiling," but the real-time physiological characteristics show an excessively fast heart rate (e.g., exceeding 100 beats / min) and a strong skin conductance, this does not match the physiological characteristics of a "happy" state, but is more consistent with the physiological characteristics of "nervousness" or "anxiety." Based on this, the system will correct the real-time facial expression category to "nervousness" and adjust the emotion intensity level accordingly. This cross-validation mechanism effectively solves the recognition problem in complex scenarios such as users "forcing a smile" or "being expressionless but inwardly excited," significantly improving the authenticity of the recognition results.

[0104] After determining the user's emotional state, the system performs in-depth analysis of the voice content based on that state. The same voice content can represent drastically different intentions under different emotional states. For example, the voice content "Go away" might be interpreted as an intention of "expulsion / rejection" if the real-time facial expression is "angry" and the emotional intensity is high; conversely, if the real-time facial expression is "calm" or "reluctant" and the emotional intensity is low, it might be interpreted as an intention of "helplessness / compromise." By combining facial expressions and physiological states for semantic analysis, the system can accurately capture the true motivation behind the user's speech, avoiding the ambiguity of intention recognition based solely on voice text.

[0105] Furthermore, to prevent misjudgments caused by transient noise or brief changes in facial expressions, this embodiment introduces a time-dimensional tracking mechanism. Specifically, this includes: calculating the similarity between real-time facial expression features and the facial expression recognition features corresponding to each facial expression category label; combining this with real-time physiological features for cross-validation to determine the initial facial expression category and emotion intensity level; tracking changes in real-time facial expression features and real-time physiological features within a preset time interval to assess the stability of the initial facial expression category and emotion intensity level, and determining the real-time facial expression state of the interacting object.

[0106] The preset time interval can be set according to the actual application scenario, for example, 3 to 5 seconds. The system continuously monitors feature changes within this time window. If the initially determined expression category and emotion intensity level remain stable without drastic changes within the preset time interval, this state is confirmed as the final real-time expression state. Conversely, if the features frequently change within a short period (e.g., from "angry" to "calm" and then to "happy" within 1 second), the system determines that the current recognition result is unstable and may be subject to momentary interference. In this case, the system will extend the monitoring time or prompt for data re-collection until the features stabilize. This mechanism constructs a layer of defense in the time dimension, ensuring the stability and reliability of the output results.

[0107] Furthermore, the optimal interaction response scheme and its corresponding interaction mode are determined. The interaction mode is dynamically adapted and adjusted based on the historical interaction preferences of the interaction object and the current scene interference level to obtain the adapted interaction response scheme. This includes: acquiring the historical interaction preference data of the interaction object and mining preference patterns in the historical interaction preference data based on a collaborative filtering algorithm to generate a personalized preference model; adjusting the presentation form of the interaction response based on the current scene interference level to obtain the initial presentation form; adjusting the content details, initial presentation form, and interaction rhythm of the interaction response scheme based on the personalized preference model, collaborative relationships, and corresponding mapping relationships to obtain the adjusted interaction response scheme; and verifying the rationality and fit of the adjusted interaction response scheme to obtain the adapted interaction response scheme.

[0108] In this implementation, historical interaction preference data records users' past feedback on different robot response methods. For example, under specific emotional states, users may prefer voice broadcasting or screen display, or a concise dialogue style or a detailed explanation style. This embodiment employs a collaborative filtering algorithm to construct a "user-interaction behavior" matrix, calculate the similarity between the current user and other user groups, and then predict the current user's potential preferences for behaviors that have not yet occurred. For example, if the system identifies that the current user is highly similar to user group A in historical behavior (e.g., both like to play light music when tired), the preference patterns of user group A are transferred to the current user to generate a personalized preference model. This model can quantify users' interaction tendencies in different scenarios and emotional states, providing data support for subsequent adjustments.

[0109] Specifically, the presentation includes, but is not limited to, the volume, speed, and tone of voice announcements; the brightness and font size of the screen display; and the range of the robot's body movements. The system adaptively adjusts based on the level of interference in the current environment. For example, if the current environment is a "high-noise environment," the system will automatically increase the volume of the voice announcements, decrease the speed, and increase the font size on the screen to ensure that the user can clearly receive the information; if the current environment is a "quiet environment," the system will decrease the volume and use a gentle tone to avoid disturbing the user. This ensures the physical perceptibility of the interactive response.

[0110] After determining the initial presentation format, the system further refines the content details by incorporating a personalized preference model. For example, if the personalized preference model shows that the user prefers "soothing" dialogue when in a state of "anxiety," the system will adjust the wording of the response to make it more empathetic. At the same time, it will adjust the interaction rhythm by incorporating collaborative relationships (such as the correlation between facial expressions and physiological characteristics). For instance, when the system detects that the user's heart rate is too fast, it will appropriately extend the pause time between sentences to give the user a buffer.

[0111] Furthermore, the adjusted interaction response scheme is validated for rationality and fit to obtain an adapted interaction response scheme. This includes: extracting the core content of the adjusted interaction response scheme and performing semantic matching with the interaction intent data; determining whether the priority of the core content is reasonable; if the priority of the core content is reasonable and the semantic matching degree is greater than or equal to the preset matching threshold, then determining whether the tone, response speed, body movements, real-time facial expressions, and emotional intensity levels of the interaction response scheme match; if the tone, response speed, body movements, real-time facial expressions, and emotional intensity levels of the interaction response scheme match, then verifying the fit of the presentation form, content details, current scene interference level, and historical interaction preferences of the interaction response scheme to obtain the fit degree corresponding to the adjusted interaction response scheme; determining whether the fit degree meets the preset fit degree range; if it does, then the interaction response scheme is deemed reasonable, and an adapted interaction response scheme is obtained; if any item fails to meet the standard, then the content, tone, and presentation form of the interaction response scheme are readjusted until all validation items pass.

[0112] In this embodiment, the core content of the adjusted interaction response scheme is extracted and semantically matched with the interaction intent data to determine whether the priority of the core content is reasonable. For example, if the interaction intent data is "request for help," but the core content of the response scheme is "play music," then the semantic matching degree is determined to be low, and the priority is unreasonable. If the priority of the core content is reasonable and the semantic matching degree is greater than or equal to a preset matching threshold (e.g., 0.8), then the next level of verification is entered; otherwise, the verification is directly determined to fail, triggering a readjustment process. Secondly, if the core content verification passes, then it is determined whether the tone, response speed, body movements, and real-time facial expression state and emotional intensity level of the interaction response scheme match. For example, if the real-time facial expression state is "sad" and the emotional intensity level is "high," but the tone of the response scheme is "cheerful" or the body movements are too large, then it is determined to be a mismatch. Only when the tone, response speed, and body movements are all coordinated with the user's current emotional state will the final verification stage be entered.

[0113] In this embodiment, the fit is a comprehensive score reflecting the degree to which the response scheme adapts to the external environment and the user's personality in terms of form and content. The system determines whether the fit meets the preset fit range (e.g., 0.7 to 1.0). If it does, the interaction response scheme is deemed reasonable, and an adapted interaction response scheme is obtained; if any item fails to meet the standard, the content, tone, and presentation of the interaction response scheme are readjusted until all verification items pass.

[0114] The above embodiments describe a robot interaction method based on facial expression recognition from the perspective of the process flow. The following embodiments describe a robot interaction system based on facial expression recognition from the perspective of virtual modules or virtual units. For details, please refer to the following embodiments.

[0115] This application provides a robot interaction system 20 based on facial expression recognition, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of a robot interaction system based on facial expression recognition, provided as an embodiment of this application. Specifically, the system 20 may include:

[0116] Information acquisition module 21 is used to acquire facial image information, physiological sign information and voice interaction information of the interactive object;

[0117] Information acquisition module 22 is used to collect historical interaction data and the object characteristics information of the interactive objects in the historical interaction data, interaction feedback results and scene interference level;

[0118] The data screening module 23 is used to perform personalized screening of historical interaction data based on physiological and object characteristics information to obtain target interaction data that is compatible with the interaction object.

[0119] The feature extraction module 24 is used to improve the target interaction data based on the interaction feedback results, and to extract multi-dimensional features from the improved target interaction data to obtain scene features and corresponding facial expression recognition features, voice features, physiological features and interaction response features.

[0120] The model building module 25 is used to build a multimodal fusion preset expression interaction model. The preset expression interaction model includes an expression recognition branch, a speech parsing branch, and a physiological feature analysis branch. Based on the expression recognition features, speech features, physiological features, and interaction response features, the preset expression interaction model is iteratively trained to obtain the trained expression interaction model.

[0121] The collaborative processing module 26 is used to collaboratively process facial image information, physiological sign information and voice interaction information based on facial expression recognition features, voice features, physiological features and interactive response features to obtain the real-time facial expression status, emotion intensity level and interactive intent data of the interactive object.

[0122] The matching and recognition module 27 is used to input real-time facial expression status, emotion intensity level and interaction intent data into the facial expression interaction model for multi-dimensional matching and recognition, and to obtain the interaction matching result. The interaction matching result includes at least one candidate interaction response scheme and the fit score corresponding to at least one candidate interaction response scheme.

[0123] The adaptation adjustment module 28 is used to determine whether there is an optimal interaction response scheme with an adaptation score higher than the preset adaptation threshold in the interaction matching results. If there is, the interaction mode corresponding to the optimal interaction response scheme is determined. The interaction mode is dynamically adapted and adjusted in combination with the historical interaction preferences of the interaction object and the interference level of the current scene to obtain the adapted interaction response scheme.

[0124] The control execution module 29 is used to control the robot to perform corresponding interactive actions according to the adapted interactive response scheme.

[0125] In one possible implementation of this application embodiment, when the model building module 25 iteratively trains a preset facial expression interaction model based on facial expression recognition features, speech features, physiological features, and interaction response features to obtain a trained facial expression interaction model, it is specifically used for:

[0126] Based on the interactive scene environment information and scene interference level in the scene features, the historical interaction data is divided into different scene categories, and a set of facial expression recognition features, voice features, physiological features and interactive response features corresponding to each scene category are obtained.

[0127] Calculate the correlation strength between facial expression recognition features, voice features and physiological features, determine the collaborative correlation relationship corresponding to each scene category, and the corresponding mapping relationship between collaborative correlation relationship and interaction response features;

[0128] The facial expression recognition features of each scene category are finely classified and labeled to obtain facial expression category labels and emotion intensity labels;

[0129] Based on collaborative association and corresponding mapping, the facial expression recognition features, speech features, physiological features, facial expression category labels, emotion intensity labels, and interactive response features under each scene category are constructed as a training sample set for feature input and interactive response output;

[0130] The training sample set is input into the preset facial expression interaction model for iterative training, and the weights of the three feature branches of facial expression, voice and physiology are dynamically adjusted according to the level of scene interference to obtain the trained preset facial expression interaction model.

[0131] The trained preset facial expression interaction model is validated by calculating the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate of the preset facial expression interaction model. Training is stopped when the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate all reach preset thresholds, thus obtaining the trained facial expression interaction model.

[0132] In another possible implementation of this application embodiment, when the collaborative processing module 26 performs collaborative processing on facial image information, physiological sign information, and voice interaction information based on expression recognition features, voice features, physiological features, and interaction response features to obtain the real-time expression state, emotion intensity level, and interaction intent data of the interactive object, it is specifically used for:

[0133] The acquired facial image information, physiological sign information, and voice interaction information are preprocessed to extract real-time expression features, real-time physiological features, and real-time voice features.

[0134] Based on collaborative correlation, real-time facial expression features are matched with facial expression recognition features, and cross-validation is performed using real-time physiological features to determine the real-time facial expression category and emotion intensity level.

[0135] The real-time facial expression category and emotion intensity level are tracked and processed to obtain the real-time facial expression state.

[0136] Based on real-time facial expressions and physiological characteristics, semantic analysis is performed on real-time speech features to obtain the interaction intent data of the interactive object.

[0137] In another possible implementation of this application embodiment, when the collaborative processing module 26 performs tracking feature change processing on the real-time expression category and emotion intensity level to obtain the real-time expression state, it is specifically used for:

[0138] Calculate the similarity between real-time facial expression features and the facial expression recognition features corresponding to each facial expression category label, and combine real-time physiological features for cross-validation to determine the preliminary facial expression category and emotion intensity level;

[0139] Track real-time facial expression features and real-time physiological feature changes within a preset time interval, determine the stability of the initial facial expression category and emotional intensity level, and determine the real-time facial expression state of the interactive object.

[0140] In another possible implementation of this application embodiment, when the adaptation and adjustment module 28 determines the interaction mode corresponding to the optimal interaction response scheme, and dynamically adapts and adjusts the interaction mode based on the historical interaction preferences of the interaction object and the current scene interference level to obtain the adapted interaction response scheme, it is specifically used for:

[0141] Obtain historical interaction preference data of interactive objects, and mine the preference patterns in the historical interaction preference data based on collaborative filtering algorithm to generate personalized preference models;

[0142] Adjust the presentation of the interactive response based on the current interference level to obtain the initial presentation format;

[0143] Based on the personalized preference model, collaborative relationships and corresponding mapping relationships, the content details, initial presentation form and interaction rhythm of the interaction response scheme are adjusted to obtain the adjusted interaction response scheme;

[0144] The adjusted interaction response scheme is validated for its rationality and suitability to obtain the adapted interaction response scheme.

[0145] In another possible implementation of this application embodiment, when the adaptation and adjustment module 28 verifies the rationality and fit of the adjusted interaction response scheme to obtain the adapted interaction response scheme, it is specifically used for:

[0146] Extract the core content of the adjusted interaction response scheme and perform semantic matching with the interaction intent data to determine whether the priority of the core content is reasonable. If the priority of the core content is reasonable and the semantic matching degree is greater than or equal to the preset matching threshold, then determine whether the tone, response speed, body movements and real-time facial expression status and emotional intensity level of the interaction response scheme are matched.

[0147] If the tone, response speed, body movements, and real-time facial expressions and emotional intensity levels of the interactive response scheme match, then the presentation format, content details, and the degree of fit between the interactive response scheme and the current scene interference level and historical interaction preferences are verified to obtain the degree of fit corresponding to the adjusted interactive response scheme.

[0148] Determine if the fit meets the preset fit range. If it does, the interaction response scheme is deemed reasonable, and the adapted interaction response scheme is obtained. If any item fails to meet the standard, the content, tone, and presentation of the interaction response scheme are readjusted until all the verification items pass.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described robot interaction system 20 based on facial expression recognition can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0150] This application provides an electronic device, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device 300 does not constitute a limitation on the embodiments of this application.

[0151] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0152] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0153] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0154] The memory 303 is used to store application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0155] Electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Servers can also be included. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0156] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0157] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0158] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A robot interaction method based on facial expression recognition, characterized in that, include: Acquire facial image information, physiological sign information, and voice interaction information of the interactive object; Collect historical interaction data, as well as the object characteristics information of the interactive objects, interaction feedback results, and scene interference levels in the historical interaction data; Based on the physiological signs information and the object's signs information, the historical interaction data is screened in a personalized manner to obtain target interaction data that is compatible with the interaction object; Based on the interaction feedback results, the target interaction data is improved, and multi-dimensional features are extracted from the improved target interaction data to obtain scene features and corresponding facial expression recognition features, voice features, physiological features and interaction response features. A multimodal fusion preset facial expression interaction model is constructed. The preset facial expression interaction model includes an facial expression recognition branch, a speech parsing branch, and a physiological feature analysis branch. Based on the facial expression recognition features, the speech features, the physiological features, and the interaction response features, the preset facial expression interaction model is iteratively trained to obtain the trained facial expression interaction model. Based on the facial expression recognition features, the voice features, the physiological features, and the interaction response features, the facial image information, the physiological sign information, and the voice interaction information are processed collaboratively to obtain the real-time facial expression state, emotion intensity level, and interaction intent data of the interactive object. The real-time facial expression status, emotional intensity level, and interaction intent data are input into the facial expression interaction model for multi-dimensional matching and recognition to obtain the interaction matching result. The interaction matching result includes at least one candidate interaction response scheme and a fit score corresponding to at least one candidate interaction response scheme. Determine whether there is an optimal interaction response scheme in the interaction matching results whose adaptation score is higher than the preset adaptation threshold. If there is, determine the interaction mode corresponding to the optimal interaction response scheme. Combine the historical interaction preferences of the interaction object and the interference level of the current scene, dynamically adapt and adjust the interaction mode to obtain the adapted interaction response scheme. Based on the adapted interaction response scheme, the robot is controlled to perform the corresponding interactive actions.

2. The robot interaction method based on facial expression recognition according to claim 1, characterized in that, The step of iteratively training the preset facial expression interaction model based on the facial expression recognition features, the voice features, the physiological features, and the interaction response features to obtain the trained facial expression interaction model includes: Based on the interactive scene environment information and the scene interference level in the scene features, the historical interaction data is divided into different scene categories, and a set of facial expression recognition features, voice features, physiological features and interactive response features are obtained for each scene category. Calculate the correlation strength between the facial expression recognition features, the voice features, and the physiological features; determine the collaborative correlation relationship corresponding to each scene category; and the corresponding mapping relationship between the collaborative correlation relationship and the interaction response features. The facial expression recognition features of each scene category are finely classified and labeled to obtain facial expression category labels and emotion intensity labels; Based on the aforementioned collaborative association and the corresponding mapping relationship, the facial expression recognition features, speech features, physiological features, facial expression category labels, emotion intensity labels, and interactive response features under each scene category are constructed into a training sample set for feature input and interactive response output; The training sample set is input into the preset facial expression interaction model for iterative training, and the weights of the three feature branches of facial expression, voice and physiology are dynamically adjusted according to the interference level of the scene to obtain the trained preset facial expression interaction model. The trained preset facial expression interaction model is validated by calculating the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate of the preset facial expression interaction model. Training is stopped when the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate all reach preset thresholds, thus obtaining the trained facial expression interaction model.

3. The robot interaction method based on facial expression recognition according to claim 2, characterized in that, The step involves collaboratively processing the facial image information, physiological sign information, and voice interaction information based on the facial expression recognition features, the voice features, the physiological features, and the interaction response features to obtain the real-time facial expression state, emotion intensity level, and interaction intent data of the interactive object, including: The acquired facial image information, physiological sign information, and voice interaction information are preprocessed to extract real-time expression features, real-time physiological features, and real-time voice features. Based on the aforementioned collaborative association, real-time facial expression features are matched with facial expression recognition features, and cross-validation is performed using the aforementioned real-time physiological features to determine the real-time facial expression category and emotion intensity level. The real-time expression category and the emotion intensity level are processed to track feature changes and obtain the real-time expression state. Based on the real-time facial expressions and real-time physiological characteristics, semantic parsing is performed on the real-time speech features to obtain the interaction intent data of the interactive object.

4. The robot interaction method based on facial expression recognition according to claim 3, characterized in that, The step of tracking feature changes in the real-time expression category and the emotion intensity level to obtain the real-time expression state includes: The similarity between the real-time facial expression features and the facial expression recognition features corresponding to each facial expression category label is calculated, and cross-validation is performed using real-time physiological features to determine the preliminary facial expression category and emotion intensity level. The system tracks real-time facial expression features and real-time physiological feature changes within a preset time interval, judges the stability of the preliminary facial expression category and the emotional intensity level, and determines the real-time facial expression state of the interactive object.

5. The robot interaction method based on facial expression recognition according to claim 1, characterized in that, The step of determining the interaction mode corresponding to the optimal interaction response scheme, and dynamically adapting and adjusting the interaction mode based on the historical interaction preferences of the interaction object and the current scene interference level, to obtain the adapted interaction response scheme includes: The historical interaction preference data of the interactive object is obtained, and the preference patterns in the historical interaction preference data are mined based on the collaborative filtering algorithm to generate a personalized preference model; Adjust the presentation of the interactive response based on the current interference level to obtain the initial presentation format; Based on the personalized preference model, collaborative relationships and corresponding mapping relationships, the content details, initial presentation form and interaction rhythm of the interaction response scheme are adjusted to obtain the adjusted interaction response scheme; The adjusted interaction response scheme is validated for its rationality and suitability to obtain the adapted interaction response scheme.

6. The robot interaction method based on facial expression recognition according to claim 5, characterized in that, The process of verifying the rationality and fit of the adjusted interaction response scheme to obtain the adapted interaction response scheme includes: Extract the core content of the adjusted interaction response scheme and perform semantic matching with the interaction intent data to determine whether the priority of the core content is reasonable. If the priority of the core content is reasonable and the semantic matching degree is greater than or equal to the preset matching threshold, then determine whether the tone, response speed, body movements and real-time facial expression status and emotional intensity level of the interaction response scheme match. If the tone, response speed, body movements and real-time facial expression status and emotional intensity level of the interaction response scheme are matched, then the degree of fit between the presentation form and content details of the interaction response scheme and the current scene interference level and historical interaction preferences is verified to obtain the degree of fit corresponding to the adjusted interaction response scheme. Determine whether the fit meets the preset fit range. If it does, the interaction response scheme is deemed reasonable, and an adapted interaction response scheme is obtained. If any item fails to meet the standard, the content, tone, and presentation of the interaction response scheme are readjusted until all verification items pass.

7. A robot interaction system based on facial expression recognition, characterized in that, include: The information acquisition module is used to acquire facial image information, physiological sign information, and voice interaction information of the interactive object; The information collection module is used to collect historical interaction data, as well as the object characteristics information of the interactive objects, interaction feedback results, and scene interference levels in the historical interaction data; The data screening module is used to perform personalized screening of the historical interaction data based on the physiological signs information and the object's signs information to obtain target interaction data that is compatible with the interaction object. The feature extraction module is used to perform data improvement processing on the target interaction data based on the interaction feedback results, and to perform multi-dimensional feature extraction on the improved target interaction data to obtain scene features and corresponding facial expression recognition features, voice features, physiological features and interaction response features. The model building module is used to build a multimodal fusion preset facial expression interaction model. The preset facial expression interaction model includes a facial expression recognition branch, a speech parsing branch, and a physiological feature analysis branch. Based on the facial expression recognition features, the speech features, the physiological features, and the interaction response features, the preset facial expression interaction model is iteratively trained to obtain the trained facial expression interaction model. The collaborative processing module is used to collaboratively process the facial image information, the physiological sign information and the voice interaction information based on the facial expression recognition features, the voice features, the physiological features and the interaction response features, to obtain the real-time facial expression status, emotion intensity level and interaction intent data of the interactive object. The matching and recognition module is used to input the real-time facial expression state, emotion intensity level and interaction intent data into the facial expression interaction model for multi-dimensional matching and recognition, and to obtain the interaction matching result. The interaction matching result includes at least one candidate interaction response scheme and a fit score corresponding to at least one candidate interaction response scheme. The adaptation adjustment module is used to determine whether there is an optimal interaction response scheme with an adaptation score higher than a preset adaptation threshold in the interaction matching results. If there is, the interaction mode corresponding to the optimal interaction response scheme is determined. The interaction mode is dynamically adapted and adjusted in combination with the historical interaction preferences of the interaction object and the interference level of the current scene to obtain the adapted interaction response scheme. The control execution module is used to control the robot to perform corresponding interactive actions according to the adapted interaction response scheme.

8. A robot interaction system based on facial expression recognition according to claim 7, characterized in that, When the model building module iteratively trains the preset facial expression interaction model based on the facial expression recognition features, the speech features, the physiological features, and the interaction response features to obtain the trained facial expression interaction model, it is specifically used for: Based on the interactive scene environment information and the scene interference level in the scene features, the historical interaction data is divided into different scene categories, and a set of facial expression recognition features, voice features, physiological features and interactive response features are obtained for each scene category. Calculate the correlation strength between the facial expression recognition features, the voice features, and the physiological features; determine the collaborative correlation relationship corresponding to each scene category; and the corresponding mapping relationship between the collaborative correlation relationship and the interaction response features. The facial expression recognition features of each scene category are finely classified and labeled to obtain facial expression category labels and emotion intensity labels; Based on the aforementioned collaborative association and the corresponding mapping relationship, the facial expression recognition features, speech features, physiological features, facial expression category labels, emotion intensity labels, and interactive response features under each scene category are constructed into a training sample set for feature input and interactive response output; The training sample set is input into the preset facial expression interaction model for iterative training, and the weights of the three feature branches of facial expression, voice and physiology are dynamically adjusted according to the interference level of the scene to obtain the trained preset facial expression interaction model. The trained preset facial expression interaction model is validated by calculating the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate of the preset facial expression interaction model. Training is stopped when the facial expression recognition accuracy, interaction intent matching accuracy, and response adaptation rate all reach preset thresholds, thus obtaining the trained facial expression interaction model.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as any one of the facial expression recognition-based robot interaction methods as claimed in claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as any one of the facial expression recognition-based robot interaction methods as described in claims 1 to 6.

Images