A Robot Human-Computer Interaction Intent Recognition Method Based on Multimodal Perception
By collecting and processing color and depth image sequences in service robots, dividing facial expression regions and generating validity masks, and combining motion transfer and facial expression recognition branches, the problem of unstable facial expression recognition in human-computer interaction is solved, achieving improved stability and adaptability. The interaction intent selection is dynamically adjusted, reducing the repetition of unsuitable responses.
Patent Information
- Application Number
- CN202610730637.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-26
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-05-26
AI Technical Summary
In the process of human-computer interaction, existing service robots suffer from insufficient stability in the recognition of dynamic facial movements and expression categories due to fluctuations in the quality of color and depth image acquisition and inconsistencies in the reliability of information from different local facial regions. Furthermore, it is difficult to adjust the selection of subsequent interaction intentions based on user feedback.
The service robot uses an RGB-D camera to capture continuous video clips of the user, simultaneously extracting color face image sequences and depth face image sequences. Multiple expression regions are divided based on facial key points, and color validity masks and depth validity masks are generated. The regional modality fusion weights are determined, and color texture features and depth motion features of the expression regions are extracted. Consistency verification is performed by combining the action transfer recognition branch and the expression category recognition branch. The interaction intent label is determined based on the facial action category, expression recognition results, and user distance level, and the mapping priority is updated based on the execution feedback samples.
It improves the stability of facial expression recognition, enhances the correspondence between dynamic facial movements and expression categories, reduces the risk of misjudgment of candidate expression categories, improves the adaptability of interaction intent labels, and enables the dynamic adjustment of subsequent interaction intent selection, reducing the repeated execution of incompatible interaction response commands.
Smart Images

Figure CN122618672B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image recognition and intelligent robot interaction technology, and in particular to a method for recognizing robot human-computer interaction intentions based on multimodal perception. Background Technology
[0002] With the development and application of service robots, smart home systems, and human-computer interaction technologies, service robots are increasingly being used in scenarios such as home care, indoor guidance, voice question answering, companionship, and daily living assistance. In these scenarios, service robots typically acquire users' voice, images, posture, or environmental information through sensing devices such as cameras, depth vision sensors, and microphones, and generate voice responses, action demonstrations, or service task instructions based on the recognition results to achieve interaction with users.
[0003] In the prior art, there are already technical solutions for using multiple types of media information in user-robot interaction. For example, invention patent application with publication number CN115946139A discloses a user-robot cross-media interaction system and method, which acquires offline voice, online voice, identity feature images, facial emotion images, user pose images, and environmental image information through a cross-media perception subsystem, performs classification and understanding through a cross-media fusion subsystem, and then calls the corresponding cross-media expression method to output classification control commands, enabling the robot to perform interactive responses such as voice, facial expression display, or body movements.
[0004] The aforementioned existing technologies can realize the perception of user multimedia information and the response of robot interaction. However, in actual human-computer interaction, users' facial expressions and facial movements usually exhibit continuous dynamic changes, and the expression information contained in different local facial regions can be affected by factors such as changes in lighting, partial occlusion, motion blur, depth holes, or local depth abrupt changes. If interaction judgment is based solely on the overall facial emotion image or the overall multimedia classification result, the effective information in local expression regions is easily weakened, resulting in insufficient stability of facial movement categories and expression recognition results during continuous interaction.
[0005] Furthermore, in continuous interaction scenarios such as smart home systems, the interactive responses of service robots are typically related not only to the user's current facial expression, but also to the user's distance state, current interaction state, and changes in the user's state after the interactive response is executed. Existing robot interaction solutions mostly focus on classifying and understanding the current input information and outputting corresponding control commands. They lack a way to process the feedback information reflected by the user's subsequent facial expressions, facial movements, or distance changes after the interactive response is executed, in conjunction with the selection of subsequent interaction intentions. This may lead to the service robot continuously outputting inappropriate interactive responses within the same interaction round.
[0006] Therefore, existing service robots still need to solve the following technical problems in the process of human-computer interaction intent recognition: how to stably obtain the user's dynamic facial movements and expression categories when the quality of color images and depth images fluctuates and the reliability of information in different local facial regions is inconsistent, and how to make the subsequent interaction intent selection adjustable based on user feedback after the interaction response is executed. Summary of the Invention
[0007] To overcome the aforementioned technical deficiencies, the present invention aims to provide a robot human-computer interaction intent recognition method based on multimodal perception, in order to solve the problems of insufficient stability in dynamic facial action and expression category recognition, and difficulty in adjusting subsequent interaction intent selection based on user feedback, caused by fluctuations in the quality of color and depth image acquisition and inconsistent reliability of information in different local facial regions during human-computer interaction of existing service robots.
[0008] This invention discloses a robot human-computer interaction intent recognition method based on multimodal perception, comprising:
[0009] The service robot uses an RGB-D camera to capture continuous video clips of the user, simultaneously extracts color face image sequences and depth face image sequences, and divides multiple expression regions based on facial key points;
[0010] For each expression region and each frame, a color validity mask and a depth validity mask are generated. Based on the color validity value determined by the color validity mask, the depth validity value determined by the depth validity mask, and a preset stable value, the region modality fusion weight is determined.
[0011] Color texture features corresponding to each expression region are extracted from color face image sequences, and depth motion features corresponding to each expression region, including depth displacement direction, depth displacement amplitude and region deformation state, are extracted from depth face image sequences. Region fusion expression feature sequences are generated according to region modality fusion weights.
[0012] The region-fused facial expression feature sequence is input into the action transfer recognition branch and the facial expression category recognition branch respectively to obtain the region action transfer sequence and candidate facial expression categories. The facial action category and facial expression recognition result are then output according to the action-expression consistency verification rule.
[0013] The interaction intent label is determined based on the facial action category, expression recognition result, user distance level, current interaction state of the service robot, and mapping priority in the state conditional intent mapping correspondence.
[0014] The control service robot executes the interactive response instructions corresponding to the interactive intent tags, collects subsequent video clips of the user after execution to generate execution feedback samples, and updates the mapping priority based on the execution feedback samples. The updated mapping priority is used to determine the subsequent interactive intent tags.
[0015] Preferably,
[0016] Simultaneously extract color face image sequences and depth face image sequences, including:
[0017] The user's continuous video clips are divided into multiple candidate frame groups according to the acquisition timestamp. Each candidate frame group includes a color frame and a depth frame corresponding to the color frame.
[0018] Determine the face bounding box and face key points in each candidate frame group;
[0019] Candidate expression regions are determined in the color frame based on facial key points, and candidate depth regions corresponding to the candidate expression regions are determined in the depth frame.
[0020] The facial expression trigger frame is determined based on the color changes of the candidate expression region and the depth changes of the candidate depth region.
[0021] Centered on facial expression trigger frames, color face image sequences and depth face image sequences are extracted from multiple consecutive candidate frame groups.
[0022] Preferably,
[0023] The facial expression trigger frame is determined based on the color changes of candidate expression regions and the depth changes of candidate depth regions, including:
[0024] Determine the action trigger value using the following formula:
[0025]
[0026] in, Indicates the candidate expression region number. Indicates the frame number and , Indicates the first Frame number Action trigger values for each candidate expression region, Indicates the first Frame number The average grayscale value of each candidate expression region Indicates the first Frame number The average grayscale value of each candidate expression region This indicates the preset grayscale normalization value, and , Indicates the first Frame number The average depth value of each candidate depth region Indicates the first Frame number The average depth value of each candidate depth region This represents the preset depth normalization value, and ;
[0027] When the action trigger value of each of the first three consecutive frames reaches the preset action trigger threshold, and the depth change direction of the candidate depth region remains consistent in the three consecutive frames, the second frame of the three consecutive frames is determined as the expression action trigger frame.
[0028] Preferably,
[0029] Multiple expression regions are divided based on facial landmarks, including:
[0030] Establish a facial region coordinate system based on the nose tip, the center of the eyes, and the corner of the mouth among the facial key points;
[0031] Map the depth face image sequence to the face region coordinate system;
[0032] Divide the face region into eyebrow region, eye region, nose region, mouth region and cheek region in the face region coordinate system;
[0033] The eyebrow area, eye area, nose area, mouth area, and cheek area are designated as expression areas.
[0034] Preferably,
[0035] Generate a color validity mask, including:
[0036] Occlusion detection, brightness discrepancy detection, and motion blur detection are performed on the colored pixels in each expression area;
[0037] Color pixels that meet the preset occlusion conditions, color pixels that meet the preset brightness dispersion conditions, and color pixels that meet the preset motion blur conditions are marked as invalid color pixels;
[0038] A color validity mask is generated based on the positional distribution of invalid color pixels within the corresponding emoji area, and the valid color values are determined according to the following formula:
[0039]
[0040] in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number The effective color values for each facial expression region Indicates the first Frame number The number of colored pixels that meet the preset occlusion conditions within each facial expression area. Indicates the first Frame number The number of colored pixels within each facial expression area that meet the preset brightness dispersion condition. Indicates the first Frame number The number of colored pixels within each facial expression area that meet the preset motion blur conditions. Indicates the first Frame number The total number of colored pixels in each expression area, and .
[0041] Preferably,
[0042] Generate a depth validity mask, including:
[0043] For each depth pixel within the facial expression region, depth hole detection, depth abrupt change detection, and inter-frame depth consistency detection are performed respectively.
[0044] Depth pixels that meet the preset depth hole condition, depth pixels that meet the preset depth mutation condition, and depth pixels that do not meet the preset inter-frame depth consistency condition are marked as depth invalid pixels.
[0045] A depth validity mask is generated based on the positional distribution of depth-invalid pixels within the corresponding facial expression region, and the depth validity value is determined according to the following formula:
[0046]
[0047] in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number The effective depth value of each facial expression region. Indicates the first Frame number The number of depth pixels within an expression region that meet the preset depth hole condition. Indicates the first Frame number The number of depth pixels within each facial expression region that satisfy the preset depth abrupt change conditions. Indicates the first Frame number The number of depth pixels within an expression region that do not meet the preset inter-frame depth consistency condition. Indicates the first Frame number The total number of depth pixels within each facial expression region, and .
[0048] Preferably,
[0049] Determine the regional modal fusion weights, including:
[0050] Based on the color effective value, depth effective value, and preset stable value, the color blending weight and depth blending weight are determined according to the following formula:
[0051]
[0052]
[0053] in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number Color blending weights for each facial expression region Indicates the first Frame number The depth fusion weight of each facial expression region Indicates the first Frame number The effective color values for each facial expression region Indicates the first Frame number The effective depth value of each facial expression region. This indicates a preset stable value, and The regional modal fusion weights include color fusion weights and depth fusion weights.
[0054] Preferably,
[0055] Generate a region-fused facial expression feature sequence according to the region modality fusion weights, including:
[0056] Multiply the color texture feature corresponding to each expression region by the corresponding color fusion weight to obtain the weighted color texture feature;
[0057] Multiply the deep motion features corresponding to the same facial expression region by the corresponding deep fusion weight to obtain weighted deep motion features;
[0058] The weighted color texture features and weighted depth motion features corresponding to the same facial expression region are spliced together to obtain the region splicing features;
[0059] The region splicing features are compressed and encoded to obtain the region fusion expression features;
[0060] Multiple regions are combined according to frame number to generate a region-fused facial expression feature sequence.
[0061] Preferably,
[0062] Based on the action-expression consistency verification rules, the facial action category and expression recognition results are output, including:
[0063] The action transfer recognition branch outputs the regional deformation state transfer result of each expression region based on the region fusion expression feature sequence, and generates a regional action transfer sequence from multiple regional deformation state transfer results;
[0064] The facial expression category recognition branch outputs the category confidence scores of multiple preset facial expression categories based on the region fusion facial expression feature sequence, and determines the preset facial expression categories whose category confidence scores meet the preset confidence conditions as candidate facial expression categories;
[0065] When the candidate expression category and the region action transfer sequence satisfy the correspondence in the action-expression consistency verification rule, the candidate expression category is determined as the expression recognition result, and the combination of the region deformation state corresponding to the region action transfer sequence is determined as the facial action category.
[0066] When the candidate expression category and the regional action transfer sequence do not meet the correspondence in the action-expression consistency verification rules, the candidate expression category is marked as an expression category to be confirmed, and the category confidence of the expression category to be confirmed is reduced to below the preset confirmation threshold.
[0067] Preferably,
[0068] The execution feedback samples are generated in the following manner:
[0069] After the service robot executes the interactive response command, it collects subsequent video clips from the user according to the preset feedback sampling duration;
[0070] Based on subsequent video clips of the user, obtain subsequent expression recognition results, subsequent facial movement categories, and subsequent user distance change directions;
[0071] Execution feedback samples are generated based on subsequent expression recognition results, subsequent facial action categories, and subsequent user distance change directions.
[0072] Preferably,
[0073] Mapping priority is updated based on execution feedback samples, including:
[0074] When the subsequent expression recognition result belongs to the non-negative expression category, or the subsequent user distance change direction is the stationary direction, and the corresponding execution feedback sample does not meet the negative feedback judgment condition, the corresponding execution feedback sample is marked as a positive execution feedback sample.
[0075] When the subsequent expression recognition result belongs to the negative expression category, or when the subsequent user distance changes in the direction of moving away, the corresponding execution feedback sample will be marked as a negative execution feedback sample.
[0076] When the same execution feedback sample simultaneously meets both the positive feedback determination condition and the negative feedback determination condition, the execution feedback sample is marked as a negative execution feedback sample.
[0077] Based on the number of positive and negative execution feedback samples corresponding to the same interaction intent label within the preset statistical window, update the mapping priority corresponding to the interaction intent label in the state conditional intent mapping correspondence;
[0078] Negative expression categories include frowning, confusion, and discomfort, while non-negative expression categories include smiling and calm.
[0079] Preferably,
[0080] Based on the execution feedback sample update mapping priority, it also includes:
[0081] When the number of times the same interactive response instruction generates negative execution feedback samples in the same interaction round reaches a preset feedback number threshold, the interactive response instruction is added to the prohibited execution list of the current interaction round.
[0082] Based on the current interaction state, generate the set of available intents for the current interaction round from the state-conditional intent mapping correspondence;
[0083] Exclude intent tags from the available intent set that have a mapping relationship with interactive response instructions in the prohibited execution list;
[0084] The interaction intent labels are redefined based on the set of available intents after exclusion.
[0085] Preferably,
[0086] Interaction intent labels are determined based on facial action category, expression recognition result, user distance level, current interaction state of the service robot, and mapping priority in the state-conditional intent mapping correspondence, including:
[0087] Based on the depth validity mask, the effective depth pixels of the nose region and the effective depth pixels of the cheek region are extracted from the depth face image sequence, and the user's facial reference depth value is determined based on the effective depth pixels of the nose region and the effective depth pixels of the cheek region.
[0088] The user distance level is determined based on the user's facial reference depth value, and the direction of user distance change is determined based on the user's facial reference depth value in adjacent frames.
[0089] The available intent set is filtered from the state-conditional intent mapping correspondence based on the current interaction state;
[0090] Select candidate intent labels from the available intent set that simultaneously match the facial action category, expression recognition result, user distance level, and direction of user distance change.
[0091] Preferably,
[0092] Determining interaction intent tags also includes:
[0093] Read the feedback facial expression recognition results and the direction of change in distance to the user from the execution feedback samples;
[0094] The priority score for candidate intent tags is determined using the following formula:
[0095]
[0096] in, Indicates the candidate intent label number, Indicates the first Priority scoring of each candidate intent tag. The state conditionalization intention mapping correspondence in the first... The basic priority of each candidate intent tag , and Let represent the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient, respectively. , , , Indicates the first The number of times the positive feedback judgment condition is met in the execution feedback samples corresponding to each candidate intent label. Indicates the first The number of times the negative feedback judgment condition is met in the execution feedback samples corresponding to each candidate intent label. Indicates the first The number of times each candidate intent label is repeatedly executed within the same interaction round;
[0097] Positive feedback criteria include feedback expression recognition results belonging to a non-negative expression category, or feedback user distance change direction being the stationary direction;
[0098] Negative feedback criteria include feedback expression recognition results belonging to the negative expression category, or feedback user distance change direction being the away direction;
[0099] When the same execution feedback sample simultaneously meets both the positive feedback judgment condition and the negative feedback judgment condition, the execution feedback sample is counted as the number of times the negative feedback judgment condition is met.
[0100] Negative facial expression categories include frowning, confusion, and discomfort, while non-negative facial expression categories include smiling and calm.
[0101] The candidate intent tag with the highest priority score is determined as the interaction intent tag.
[0102] Preferably,
[0103] Control the service robot to execute interactive response instructions corresponding to the interactive intent tags, including:
[0104] The interaction response instruction is determined from the robot service instruction library based on the interaction intent label;
[0105] When the interactive response command is a voice command, the voice broadcasting unit of the service robot is invoked to execute the voice response;
[0106] When the interactive response command is an action command, the service robot's body motion unit is invoked to execute the action for display;
[0107] When the interactive response command is a service-type command, the service robot's task execution unit is invoked to execute the service task.
[0108] Write the execution time, execution type, interaction intent label, and corresponding execution feedback sample of the interactive response command into the interaction record table;
[0109] The interaction log table is used to count the number of times positive feedback conditions are met, the number of times negative feedback conditions are met, and the number of times the operation is repeated.
[0110] Compared with existing technologies, the above technical solution has the following advantages:
[0111] First, this application improves the stability of facial expression recognition under scenarios with fluctuating multimodal acquisition quality. This application generates color validity masks and depth validity masks for each expression region in both color and depth face image sequences. Based on color validity values, depth validity values, and preset stability values, it determines the region modality fusion weights, enabling the fusion of color texture features and depth motion features of different expression regions in different frames according to image validity. Therefore, under conditions such as illumination changes, partial occlusion, motion blur, depth holes, or local depth abrupt changes, the impact of single modality or locally invalid information on the expression recognition results can be reduced.
[0112] Second, this application enhances the correspondence between dynamic facial action categories and expression recognition results. This application extracts depth motion features from depth face image sequences, including depth displacement direction, depth displacement amplitude, and regional deformation state, and combines these features with color texture features to generate a region-fused expression feature sequence. This allows the expression recognition process to not only rely on static color texture information but also incorporate the three-dimensional changes in the user's facial region across consecutive frames. Through this processing method, it can more accurately reflect facial action changes corresponding to expression categories such as smiling, frowning, surprise, confusion, discomfort, and calmness.
[0113] Third, this application can reduce the risk of misjudgment of candidate expression categories. This application sets up an action transfer recognition branch and an expression category recognition branch, which output the region action transfer sequence and candidate expression categories respectively, and perform consistency verification between the two using action-expression consistency rules. When a candidate expression category does not satisfy the correspondence with the region action transfer sequence, the candidate expression category is marked as an expression category to be confirmed and its category confidence is reduced. Therefore, the misjudgment problem caused by directly outputting the recognition result based solely on the expression category confidence can be avoided.
[0114] Fourth, this application improves the adaptability of service robot interaction intent label determination. When determining interaction intent labels, this application not only uses facial expression recognition results but also combines facial action categories, user distance levels, user distance change directions, the service robot's current interaction state, and the mapping priority in the state-conditional intent mapping correspondence for comprehensive judgment. This allows the interaction intent label determination process to simultaneously reflect the user's facial expression state, facial action state, and spatial interaction state, thereby improving the matching degree between interaction response commands and the user's current state.
[0115] Fifth, this application enables the selection of subsequent interaction intents to be dynamically adjusted based on user feedback. After the service robot executes the interaction response command, this application continues to collect subsequent video clips from the user and generates execution feedback samples based on subsequent expression recognition results, subsequent facial action categories, and subsequent user distance change directions. The mapping priority is then updated based on the execution feedback samples. Therefore, the determination of subsequent interaction intent labels no longer relies solely on fixed mapping rules but can be adjusted based on user feedback after the interaction response is executed.
[0116] Sixth, this application can reduce the repeated execution of incompatible interactive response commands within the same interaction round. This application counts the number of negative execution feedback samples generated by an interactive response command within the same interaction round. When this number reaches a preset feedback count threshold, the corresponding interactive response command is added to a prohibited execution list, and intent tags that have a mapping relationship with that interactive response command are excluded from the available intent set of the current interaction round. Therefore, the probability of the service robot continuously outputting incompatible interactive response commands within the same interaction round can be reduced.
[0117] Seventh, this application can form a continuous processing chain centered on facial expression recognition. This application continuously processes color facial image sequences, depth facial image sequences, expression regions, region-fused expression feature sequences, candidate expression categories, facial action categories, and expression recognition results, and uses the expression recognition results as an important input for determining interaction intent labels. This makes the technical solution of this application revolve around facial expression recognition, which is convenient for realizing human-computer interaction intent judgment based on facial expression recognition in continuous interaction scenarios of service robots. Attached Figure Description
[0118] Figure 1 A flowchart illustrating the robot human-computer interaction intent recognition method based on multimodal perception provided in this application embodiment;
[0119] Figure 2 A schematic diagram illustrating the division of facial expression regions based on facial key points, provided in an embodiment of this application;
[0120] Figure 3 This is a schematic diagram of the regional modality fusion and facial expression recognition processing structure provided in an embodiment of this application;
[0121] Figure 4 This is a schematic diagram illustrating the process of generating execution feedback samples and updating mapping priorities provided in an embodiment of this application.
[0122] Figure 5 A comparative curve of facial expression recognition accuracy and interaction intent tag matching rate under different test scenarios provided in the embodiments of this application;
[0123] Figure 6 A graph showing the variation of color fusion weight and depth fusion weight in the oral region provided in this application embodiment;
[0124] Figure 7 This application provides an example of an execution feedback sample-driven mapping priority change curve.
[0125] Figure 8 A bar chart comparing the recognition accuracy of different facial expression categories provided in the embodiments of this application. Detailed Implementation
[0126] The embodiments of this invention will be described below with reference to the accompanying drawings. It should be understood that the following embodiments are used to provide a clear and complete description of the technical solution and are not intended to limit its scope of protection. Without departing from the concept of this technical solution, those skilled in the art can make adaptive adjustments to the specific parameters, model structure, sampling window, threshold setting method, and execution terminal configuration. Figure 1 This illustrates the overall process of a robot human-computer interaction intent recognition method. Figure 2This shows the method of dividing the facial expression area. Figure 3 The structure of region modality fusion and facial expression recognition is shown. Figure 4 The process of executing feedback samples and updating mapping priorities is shown. Figures 5 to 8 Comparison curves and statistical charts are shown to illustrate the effects of the implementation.
[0127] This embodiment provides a robot human-computer interaction intent recognition method based on multimodal perception. This method is applicable to service robots, and is particularly suitable for continuous interaction scenarios in smart home systems, such as home companionship, indoor guidance, voice question and answer, companion interaction, elderly care, and daily living assistance. The main image recognition task in this embodiment is to perform expression recognition on a sequence of user facial images and output the expression recognition result and facial action category. The subsequent determination of the interaction intent label uses the expression recognition result, facial action category, user distance level, and current interaction state as input conditions.
[0128] In this embodiment, the service robot is equipped with an RGB-D camera, a voice broadcasting unit, a body motion unit, a task execution unit, and a processor. The RGB-D camera is used to capture continuous video clips of the user, which include color frames and depth frames corresponding to the color frames. The processor receives the color frames and depth frames output by the RGB-D camera and performs simultaneous extraction of color face image sequences and depth face image sequences, expression region segmentation, color validity mask generation, depth validity mask generation, region modality fusion weight calculation, region fusion expression feature sequence generation, action-expression consistency verification, interaction intent label determination, and mapping priority update. The processor is also used to send interactive response instructions to the voice broadcasting unit, body motion unit, or task execution unit. The voice broadcasting unit, body motion unit, and task execution unit are used to execute the interactive response instructions corresponding to the interaction intent labels.
[0129] In this embodiment, the RGB-D camera captures continuous video clips of the user at a sampling frequency of 30 frames per second. Each frame includes a color frame and a depth frame. The processor divides the continuous video clips of the user into multiple candidate frame groups according to the acquisition timestamp. Each candidate frame group includes a color frame and a corresponding depth frame. To ensure synchronization between color frames and depth frames, color frames and depth frames with an acquisition timestamp difference of no more than 10 milliseconds can be grouped into the same candidate frame group. In other embodiments, the synchronization time difference can also be set to no more than one sampling frame period according to the frame rate of the RGB-D camera. For each candidate frame group, the processor detects facial bounding boxes and facial key points in the color frames. Facial key points include the tip of the nose, the center points of both eyes, the corners of the mouth, the eyebrow key points, the eye key points, the nose key points, the mouth key points, and the cheek contour points.
[0130] In one implementation, the processor establishes a facial region coordinate system based on the nose tip, the center points of both eyes, and the corners of the mouth among facial key points, and maps the depth face image sequence to this facial region coordinate system. Specifically, the midpoint of the line connecting the center points of the two eyes and the nose tip are used to determine the facial midline, the distance between the center points of the two eyes is used as a scale reference, and the corners of the mouth are used to determine the spatial range of the lower half of the face. Based on this facial region coordinate system, the processor divides the color face image sequence and the depth face image sequence into eyebrow region, eye region, nose region, mouth region, and cheek region, and uses these regions as expression regions. For the cheek region, it can be further divided into left cheek region and right cheek region. In subsequent calculations, these two can be used as two independent expression regions, or they can be merged into one cheek region for statistical analysis.
[0131] When simultaneously extracting color and depth face image sequences, the processor first determines candidate expression regions in the color frames based on facial key points, and then determines candidate depth regions corresponding to the candidate expression regions in the depth frames of the same candidate frame group. Subsequently, the processor determines expression action trigger frames based on the color changes of the candidate expression regions and the depth changes of the candidate depth regions. These expression action trigger frames are used to determine the image segments subsequently entering the expression recognition process, allowing the service robot to extract image sequences for expression recognition around the moment the user's facial movements occur, rather than performing equal-intensity calculations on all frames in the entire video.
[0132] In one implementation, the processor determines the action trigger value according to the following formula (1):
[0133]
[0134] in, Indicates the candidate expression region number. Indicates the frame number and , Indicates the first Frame number Action trigger values for each candidate expression region, Indicates the first Frame number The average grayscale value of each candidate expression region Indicates the first Frame number The average grayscale value of each candidate expression region This indicates the preset grayscale normalization value, and , Indicates the first Frame number The average depth value of each candidate depth region Indicates the first Frame number The average depth value of each candidate depth region This represents the preset depth normalization value, and The average depth value is in millimeters, and the average grayscale value is the average pixel value after grayscale conversion.
[0135] For example, when the mouth area is the third candidate expression area, the preset grayscale normalization value is 255, the preset depth normalization value is 80 mm, the average grayscale value of the mouth area in frame 11 is 116, the average grayscale value of the mouth area in frame 10 is 94, the average depth value of the candidate depth area in frame 11 is 1048 mm, and the average depth value of the candidate depth area in frame 10 is 1032 mm. Then, according to formula (1), the action trigger value of the third candidate expression area in frame 11 is:
[0136]
[0137]
[0138] When the preset action trigger threshold is 0.18, the action trigger value of the third candidate expression region in frame 11 reaches the preset action trigger threshold. If the action trigger values corresponding to frames 10, 11, and 12 are 0.2130, 0.2863, and 0.2475 respectively, and the depth change direction of the candidate depth region corresponding to the mouth region in these three consecutive frames is away from the camera, then the second frame of these three consecutive frames, i.e., frame 11, is determined as the expression action trigger frame. Centered on the expression action trigger frame, the processor can select 6 frames forward and 9 frames backward to form an image segment containing 16 frames, and extract the color face image sequence and the depth face image sequence from this image segment. The above frame number is only one implementation method; in actual deployment, 8, 12, 24, or 32 frames can also be used as the sequence length for expression recognition.
[0139] After obtaining the color face image sequence and the depth face image sequence, the processor generates a color validity mask and a depth validity mask for each expression region and each frame, respectively. The color validity mask is used to mark the effective pixel distribution in the color image that can be used for expression recognition, and the depth validity mask is used to mark the effective pixel distribution in the depth image that can be used for depth motion analysis. By generating the above masks separately using the expression region and frame as indices, the entire face region can be avoided from being evaluated uniformly, thereby reducing the impact of local occlusion, local overexposure, local depth holes, or local motion blur on the overall expression recognition results.
[0140] For color validity masks, the processor performs occlusion detection, brightness dispersion detection, and motion blur detection on the color pixels within each expression region. Occlusion detection can be determined by considering skin color distribution, edge continuity, keypoint confidence, and regional texture anomalies. For example, if a connected region to which a color pixel belongs does not meet a preset skin color range, and the confidence of the facial keypoints in the expression region where the color pixel is located is below 0.60, the color pixel can be determined as meeting the preset occlusion condition. Brightness dispersion detection can be determined based on local brightness variance, the proportion of overexposed pixels, and the proportion of underexposed pixels. For example, if the proportion of pixels with a grayscale value greater than 245 or the proportion of pixels with a grayscale value less than 10 in a certain expression region exceeds 20%, the corresponding pixel can be determined as meeting the preset brightness dispersion condition. Motion blur detection can be determined based on regional gradient energy or Laplacian response value. For example, if the Laplacian response value of a pixel's neighborhood is lower than a preset blur response threshold, the pixel can be determined as meeting the preset motion blur condition. The preset skin tone range, facial landmark confidence level, pixel ratio, and preset blur response threshold can be set based on RGB-D camera calibration data or training sample statistical results. When a color pixel meets preset occlusion conditions, preset brightness dispersion conditions, or preset motion blur conditions, the color pixel is marked as an invalid color pixel.
[0141] The processor generates a color validity mask based on the positional distribution of invalid color pixels within the corresponding facial expression area, and determines the valid color values according to the following formula (2):
[0142]
[0143] in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number The effective color values for each facial expression region Indicates the first Frame number The number of colored pixels that meet the preset occlusion conditions within each facial expression area. Indicates the first Frame number The number of colored pixels within each facial expression area that meet the preset brightness dispersion condition. Indicates the first Frame number The number of colored pixels within each facial expression area that meet the preset motion blur conditions. Indicates the first Frame number The total number of colored pixels in each expression area, and .
[0144] For example, in the mouth region of frame 11, the expression region is numbered 3, the total number of color pixels is 1200, the number of color pixels that meet the preset occlusion condition is 90, the number of color pixels that meet the preset brightness dispersion condition is 150, and the number of color pixels that meet the preset motion blur condition is 60. Then, according to formula (2), the effective color value of the 3rd expression region in frame 11 is:
[0145]
[0146]
[0147] Therefore, the processor determines the effective color value of the third facial expression region in frame 11 to be 0.9167. This value is used for the calculation of subsequent region modality fusion weights.
[0148] For the depth validity mask, the processor performs depth hole detection, depth abrupt change detection, and inter-frame depth consistency detection on depth pixels within each expression region. Depth hole detection determines whether a depth pixel is missing or invalid. For example, if a depth pixel value is 0, exceeds the effective range of the RGB-D camera, or is marked as an invalid depth value by the device, the depth pixel is considered to meet a preset depth hole condition. Depth abrupt change detection determines whether a local depth value exhibits an abnormal abrupt change relative to its neighboring depth values. For example, if the difference between a depth pixel and the median of its neighboring valid depth pixels is greater than 80 mm, the depth pixel can be considered to meet a preset depth abrupt change condition. Inter-frame depth consistency detection determines whether the depth changes of the same expression region in adjacent frames conform to the continuous change pattern of facial movements. For example, if the change in depth of a depth pixel within the same expression region exceeds 120 mm between adjacent frames, and this change is inconsistent with the direction of change of surrounding valid depth pixels, the depth pixel can be considered to not meet a preset inter-frame depth consistency condition. The thresholds in the preset depth abrupt change condition and the preset inter-frame depth consistency condition can be set according to the depth noise of the RGB-D camera, the sampling distance, and the scale of the expression region. When a depth pixel meets the preset depth hole condition, the preset depth abrupt change condition, or does not meet the preset inter-frame depth consistency condition, the depth pixel is marked as an invalid depth pixel.
[0149] The processor generates a depth validity mask based on the positional distribution of depth invalid pixels within the corresponding facial expression area, and determines the depth validity value according to the following formula (3):
[0150]
[0151] in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number The effective depth value of each facial expression region. Indicates the first Frame number The number of depth pixels within an expression region that meet the preset depth hole condition. Indicates the first Frame number The number of depth pixels within each facial expression region that satisfy the preset depth abrupt change conditions. Indicates the first Frame number The number of depth pixels within an expression region that do not meet the preset inter-frame depth consistency condition. Indicates the first Frame number The total number of depth pixels within each facial expression region, and .
[0152] For example, in the mouth region of frame 11, the expression region is numbered 3, the total number of depth pixels is 1100, the number of depth pixels that meet the preset depth hole condition is 55, the number of depth pixels that meet the preset depth abrupt change condition is 44, and the number of depth pixels that do not meet the preset inter-frame depth consistency condition is 22. Then, according to formula (3), the effective depth value of the 3rd expression region in frame 11 is:
[0153]
[0154]
[0155] Therefore, the processor determined the effective depth value of the third facial expression region in frame 11 to be 0.9633.
[0156] The processor determines the region modality fusion weights based on the effective color value, the effective depth value, and a preset stable value. The region modality fusion weights include color fusion weights and depth fusion weights. The color fusion weights control the contribution of color texture features in the fusion process, while the depth fusion weights control the contribution of depth motion features. The processor determines the color fusion weights and depth fusion weights according to the following formulas (4) and (5):
[0157]
[0158]
[0159] in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number Color blending weights for each facial expression region Indicates the first Frame number The depth fusion weight of each facial expression region Indicates the first Frame number The effective color values for each facial expression region Indicates the first Frame number The effective depth value of each facial expression region. This indicates a preset stable value, and In one example, the default stability value is 0.05.
[0160] Taking the mouth region of frame 11 above as an example, the effective value of color is 0.9167, the effective value of depth is 0.9633, and the preset stability value is 0.05. Then, according to formula (4), the color fusion weight is:
[0161]
[0162]
[0163] According to formula (5), the deep fusion weights are:
[0164]
[0165]
[0166] Therefore, in the mouth region of frame 11, the depth fusion weight is 0.5118 and the color fusion weight is 0.4882. Figure 6 The curves showing the changes in color fusion weight and depth fusion weight of the mouth region over eight consecutive frames are presented. These curves represent the process by which the regional modal fusion weights adjust as the effective values of color and depth change in different frames.
[0167] During the region fusion feature generation process, the processor extracts color texture features corresponding to each expression region from the color face image sequence. Color texture features may include region grayscale gradients, local binary textures, convolutional feature maps, edge orientation histograms, or region texture embedding features extracted by a neural network. The processor also extracts depth motion features corresponding to each expression region from the depth face image sequence. Depth motion features include depth displacement direction, depth displacement magnitude, and region deformation state. The depth displacement direction describes the three-dimensional movement direction of the expression region in consecutive frames; the depth displacement magnitude describes the depth change of the expression region in consecutive frames; and the region deformation state describes at least one of the following states: expansion, contraction, upward movement, downward movement, forward convexity, and backward retraction of the expression region.
[0168] In one specific implementation, the processor determines the depth displacement direction and magnitude based on the change in the depth center point of each expression region across consecutive frames. The depth center point can be determined by the weighted center of the effective depth pixels within the corresponding expression region. The processor also determines the region deformation state based on the changes in the region boundary and depth distribution of each expression region across consecutive frames. For example, when the horizontal boundary distance of the mouth region increases, it can be determined as a lateral expansion state; when the distance between the left and right brow keypoints of the eyebrow region decreases, it can be determined as a converging state; when the effective depth pixels of the cheek region move as a whole towards the camera, it can be determined as a forward convex state; when the mouth region or cheek region moves as a whole away from the camera, it can be determined as a backward retraction state. The above-mentioned region deformation states are used as input for subsequent motion transfer recognition branches.
[0169] In one example, the depth value of the center point of the mouth region is 1032 mm in frame 10, 1048 mm in frame 11, and 1063 mm in frame 12. Taking the direction away from the camera as positive, the depth displacement amplitude from frame 10 to frame 11 is 16 mm, and from frame 11 to frame 12 is 15 mm, with the depth displacement direction always away from the camera. If, within the same time period, the horizontal distance between the left and right corners of the mouth region increases from 42 pixels to 51 pixels, and the vertical distance between the lower boundary of the mouth region and the tip of the nose increases from 86 pixels to 93 pixels, the processor determines the deformation state of the mouth region as a lateral expansion state and a downward shift state. The aforementioned depth displacement direction, depth displacement amplitude, and regional deformation state together constitute the depth motion features of the mouth region.
[0170] The processor multiplies the color texture features corresponding to each expression region by the corresponding color fusion weight to obtain weighted color texture features; it also multiplies the depth motion features corresponding to the same expression region by the corresponding depth fusion weight to obtain weighted depth motion features. Subsequently, the weighted color texture features and weighted depth motion features corresponding to the same expression region are concatenated to obtain region-concatenated features; finally, the region-concatenated features are compressed and encoded to obtain region-fused expression features. Compression encoding can be implemented using fully connected layers, convolutional compression layers, gated coding layers, or lightweight transform networks. The processor combines multiple region-fused expression features according to frame sequence numbers to generate a region-fused expression feature sequence.
[0171] In the expression recognition stage, the processor inputs the region-fused expression feature sequence into the action transfer recognition branch and the expression category recognition branch, respectively. The action transfer recognition branch outputs the region deformation state transition result for each expression region based on the region-fused expression feature sequence, and generates a region action transfer sequence from multiple region deformation state transition results. The expression category recognition branch outputs the category confidence scores for multiple preset expression categories based on the region-fused expression feature sequence, and determines the preset expression categories whose category confidence scores meet the preset confidence conditions as candidate expression categories.
[0172] In one implementation, the action transfer recognition branch and the expression category recognition branch can use the region fusion expression feature sequence output by the same feature encoding network as input. Training samples include color face image sequences, depth face image sequences, expression category annotations, and region deformation state annotations. The action transfer recognition branch is trained using manually annotated or semi-automatically annotated region deformation state transfer results as supervisory labels, and the expression category recognition branch is trained using manually annotated expression categories as supervisory labels. During training, the weighted sum of the region deformation state cross-entropy loss and the expression category cross-entropy loss can be used as the training loss, or temporal consistency constraints can be applied to constrain the output results of consecutive frames. Preset expression categories can include smiling, frowning, surprised, confused, uncomfortable, and calm categories. Preset confidence conditions can be that the highest category confidence is not less than 0.65, or the difference between the highest category confidence and the second highest category confidence is not less than 0.12.
[0173] The processor further outputs facial action categories and expression recognition results based on action-expression consistency verification rules. These rules store or express the correspondence between candidate expression categories and region action transfer sequences. These rules are not subjective human evaluations, but rather a set of rules for technically verifying the model's output. Table 1 shows an example of action-expression consistency verification rules.
[0174] Table 1. Expression Category Deformation Status and Verification Method
[0175] Candidate emoji categories Necessary regional deformation state combinations in regional action transfer sequences Verification processing method Smile Category The mouth area is in a state of horizontal expansion, and the cheek area is in a state of upward shift. Output the smile category if the expression meets the criteria; otherwise, mark it as an expression category to be confirmed. Frowning categories The eyebrow area is in a state of contraction towards each other, and the eyebrow area is in a state of downward shift. Output the frown category if the condition is met; otherwise, mark it as an expression category to be confirmed. Surprise Category The eye area is in a state of expansion, and the mouth area is in a state of downward displacement. Output the "surprise" category if the condition is met; otherwise, mark it as an "expression to be confirmed" category. Category of doubt The eyebrow area has shifted upwards, while the mouth area has contracted. Output the "Question" category if the condition is met; otherwise, mark it as an "Unresolved" expression category. Discomfort Category The mouth area is retracted, or the cheek area is shifted downward. Output the discomfort category if the condition is met; mark it as an expression category to be confirmed if the condition is not met. Calm category Each facial expression area did not form a continuous deformation state that reached the preset deformation threshold. Output the calm category if the condition is met; otherwise, mark it as an expression category to be confirmed.
[0176] When a candidate expression category and the region action transfer sequence satisfy the correspondence in the action-expression consistency verification rules, the processor determines the candidate expression category as the expression recognition result and determines the combination of region deformation states corresponding to the region action transfer sequence as the facial action category. When a candidate expression category and the region action transfer sequence do not satisfy the correspondence in the action-expression consistency verification rules, the processor marks the candidate expression category as an expression category to be confirmed and reduces the category confidence of the expression category to be confirmed to below a preset confirmation threshold. For example, the preset confirmation threshold can be 0.45, so that the candidate expression category will not be directly output as the highest priority expression recognition result, or will only be used as low-priority reference information in subsequent processing.
[0177] For example, the category confidence scores output by the expression category recognition branch are: surprise 0.71, smile 0.14, confusion 0.08, discomfort 0.04, and calm 0.03. Since the highest category confidence score of 0.71 is higher than 0.65, the processor identifies the surprise category as a candidate expression category. Meanwhile, the region motion transfer sequence output by the action transfer recognition branch includes an eye region expansion state and a mouth region downward movement state. According to the action-expression consistency verification rule, the surprise category and the combination of "eye region expansion state and mouth region downward movement state" satisfy a correspondence. Therefore, the processor identifies the surprise category as the expression recognition result and identifies the combination of eye region expansion state and mouth region downward movement state as the facial action category. If the region motion transfer sequence output by the action transfer recognition branch only includes the mouth region horizontal expansion state and does not include the eye region expansion state, the processor marks the surprise category as an expression category to be confirmed and reduces the category confidence score corresponding to this candidate expression category to below 0.45.
[0178] When determining the interaction intent label, the processor extracts the effective depth pixels of the nose region and the effective depth pixels of the cheek region from the depth face image sequence based on the depth validity mask, and determines the user's facial reference depth value based on the effective depth pixels of the nose region and the effective depth pixels of the cheek region. The user's facial reference depth value can be a weighted average of the median effective depth pixels of the nose region and the median effective depth pixels of the cheek region. The processor determines the user's facial reference depth value according to the following formula (6):
[0179]
[0180] in, This represents the user's facial reference depth value, in meters. This represents the median effective depth in pixels for the nasal region, expressed in meters. This represents the median effective depth pixels of the cheek region, in meters. Indicates the weight of the nasal region. This represents the weight of the cheek region, and .
[0181] The processor determines the user distance level based on the user's facial reference depth value and determines the direction of distance change based on the user's facial reference depth values in adjacent frames. The user distance level can include near distance, medium distance, and far distance levels. For example, a user facial reference depth value less than 0.8 meters is determined to be at a near distance level; a user facial reference depth value not less than 0.8 meters and less than 1.6 meters is determined to be at a medium distance level; and a user facial reference depth value not less than 1.6 meters is determined to be at a far distance level. The direction of distance change can include the direction of approach, the direction of departure, and the direction of stillness.
[0182] In one example, the median effective depth pixel value of the nose region in frame 11 is 1.12 meters, the median effective depth pixel value of the cheek region is 1.18 meters, the weight of the nose region is 0.6, and the weight of the cheek region is 0.4. Therefore, according to formula (6), the user's facial reference depth value is:
[0183]
[0184]
[0185] Since the user's facial reference depth value is 1.144 meters, falling between 0.8 meters and 1.6 meters, the processor classifies the user's distance level as medium. If the user's facial reference depth value is 1.102 meters in frame 10 and 1.144 meters in frame 11, and the distance change threshold is 0.03 meters, then the distance change in frame 11 relative to frame 10 is 0.042 meters, which is greater than the distance change threshold. Therefore, the processor classifies the direction of the user's distance change as the direction of moving away.
[0186] The processor filters the available intent set from the state-conditional intent mapping correspondence based on the current interaction state. The current interaction state can include standby state, question-and-answer state, service execution state, and soothing state. The state-conditional intent mapping correspondence records the correspondence between facial action category, expression recognition result, user distance level, user distance change direction, current interaction state, candidate intent label, and mapping priority. The processor selects candidate intent labels from the available intent set that simultaneously match the facial action category, expression recognition result, user distance level, and user distance change direction.
[0187] In one example, when the current interaction state is a service execution state, the facial expression recognition result is a questioning category, and the user distance level is a medium distance or a near distance level, the candidate intent label can be a task confirmation label; when the current interaction state is a soothing state, the facial expression recognition result is an discomfort category, and the user distance change direction is a moving away direction, the candidate intent label can be a soothing adjustment label; when the current interaction state is a question-and-answer state, the facial expression recognition result is a smiling category, and the user distance change direction is a staying direction, the candidate intent label can be a continue interaction label; when the user distance level is a near distance level and the facial action category includes a recoil state, the candidate intent label can be an avoidance prompt label.
[0188] The processor further reads the feedback expression recognition results and the direction of change in the distance to the user from the execution feedback samples, and determines the priority score of the candidate intent label according to the following formula (7):
[0189]
[0190] in, Indicates the candidate intent label number, Indicates the first Priority scoring of each candidate intent tag. The state conditionalization intention mapping correspondence in the first... The basic priority of each candidate intent tag , and Let represent the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient, respectively. , , ShS_hSh represents the first... The number of times the positive feedback judgment condition is met in the execution feedback samples corresponding to each candidate intent label. Indicates the first The number of times the negative feedback judgment condition is met in the execution feedback samples corresponding to each candidate intent label. Indicates the first The number of times a candidate intent label is repeatedly executed within the same interaction round. Positive feedback criteria include a non-negative expression category for the feedback facial expression recognition result, or the user's distance change direction being the direction of stillness. Negative feedback criteria include a negative expression category for the feedback facial expression recognition result, or the user's distance change direction being the direction of movement away. Negative expression categories include frowning, confusion, and discomfort; non-negative expression categories include smiling and calm. When the same execution feedback sample simultaneously meets both positive and negative feedback criteria, it is marked as a negative execution feedback sample and counted in the number of times the negative feedback criteria are met. The processor determines the candidate intent label with the highest priority score as the interaction intent label.
[0191] For example, in a certain interaction round, the available intent set includes the task confirmation label, the reassurance and adjustment label, and the continue interaction label. For the task confirmation label, the basic priority is 0.60, the first weight coefficient is 0.06, the second weight coefficient is 0.08, and the third weight coefficient is 0.04; within the preset statistical window, the number of times the positive feedback judgment condition is met in the execution feedback sample corresponding to the task confirmation label is 4, the number of times the negative feedback judgment condition is met is 1, and the number of repeated executions in the same interaction round is 1. Then, according to formula (7), the priority score of the task confirmation label is:
[0192]
[0193]
[0194] in, This indicates the priority score of the task confirmation tag. For the appeasement and adjustment tag, the base priority is 0.58, the number of times the positive feedback judgment condition is met is 2, the number of times the negative feedback judgment condition is met is 2, and the number of repeated executions is 0. Therefore, according to formula (7), the priority score of the appeasement and adjustment tag is:
[0195]
[0196]
[0197] in, This indicates the priority score for the "Reassure and Adjust" tag. For the "Continue Interaction" tag, the base priority is 0.52, the number of times the positive feedback condition is met is 3, the number of times the negative feedback condition is met is 1, and the number of repetitions is 2. Therefore, according to formula (7), the priority score for the "Continue Interaction" tag is:
[0198]
[0199]
[0200] in, This indicates the priority score for the "continue interaction" tag. Therefore, the processor determines the task confirmation tag with the highest priority score as the interaction intent tag. Figure 7 The curve showing the priority score change of the task confirmation label in multiple feedback statistical rounds is shown. This curve is used to represent the combined effect of the number of times the positive feedback judgment condition is met, the number of times the negative feedback judgment condition is met, and the number of repeated executions on the mapping priority.
[0201] After the service robot executes the interaction response command, the processor collects subsequent video clips of the user according to a preset feedback sampling duration, and obtains subsequent expression recognition results, subsequent facial action categories, and subsequent user distance change directions based on these video clips. The processor generates execution feedback samples based on these factors. When the subsequent expression recognition result belongs to a non-negative expression category, or the subsequent user distance change direction is a stationary direction, and the same execution feedback sample does not meet the negative feedback judgment condition, the processor marks the corresponding execution feedback sample as a positive execution feedback sample. When the subsequent expression recognition result belongs to a negative expression category, or the subsequent user distance change direction is a moving away direction, the processor marks the corresponding execution feedback sample as a negative execution feedback sample. The processor updates the mapping priority corresponding to the interaction intent label in the state conditional intent mapping correspondence based on the number of positive and negative execution feedback samples corresponding to the same interaction intent label within a preset statistical window.
[0202] Within the same interaction round, when the number of times the same interaction response instruction generates negative execution feedback samples reaches a preset feedback count threshold, the processor adds the interaction response instruction to the prohibited execution list for the current interaction round. Then, based on the current interaction state, the processor generates a set of available intents for the current interaction round from the state-conditional intent mapping correspondence. Intent tags that have a mapping relationship with the interaction response instructions in the prohibited execution list are excluded from the set of available intents, and the interaction intent tags are re-determined based on the excluded set of available intents. In this way, the service robot can avoid continuously repeating interaction response instructions that have generated negative feedback within the same interaction round.
[0203] In terms of execution configuration, the processor determines the interaction response instruction from the robot service instruction library based on the interaction intent label. When the interaction response instruction is a voice instruction, the service robot's voice broadcasting unit is invoked to execute the voice response; when the interaction response instruction is a motion instruction, the service robot's body motion unit is invoked to execute the motion demonstration; when the interaction response instruction is a service instruction, the service robot's task execution unit is invoked to execute the service task. The processor writes the execution time, execution type, interaction intent label, and corresponding execution feedback sample of the interaction response instruction into the interaction record table. The interaction record table is used to count the number of times positive feedback judgment conditions are met, the number of times negative feedback judgment conditions are met, and the number of repeated executions, and provides a data source for the priority scoring of subsequent candidate intent labels.
[0204] To verify the technical effectiveness of this embodiment, an interaction scheme using only overall facial emotion images and fixed mapping rules was used as a comparative example, and the method described in this embodiment was used as the experimental group for simulated testing in a smart home system scenario. The comparative example did not use expression region-level color validity masks or depth validity masks, nor did it employ action-expression consistency verification rules or execution feedback sample update mapping priority. The tests were conducted under the same RGB-D camera, the same lighting arrangement, the same set of continuous user video clips, and the same robot service instruction library. The test environment included five types of scenarios: normal lighting, backlighting, partial occlusion, motion blur, and deep holes. For each scenario, 120 continuous user video clips were collected, and each video clip contained a major expression change process. During the test, manually labeled expression categories, facial action categories, and interaction intent labels were used as evaluation benchmarks. Test indicators included expression recognition accuracy, facial action category consistency rate, interaction intent label matching rate, incompatible response repetition rate within the same interaction round, and average response adjustment rounds. Figure 5 The curves showing the comparison between recognition and intent matching results under different test scenarios are presented. Figure 8 A bar chart comparing the recognition accuracy of different facial expression categories is shown, and Table 2 shows the statistical results for each test scenario.
[0205] Table 2 Comparison of Test Results
[0206] Test Scenario Test Plan Facial expression recognition accuracy Consistency rate of facial movement categories Interaction Intent Tag Matching Rate Mismatched response repetition rate Average response adjustment rounds Normal lighting Comparative Example 89.2% 84.6% 82.5% 13.8% 2.1 Normal lighting This embodiment 94.8% 91.7% 90.4% 6.2% 1.4 Backlight Comparative Example 76.5% 70.8% 68.9% 22.7% 2.9 Backlight This embodiment 88.6% 84.2% 82.8% 9.5% 1.8 Partial occlusion Comparative Example 72.1% 69.3% 66.4% 25.4% 3.2 Partial occlusion This embodiment 86.9% 83.5% 81.7% 10.8% 1.9 Motion blur Comparative Example 74.8% 68.7% 67.2% 24.9% 3.1 Motion blur This embodiment 87.5% 82.9% 80.6% 11.6% 2.0 Deep Void Comparative Example 78.6% 71.5% 69.7% 21.2% 2.8 Deep Void This embodiment 89.1% 85.6% 83.3% 9.9% 1.7
[0207] As shown in Table 2, under backlighting, partial occlusion, motion blur, and deep hole scenarios, the expression recognition accuracy of this embodiment is 88.6%, 86.9%, 87.5%, and 89.1%, respectively, and the facial action category consistency rate is 84.2%, 83.5%, 82.9%, and 85.6%, respectively. Compared with the comparative example, this embodiment has higher expression recognition accuracy and facial action category consistency rate under the above four types of acquisition quality fluctuation scenarios. At the same time, the mismatch response repetition rate of this embodiment under the above four scenarios is 9.5%, 10.8%, 11.6%, and 9.9%, respectively, which is lower than 22.7%, 25.4%, 24.9%, and 21.2% in the comparative example; the average response adjustment rounds are 1.8, 1.9, 2.0, and 1.7, respectively, which is lower than 2.9, 3.2, 3.1, and 2.8 in the comparative example. The test results above demonstrate that this embodiment can address the issues of fluctuating quality in color and depth image acquisition, inconsistent reliability of information in different facial local areas, and underutilization of continuous interactive feedback by providing a dynamic expression recognition and interactive intent adjustment process.
[0208] In another implementation, the RGB-D camera can be replaced with a combined camera with a color imaging unit and a structured light depth imaging unit, or with a combined camera with a color imaging unit and a time-of-flight depth imaging unit. The action transfer recognition branch can employ a recurrent neural network, a temporal convolutional network, a Transformer coding network, or a gated recurrent network. The expression category recognition branch can employ a convolutional neural network, a lightweight visual network, or a visual Transformer network. Compression coding can employ fully connected dimensionality reduction, convolutional dimensionality reduction, attention coding, or channel compression coding. All of the above models are used to process region-fused expression feature sequences, and output expression recognition results, facial action categories, and interaction intent labels, without changing the processing object and data flow direction for determining interaction intent labels based on facial expression recognition results.
[0209] In this embodiment, a continuous processing relationship is formed among the color validity mask, depth validity mask, regional modality fusion weights, regional fusion facial expression feature sequences, action-expression consistency verification rules, state-conditional intent mapping correspondence, execution feedback samples, and mapping priorities. This continuous processing relationship enables the service robot to recognize interaction intent in continuous interactive environments such as smart home systems, based on facial expression recognition results, facial action categories, user distance levels, current interaction states, and changes in interaction feedback in the user's facial image sequence, and to control the service robot to execute corresponding voice responses, action demonstrations, or service tasks.
[0210] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.
Claims
1. A method for recognizing robot human-computer interaction intentions based on multimodal perception, characterized in that, include: The service robot uses an RGB-D camera to capture continuous video clips of the user, simultaneously extracts color face image sequences and depth face image sequences, and divides multiple expression regions based on facial key points; For each of the aforementioned expression regions and each frame, a color validity mask and a depth validity mask are generated. Based on the color validity value determined by the color validity mask, the depth validity value determined by the depth validity mask, and a preset stable value, the region modality fusion weight is determined. Color texture features corresponding to each expression region are extracted from the color face image sequence, and depth motion features including depth displacement direction, depth displacement amplitude and region deformation state are extracted from each expression region from the depth face image sequence. A region fusion expression feature sequence is generated according to the region modality fusion weight. The region-fused facial expression feature sequence is input into the action transfer recognition branch and the expression category recognition branch respectively to obtain the region action transfer sequence and candidate expression categories. The facial action category and expression recognition result are then output according to the action-expression consistency verification rule. The interaction intent label is determined based on the facial action category, the expression recognition result, the user distance level, the current interaction state of the service robot, and the mapping priority in the state conditional intent mapping correspondence. The service robot is controlled to execute the interactive response instruction corresponding to the interactive intent tag. Subsequent video clips of the user after execution are collected to generate execution feedback samples. The mapping priority is updated based on the execution feedback samples. The updated mapping priority is used to determine the subsequent interactive intent tags.
2. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 1, characterized in that, The simultaneous extraction of color face image sequences and depth face image sequences includes: The user's continuous video segments are divided into multiple candidate frame groups according to the acquisition timestamp. Each candidate frame group includes a color frame and a depth frame corresponding to the color frame. In each of the candidate frame groups, a face bounding box and the face key points are determined; Based on the facial key points, candidate expression regions are determined in the color frame, and candidate depth regions corresponding to the candidate expression regions are determined in the depth frame. The facial expression trigger frame is determined based on the color changes of the candidate expression region and the depth changes of the candidate depth region. Centered on the facial expression trigger frame, the color face image sequence and the depth face image sequence are extracted from multiple consecutive candidate frame groups.
3. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 2, characterized in that, The step of determining the facial expression trigger frame based on the color changes of the candidate expression region and the depth changes of the candidate depth region includes: Determine the action trigger value using the following formula: in, Indicates the candidate expression region number. Indicates the frame number and , Indicates the first Frame number Action trigger values for each candidate expression region, Indicates the first Frame number The average grayscale value of each candidate expression region Indicates the first Frame number The average grayscale value of each candidate expression region This indicates the preset grayscale normalization value, and , Indicates the first Frame number The average depth value of each candidate depth region Indicates the first Frame number The average depth value of each candidate depth region This represents the preset depth normalization value, and ; When the action trigger value in each of the first three consecutive frames reaches the preset action trigger threshold, and the depth change direction of the candidate depth region remains consistent in the three consecutive frames, the second frame in the three consecutive frames is determined as the expression action trigger frame.
4. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 1, characterized in that, The method of dividing multiple expression regions based on facial key points includes: Establish a facial region coordinate system based on the nose tip, the center of the eyes, and the corner of the mouth among the facial key points; Map the depth face image sequence to the face region coordinate system; The face region coordinate system is divided into eyebrow region, eye region, nose region, mouth region and cheek region; The eyebrow area, the eye area, the nose area, the mouth area, and the cheek area are defined as the expression area.
5. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 4, characterized in that, The generation of the color validity mask includes: Occlusion detection, brightness discrepancy detection, and motion blur detection are performed on the colored pixels in each of the aforementioned expression regions; Color pixels that meet the preset occlusion conditions, color pixels that meet the preset brightness dispersion conditions, and color pixels that meet the preset motion blur conditions are marked as invalid color pixels; The color validity mask is generated based on the positional distribution of the invalid color pixels within the corresponding expression area, and the color validity value is determined according to the following formula: in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number The effective color values for each facial expression region Indicates the first Frame number The number of colored pixels within each facial expression area that satisfy the preset occlusion conditions. Indicates the first Frame number The number of color pixels within each facial expression region that satisfy the preset brightness dispersion condition. Indicates the first Frame number The number of color pixels within each facial expression region that satisfy the preset motion blur condition. Indicates the first Frame number The total number of colored pixels in each expression area, and .
6. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 5, characterized in that, The generation of the depth validity mask includes: For each depth pixel within the aforementioned expression region, depth hole detection, depth abrupt change detection, and inter-frame depth consistency detection are performed respectively. Depth pixels that meet the preset depth hole condition, depth pixels that meet the preset depth mutation condition, and depth pixels that do not meet the preset inter-frame depth consistency condition are marked as depth invalid pixels. The depth validity mask is generated based on the positional distribution of the depth invalid pixels within the corresponding expression area, and the depth validity value is determined according to the following formula: in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number The effective depth value of each facial expression region. Indicates the first Frame number The number of depth pixels within an expression region that satisfy the preset depth hole condition. Indicates the first Frame number The number of depth pixels within each facial expression region that satisfy the preset depth abrupt change condition. Indicates the first Frame number The number of depth pixels within an expression region that do not meet the preset inter-frame depth consistency condition. Indicates the first Frame number The total number of depth pixels within each facial expression region, and .
7. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 6, characterized in that, The determined regional modal fusion weights include: Based on the effective color value, the effective depth value, and the preset stable value, the color blending weight and the depth blending weight are determined according to the following formula: in, Indicates the number of the expression area. Indicates the frame number. Indicates the first Frame number Color blending weights for each facial expression region Indicates the first Frame number The depth fusion weight of each facial expression region Indicates the first Frame number The effective color values for each facial expression region Indicates the first Frame number The effective depth value of each facial expression region. This represents the preset stable value, and The regional modal fusion weights include the color fusion weights and the depth fusion weights.
8. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 7, characterized in that, The step of generating a region-fused facial expression feature sequence according to the region modality fusion weights includes: Multiply the color texture feature corresponding to each of the aforementioned expression regions by the corresponding color fusion weight to obtain a weighted color texture feature; Multiply the deep motion features corresponding to the same facial expression region by the corresponding depth fusion weight to obtain weighted deep motion features; The weighted color texture features and the weighted depth motion features corresponding to the same facial expression region are spliced together to obtain the region splicing features; The region splicing features are compressed and encoded to obtain region fusion expression features; The region-fused facial expression features are generated by combining multiple region-fused facial expression features according to the frame number.
9. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 8, characterized in that, The step of outputting facial action categories and expression recognition results based on action-expression consistency verification rules includes: The action transfer recognition branch outputs the regional deformation state transfer result of each of the facial expression regions based on the regional fusion facial expression feature sequence, and generates the regional action transfer sequence from multiple regional deformation state transfer results; The expression category recognition branch outputs the category confidence scores of multiple preset expression categories based on the region fusion expression feature sequence, and determines the preset expression categories whose category confidence scores meet the preset confidence conditions as the candidate expression categories; When the candidate expression category and the region action transfer sequence satisfy the correspondence in the action-expression consistency verification rule, the candidate expression category is determined as the expression recognition result, and the combination of the region deformation state corresponding to the region action transfer sequence is determined as the facial action category; When the candidate expression category and the region action transfer sequence do not satisfy the correspondence in the action-expression consistency verification rule, the candidate expression category is marked as an expression category to be confirmed, and the category confidence of the expression category to be confirmed is reduced to below the preset confirmation threshold.
10. The robot human-computer interaction intent recognition method based on multimodal perception according to claim 1, characterized in that, The execution feedback sample is generated in the following manner: After the service robot executes the interactive response command, it collects subsequent video clips from the user according to a preset feedback sampling duration. Based on the user's subsequent video clips, obtain subsequent expression recognition results, subsequent facial action categories, and subsequent user distance change directions; The execution feedback sample is generated based on the subsequent expression recognition results, the subsequent facial action category, and the subsequent user distance change direction.
Citation Information
Patent Citations
User-robot cross-media interaction system and method
CN115946139A
Robot behavior mode dynamic adjustment method based on multi-mode perception
CN120257050A
Industrial robot intelligent obstacle avoidance control method and system based on visual identification
CN121290406A
