A user expression prediction and robot proactive response method, device and medium

Abstract

The application discloses a user expression prediction and robot active response method, equipment and medium, relates to the technical field of expression prediction and human-computer interaction, and comprises the following steps: aligning historical dialogue window data and historical face image window data to obtain multi-modal input data, and calculating visual reliability and dialogue reliability; only screening new dialogue segments and new face image frames as to-be-encoded data to carry out encoding and complete incremental time sequence state updating in combination with the buffered historical hidden state; completing causal cross-modal fusion through a gating network, and realizing dynamic allocation of modal contribution in combination with the reliability of each mode; completing multi-task prediction output by a future expression prediction head, carrying out confidence gating judgment, generating an early trigger flag to drive the robot to output a matching expression, giving consideration to data quality constraints and time sequence reasoning efficiency of the multi-modal input, and improving the reliability and rationality of the robot in responding to the future expression of the user.

Description

Technical Field

[0001] This invention relates to the fields of facial expression prediction and human-computer interaction technology, and in particular to a method, device and medium for predicting user facial expressions and for a robot to actively respond. Background Technology

[0002] Companion robots, service robots, and digital human interaction systems have been widely used in human-computer interaction scenarios. During human-computer interaction, interaction data such as facial images and dialogue content can be collected to perceive the user's emotional expressions and complete the interactive feedback on the robot side. Continuous expression sequences and dialogue context information can also be used to analyze and judge the user's psychological state and emotional change trends.

[0003] One existing interactive processing method involves collecting the current facial image or current dialogue content, recognizing the user's expressions or emotions, and then controlling the robot to provide corresponding feedback based on the recognition results. Other technical solutions rely on continuous facial expression sequences to infer the user's psychological state, or combine the dialogue context to judge the trend of emotional changes and output the user's current emotions, psychological state, or general emotional change trends.

[0004] Existing post-recognition response solutions are affected by the time consumption of multiple processing stages such as image acquisition, speech transcription, feature extraction, model inference, decision-making, and actuator motion. The robot's facial expression feedback lags behind the user's actual facial expression, which can easily lead to mechanical following, delayed response, and emotional misalignment. Some emotion trend analysis solutions can only output general emotion trends and cannot output the expression category, occurrence time, and prediction confidence in the future time period, making it difficult to combine with the latency of various robot systems to carry out early response. Single-modal solutions are prone to misjudgment due to problems such as lighting, occlusion, speech recognition errors, and inconsistency between speech and facial expression. Under low confidence, triggering response may also result in situations such as preemptive expression and inappropriate interaction. Summary of the Invention

[0005] This invention provides a method, device, and medium for user facial expression prediction and robot proactive response, in order to solve the problems of delayed or misaligned facial expression responses when relying on existing recognition results for robot feedback, the inability of existing emotion analysis to support early responses with time delay adaptation, easy misjudgment of single modality, erroneous triggering of interaction under low confidence, and high computing power overhead for continuous inference at the edge.

[0006] According to one aspect of the present invention, a method for predicting user facial expressions and for a robot to actively respond is provided, comprising:

[0007] Collect historical dialogue window data and historical face image window data up to the current prediction time, and align the data to obtain multimodal input data;

[0008] Perform dialogue modality extraction and visual modality extraction operations on multimodal input data to obtain dialogue modality and visual modality. Calculate visual reliability based on multimodal input data and face image quality indicators, calculate dialogue reliability based on multimodal input data and speech recognition and sentence completeness indicators, and calculate overall modality reliability based on dialogue reliability and visual reliability.

[0009] If the current prediction time is the first round of prediction, then the preset initial state will be used as the historical hidden state, and the multimodal input data will be used as the data to be encoded.

[0010] If the current prediction time is not the first prediction, then obtain the historical hidden state cached from the previous prediction as well as the encoded dialogue and image vectors. Compare the multimodal input data with the encoded dialogue and image vectors, and filter out the new dialogue fragments and new face image frames as the data to be encoded.

[0011] Encoding processing is performed on the data to be encoded, and incremental temporal state update operation is performed in combination with the historical hidden state to obtain the updated temporal context state. The temporal context state, dialogue modality, visual modality, dialogue reliability and visual reliability are then input into the gating network to perform causal cross-modal fusion processing to obtain the multimodal fusion state.

[0012] The multimodal fusion state is input into the future expression prediction head, multi-task prediction processing is performed, and the expression category probability, the expected occurrence time of the expression, and the expression intensity are output within a preset time period in the future.

[0013] The probability of each expression category is calibrated. Based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time, confidence gating is performed. When the maximum expression category probability, the difference between the maximum and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, an early trigger flag is generated. This flag triggers the robot to output an expression that matches the maximum expression category probability, the expected expression occurrence time, and the expression intensity.

[0014] According to another aspect of the present invention, a user expression prediction and robot active response device is provided, comprising:

[0015] The data acquisition and alignment module is used to acquire historical dialogue window data and historical face image window data up to the current prediction time, and perform data alignment to obtain multimodal input data;

[0016] The reliability calculation module is used to perform dialogue modality extraction and visual modality extraction operations on multimodal input data to obtain dialogue modality and visual modality. It calculates visual reliability based on multimodal input data and face image quality indicators, calculates dialogue reliability based on multimodal input data and speech recognition and sentence completeness indicators, and calculates modality comprehensive reliability based on dialogue reliability and visual reliability.

[0017] The first-round module is used to take the preset initial state as the historical hidden state and the multimodal input data as the data to be encoded if the current prediction time is the first-round prediction.

[0018] The non-first-round module is used to obtain the historical hidden state cached from the previous prediction and the encoded dialogue and image vectors if the current prediction time is not the first-round prediction. It compares the multimodal input data with the encoded dialogue and image vectors and filters out the newly added dialogue fragments and newly added face image frames as the data to be encoded.

[0019] The modality fusion module is used to perform encoding processing on the data to be encoded, perform incremental temporal state update operations in combination with historical hidden states to obtain the updated temporal context state, and input the temporal context state, dialogue modality, visual modality, dialogue reliability and visual reliability into the gating network to perform causal cross-modal fusion processing to obtain the multimodal fusion state;

[0020] The task prediction module is used to input the multimodal fusion state into the future expression prediction head, perform multi-task prediction processing, and output the probability of expression category, the expected time of expression occurrence, and the intensity of expression within a preset time period in the future.

[0021] The "Allow Early Trigger" module is used to perform probability calibration on the expression category probabilities. Based on the calibrated expression category probabilities, modal comprehensive reliability, and expected expression occurrence time, it performs confidence gating judgment. When the maximum expression category probability, the difference between the maximum and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, an "Allow Early Trigger" flag is generated. This flag triggers the robot to output an expression that matches the maximum expression category probability, the expected expression occurrence time, and the expression intensity.

[0022] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0023] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the user expression prediction and robot active response method according to any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the user expression prediction and robot active response method according to any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the method as described in any embodiment of the present invention.

[0026] The technical solution of this invention obtains multimodal input data by aligning historical dialogue window data and historical face image window data. Visual reliability and dialogue reliability are calculated based on face image quality indicators, speech recognition, and sentence completeness indicators, respectively, which can quantitatively characterize the data quality of the two types of modal inputs. Using a first-round or non-first-round prediction differentiation processing logic, the non-first round only selects newly added dialogue segments and newly added face image frames as data to be encoded, and combines this with cached historical hidden states to complete incremental temporal state updates. This eliminates the need to repeatedly encode all historical window data, reducing the encoding computation overhead and memory access volume at the edge. The process is completed through a gating network. Causal cross-modal fusion dynamically allocates modal contributions by combining the reliability of each modality, reducing the interference of low-quality modalities on the fusion results. The future expression prediction head completes multi-task prediction output of expression category probability, expected expression occurrence time, and expression intensity. After probability calibration, confidence gating judgment is carried out by combining probability conditions, modal comprehensive reliability, and expected expression occurrence time. Only when multiple conditions are met simultaneously will an early trigger flag be generated to drive the robot to output a matching expression. Under the condition of limited computing power at the edge, it takes into account the data quality constraints of multimodal input and the efficiency of temporal inference, reduces invalid triggers, and improves the reliability and rationality of the robot's early response to the user's future expression.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1This is a flowchart of a user expression prediction and robot active response method according to Embodiment 1 of the present invention;

[0030] Figure 2 This is a flowchart of another user expression prediction and robot active response method provided in Embodiment 2 of the present invention;

[0031] Figure 3 This is a flowchart of another user expression prediction and robot active response method provided in Embodiment 3 of the present invention;

[0032] Figure 4 This is an advance response timing diagram based on predicted occurrence time and actuator delay applicable to embodiments of the present invention;

[0033] Figure 5 This is a schematic diagram of a user expression prediction and robot active response device according to Embodiment 4 of the present invention;

[0034] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the user expression prediction and robot active response method according to embodiments of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Example 1

[0038] Figure 1This is a flowchart of a user expression prediction and robot active response method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where user expressions are predicted and responded to actively by a robot. This method can be executed by a user expression prediction and robot active response device, which can be implemented in hardware and / or software and is generally configured in an electronic device. Figure 1 As shown, the method includes:

[0039] S110. Collect historical dialogue window data and historical face image window data up to the current prediction time, and align the data to obtain multimodal input data.

[0040] In this embodiment, the current prediction time refers to the system time point when a user expression prediction task is performed. Historical dialogue window data refers to the valid dialogue data between the user and the robot collected within a specified time or dialogue round range up to the current prediction time. Historical face image window data refers to the user face image frame data collected within a specified time range up to the current prediction time.

[0041] Data alignment refers to establishing a temporal mapping relationship between facial image segments and adjacent dialogue turns based on the timestamps or other indicators inherent in each data set, thereby achieving matching of different modalities in the temporal dimension. Multimodal input data refers to fused input data that has undergone time alignment processing and has been marked with missing modality masks, making it usable for subsequent model feature extraction. Missing modality masks are markers used to indicate modalities that experienced data loss during the acquisition process.

[0042] Specifically, historical dialogue window data and historical face image window data up to the current prediction time are collected. Only information from the current time and before is used to generate a multimodal synchronization window, and any data after the current time is prohibited. Then, data alignment processing is performed on the two types of window data to eliminate the time misalignment between dialogue and face images. This can be done by using timestamp mapping and generating a missing modality mask, or by using other alternative methods such as frame number association matching and event triggering markers to generate multimodal input data that matches the time dimension and has modality missing markers, providing basic input materials for subsequent modality feature extraction and expression prediction.

[0043] For example, the dialog window D[tW] before the current time t. d ,t] and face image window I[tW v The input is [t, t], and the output is the probability p(y) of the user's facial expression category and the expected occurrence time τ within the future prediction interval [t+Δt1, t+Δt2]. pred And the predicted intensity spred. The system does not read data after t to ensure causality in online applications.

[0044] Among them, W d This represents the time length (or dialogue round range) corresponding to the dialogue window. The window contains all dialogue content generated at or before time t. v τ represents the time length corresponding to the face image window, which contains all face images acquired at or before time t. Δt1 is the start time offset of the prediction interval from the current time, and Δt2 is the end time offset of the prediction interval from the current time; this interval belongs to the future time period that has not yet occurred. p(y) refers to the probability of the expression category, which is the predicted probability distribution corresponding to each expression category within the future prediction interval, output by the model. pred "Expected occurrence time" refers to the predicted time point within the future prediction interval, based on the model's prediction of the user's target expression. "Spred" refers to the predicted intensity, the predicted magnitude of the user's target expression within the future prediction interval, based on the model's prediction.

[0045] S120. Perform dialogue modality extraction and visual modality extraction operations on the multimodal input data to obtain the dialogue modality and visual modality. Calculate the visual reliability based on the multimodal input data and face image quality indicators. Calculate the dialogue reliability based on the multimodal input data and speech recognition and sentence completeness indicators. Calculate the overall modality reliability based on the dialogue reliability and visual reliability.

[0046] In this embodiment, dialogue modality refers to the set of dialogue-related features obtained after dialogue modality extraction. Visual modality refers to the set of face image-related features obtained after visual modality extraction. Visual reliability is a metric calculated based on face image quality indicators and used to characterize the credibility of face image modality data.

[0047] Speech recognition and sentence completeness metrics are evaluation indicators used to measure the quality of speech-to-text transcription results and the completeness of sentence information. Dialogue reliability is a measure calculated based on speech recognition and sentence completeness metrics, used to characterize the credibility of dialogue modal data. Modal comprehensive reliability is a measure calculated by combining dialogue reliability and visual reliability, characterizing the overall credibility of multimodal input data.

[0048] Specifically, dialogue modality extraction and visual modality extraction operations are performed on the multimodal input data. For example, the dialogue modality is obtained by extracting semantics, emotional cues, negation and irony prompts, dialogue behavior and turn relationships from the dialogue text, and the visual modality is obtained by extracting facial expression action units, key point displacement, local texture and short-term motion changes from the image.

[0049] Visual reliability is calculated based on multimodal input data and face image quality metrics, such as face detection confidence (cface), blur (bblur), and occlusion rate (oocc). For example, it can be solved using a product operation with a limited range of values: qv = clip(cface × (1-bblur) × (1-oocc), 0, 1), where qv is the visual reliability, an output metric representing the trustworthy quality of the face image modality data, with a value limited to the range of 0-1; a larger value indicates higher quality and greater reliability of the current face image modality. cface is the face detection confidence, representing the degree of confidence that the algorithm has detected a face target; its value is generally 0-1, with a higher value indicating a greater probability of successful face detection. bblur is the image blur, representing the degree of blurriness in the face image; its value is 0-1, with a larger value indicating a more blurred image. (1-bblur) is used for attenuation correction; the more blurred the image, the smaller this result, which suppresses the overall reliability.

[0050] oocc is the occlusion rate, representing the proportion of the face area obscured by external objects; it takes a value of 0-1, with a larger value indicating a larger obscured area of ​​the face. (1-oocc) is used for attenuation correction; the more severe the face occlusion, the smaller this result, reducing the overall reliability.

[0051] cface×(1-bblur)×(1-oocc) is an intermediate calculation result of the original visual reliability obtained by multiplying the face detection confidence, blur attenuation term, and occlusion attenuation term, and taking into account the three image quality factors of face detection, blur, and occlusion. The lower the face detection confidence, the more blurred the image, and the more severe the face occlusion, the smaller the product result.

[0052] `clip(…,0,1)` is a clipping and limiting function; it is used to force the intermediate result of the product to be bounded between a lower bound of 0 and an upper bound of 1. If the product result is greater than 1, the output is 1; if it is less than 0, the output is 0; intermediate values ​​are kept unchanged.

[0053] Dialogue reliability is calculated based on multimodal input data and speech recognition and sentence completeness metrics, such as speech recognition confidence, sentence completeness, and effective word ratio.

[0054] Further calculations are performed by combining dialogue reliability and visual reliability, such as using weighted averages or minimum values, to obtain the overall modal reliability. This provides modal features and corresponding reliability metrics for subsequent cross-modal fusion processing.

[0055] S130. If the current prediction time is the first round of prediction, then the preset initial state is used as the historical hidden state, and the multimodal input data is used as the data to be encoded.

[0056] In this embodiment, the first round of prediction refers to the first execution of the facial expression prediction task after the system starts, and the prediction process does not include cached data from the previous round. The preset initial state refers to the pre-configured initial historical hidden state parameters used in the first round of prediction. For example, an all-zero vector can be used as the preset initial state, or a statistically obtained mean feature vector can be used. The historical hidden state refers to the state variables that accumulate historical interaction time-series information stored in the time-series model. The data to be encoded refers to the data content that needs to be sent to the encoder for encoding operations in this prediction cycle.

[0057] S140. If the current prediction time is not the first round of prediction, obtain the historical hidden state cached in the previous prediction as well as the encoded dialogue and image vectors. Compare the multimodal input data with the encoded dialogue and image vectors, and filter out the newly added dialogue segments and newly added face image frames as data to be encoded.

[0058] In this embodiment, the encoded dialogue and image vectors refer to the dialogue feature vectors and face image feature vectors that have been encoded and cached in the previous round of prediction. New dialogue fragments refer to newly generated dialogue content fragments in the current round of multimodal input data compared to the previous round of prediction. New face image frames refer to newly acquired face image frames in the current round of multimodal input data compared to the previous round of prediction.

[0059] S150. Perform encoding processing on the data to be encoded, combine the historical hidden state to perform incremental temporal state update operation to obtain the updated temporal context state, and input the temporal context state, dialogue modality, visual modality, dialogue reliability and visual reliability into the gating network to perform causal cross-modal fusion processing to obtain the multimodal fusion state.

[0060] In this embodiment, encoding refers to the computational process of converting the original dialogue and image data to be encoded into feature vectors. Incremental temporal state update operation refers to the process of encoding only newly added data, reusing cached historical hidden states for recursive computation, and obtaining the new temporal state. Temporal context state refers to the temporal feature state output after fusing historical and currently added temporal information.

[0061] Gated networks are network modules used to achieve causal cross-modal fusion, which can adjust the contribution weights of each modality based on modal reliability. Causal cross-modal fusion processing refers to the computational process of fusing multimodal information by combining modal features and modal reliability while adhering to temporal causality constraints. The multimodal fusion state refers to the comprehensive feature state output after the cross-modal fusion processing is completed.

[0062] Specifically, it determines whether the current prediction time is the first round of prediction. If it is the first round of prediction, the preset initial state is used as the historical hidden state, and the multimodal input data is used as the data to be encoded. If it is not the first round of prediction, the historical hidden state h of the previous prediction is read from the cache. t-1 In addition to the encoded dialogue turns and image fragment vectors, the multimodal input data is compared with the encoded vectors to select new dialogue fragments and new face image frames as data to be encoded.

[0063] Encoding processing is performed on the data to be encoded. Incremental temporal state updates are completed by combining the historical hidden state to obtain the updated temporal context state. Then, the temporal context state, dialogue modality, visual modality, dialogue reliability, and visual reliability are input into the gating network to perform causal cross-modal fusion processing. The contribution weights of dialogue and visual modalities are dynamically allocated through modal reliability to carry out fusion, and causal cross-modal fusion is completed to obtain a multimodal fusion state. This avoids repeatedly encoding the complete historical window in each cycle, thereby reducing the inference latency and memory access volume at the edge.

[0064] For example, dialogue reliability and visual reliability can be used to dynamically generate contribution weights for dialogue modalities and visual modalities through normalization operations. The generated contribution weights are then used to perform weighted operations on the dialogue modalities and visual modalities. The dialogue modal features and visual modal features after weighting are then fed into the computing unit along with the temporal context state carrying historical time-series accumulated information. Feature concatenation, feature addition, or gating update operations are then performed to further fuse the latest multimodal information from a single round with the accumulated historical time-series information, generating a complete multimodal fusion state.

[0065] S160. Input the multimodal fusion state into the future expression prediction head, perform multi-task prediction processing, and output the expression category probability, the expected occurrence time of the expression, and the expression intensity within the preset time period in the future.

[0066] In this embodiment, the future expression prediction head refers to a network functional module that takes multimodal fusion state as input and is used to complete multi-task expression prediction. Multi-task prediction processing refers to a multi-objective inference operation that simultaneously performs expression category classification, expression occurrence time regression, and expression intensity regression under a single input. The future preset time period refers to the target time interval after the current prediction time where expression prediction needs to be performed.

[0067] The probability of an expression category refers to the predicted probability distribution of each expression category within a preset future time period. The predicted occurrence time of an expression refers to the time parameter that the model predicts will occur within the preset future time period. The expression intensity refers to the magnitude of the expression corresponding to the predicted target expression.

[0068] Specifically, the multimodal fusion state f is input into the future expression prediction head to perform multi-task prediction processing, and synchronously infers and outputs the expression category probability p(y) and the expected appearance time τ of the expression within a preset time period in the future. pred And the intensity of facial expressions, for example, , , Among them, W p The weight matrix for predicting expression categories is a trainable network parameter for the future expression prediction head; the softmax activation function converts the original scores into a probability distribution with a sum of 1; p(y) is the predicted probability distribution of each expression category in the future preset interval, with each item corresponding to the predicted probability of a certain type of expression, used to determine which expression is most likely to occur. The time prediction weight vector (transposed) is a trainable network parameter for the future expression prediction head; softplus ensures that the output is always greater than or equal to 0, which fits the physical meaning of "time cannot be negative". The intensity prediction weight vector (transposed) is a parameter of the trainable network of the future expression prediction head; the sigmoid is used to compress the output to the 0-1 range.

[0069] The model employs a multi-task joint training mechanism. The classification loss is used to measure the deviation between the predicted expression category and the actual labeled expression category. The occurrence time regression loss is used to measure the deviation between the predicted expression occurrence time and the actual labeled time. The intensity regression loss is used to measure the deviation between the predicted expression intensity and the actual labeled expression intensity. The three types of losses are assigned corresponding weights and then summed to obtain the comprehensive loss. The network parameters are updated by backpropagation based on the comprehensive loss, allowing a single model to simultaneously optimize the three tasks of expression category recognition, expression occurrence time prediction, and expression intensity prediction, rather than training multiple independent sub-models separately.

[0070] For example, cross-entropy can be selected as the classification loss to measure the prediction bias of the expression category, regression loss function can be selected as the occurrence time regression loss to measure the prediction bias of the expression occurrence time, and regression loss function can be selected as the intensity regression loss to measure the prediction bias of the expression amplitude. After configuring the corresponding loss weight coefficients for the three types of losses, the overall comprehensive loss is obtained by summing them. Backpropagation is performed based on this comprehensive loss to iteratively update the network weight parameters, so as to realize the synchronous joint learning of the three tasks of expression category, expression occurrence time and expression intensity.

[0071] The above mapping relationship is used to illustrate the input-output relationship of this step, and does not limit the use of a specific neural network or training method.

[0072] S170. Perform probability calibration on the expression category probability. Based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time, perform confidence gating judgment. When the maximum expression category probability, the difference between the maximum and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, generate an early trigger flag. By using the early trigger flag, the robot is triggered to output an expression that matches the maximum expression category probability, the expected expression occurrence time, and the expression intensity.

[0073] In this embodiment, probability calibration refers to the computational process of correcting the original expression category probabilities output by the model to reduce the deviation between the model's predicted probabilities and the actual event occurrence probabilities. The calibrated expression category probabilities refer to the probability distribution results of each expression category obtained after probability calibration. Confidence gating judgment processing refers to a conditional verification process that combines multiple predictions and input quality indicators to comprehensively determine whether the robot is allowed to execute premature expression triggering.

[0074] The maximum expression category probability is the probability value of the expression category with the highest value in the calibrated expression category probability distribution. The difference between the maximum and second-highest expression category probabilities is the difference between the probabilities of the highest-probability expression category and the second-highest-probability expression category in the probability distribution, used to distinguish whether there is a significantly dominant expression category in the prediction results. Preset conditions are the judgment thresholds pre-set for each judgment indicator. The executable interval refers to the effective time range of the expected appearance time of the expression that is adapted to the latency of the entire robot system and allows for early response. The allow early trigger flag is a status flag generated when all conditions are met, used to notify the robot that it can execute the pre-response expression output.

[0075] Specifically, probabilistic calibration of expression category probabilities can be performed using a temperature-scaled probability calibration method. This method uses a temperature parameter to rescale and correct the original expression category probability distribution output by the network. For example, the original logits scores of the expression categories output by the network can be extracted, and the original logits scores can be scaled using a temperature parameter pre-tuned on an independent validation dataset. The scaled scores are then fed into the softmax function to recalculate the probability distribution, resulting in the temperature-scaled expression category probabilities. This entire process does not modify the already trained network weights; post-processing corrections are only performed during the inference stage, preserving the probability of each expression category while adjusting the confidence level of each category. Here, logits are the original scores of each category output by the network before softmax; the temperature parameter T is a learnable scalar hyperparameter optimized on an independent validation set, which does not change the model's network weights and is only used for post-processing during the inference stage.

[0076] Other probability calibration methods, such as ordinal-preserving regression, can also be used to correct the deviation between the original output probability and the actual probability of the event, thus obtaining the calibrated probability distribution of the expression category.

[0077] The confidence gating judgment process is carried out using the calibrated expression category probability, modal comprehensive reliability, and expression expected occurrence time. When the maximum expression category probability, the difference between the maximum and second largest expression category probabilities, and the modal comprehensive reliability all meet their respective preset conditions, such as the maximum expression category probability, the difference between the maximum and second largest expression category probabilities, and the modal comprehensive reliability are all greater than or equal to the corresponding judgment threshold, and the expression expected occurrence time falls within the executable range, an early triggering flag is generated.

[0078] By using this early trigger flag, the robot can output an expression that matches the probability of the maximum expression category, the expected time of expression occurrence, and the intensity of the expression. For example, the robot's expression start time can be calculated by combining robot communication latency, control latency, and actuator latency. During the expression execution phase, the robot continuously collects the user's actual facial expressions and compares the deviation between the prediction and the actual expression. Based on the deviation, the robot can choose to adjust the expression amplitude or revert to a neutral state and re-predict. Alternatively, alternative processing logics such as adding mid-process secondary confidence checks and using deviations to update modal reliability can be added to achieve early response with latency compensation.

[0079] The technical solution of this invention obtains multimodal input data by aligning historical dialogue window data and historical face image window data, and calculates visual reliability and dialogue reliability based on face image quality indicators, speech recognition and sentence completeness indicators, respectively, which can quantitatively characterize the data quality of the two types of modal inputs; by using the first-round or non-first-round prediction differentiation processing logic, the non-first round only selects newly added dialogue segments and newly added face image frames as data to be encoded and performs incremental temporal state updates by combining the cached historical hidden states, without having to repeatedly encode all historical window data, reducing the encoding computation overhead and memory access volume at the edge; and by using a gating network to complete causal cross-modal fusion, combining the various modalities The system dynamically adjusts modal contributions based on its own reliability, reducing the interference of low-quality modalities on the fusion results. The future expression prediction head performs multi-task prediction, outputting expression category probabilities, expected expression occurrence times, and expression intensities to support early responses with latency adaptation. After probability calibration, confidence gating is performed based on probability conditions, overall modal reliability, and expected expression occurrence times. Only when multiple conditions are met simultaneously is an early trigger flag generated to drive the robot to output matching expressions. Under the limited computing power at the edge, this system balances data quality constraints of multimodal inputs with temporal inference efficiency, reduces invalid triggers, and improves the reliability and rationality of the robot's early response to users' future expressions, avoiding delayed or misaligned expression responses.

[0080] Example 2

[0081] Figure 2 This is a flowchart of another user expression prediction and robot proactive response method provided in Embodiment 2 of the present invention. This embodiment is a refinement of the above embodiment's step of "inputting the temporal context state, dialogue modality, visual modality, dialogue reliability, and visual reliability into a gating network to perform causal cross-modal fusion processing to obtain a multimodal fusion state." Figure 2 As shown, the method includes:

[0082] S210. Collect historical dialogue window data and historical face image window data up to the current prediction time, and align the data to obtain multimodal input data.

[0083] S220. Perform dialogue modality extraction and visual modality extraction operations on the multimodal input data to obtain the dialogue modality and visual modality. Calculate the visual reliability based on the multimodal input data and face image quality indicators. Calculate the dialogue reliability based on the multimodal input data and speech recognition and sentence completeness indicators. Calculate the overall modality reliability based on the dialogue reliability and visual reliability.

[0084] S230. If the current prediction time is the first round of prediction, then the preset initial state will be used as the historical hidden state, and the multimodal input data will be used as the data to be encoded.

[0085] S240. If the current prediction time is not the first round of prediction, obtain the historical hidden state cached in the previous prediction as well as the encoded dialogue and image vectors. Compare the multimodal input data with the encoded dialogue and image vectors, and filter out the newly added dialogue segments and newly added face image frames as data to be encoded.

[0086] S250. Perform encoding processing on the data to be encoded, combine the historical hidden state to perform incremental temporal state update operation to obtain the updated temporal context state, and determine the target modality with reduced or missing reliability based on the dialogue reliability and visual reliability input to the gating network, and reduce the contribution weight corresponding to the target modality.

[0087] In this embodiment, the target modality refers to a type of modality among dialogue modalities and visual modalities that has a decreased reliability value or missing modal data. The contribution weight is a weight parameter assigned to each modality to control the magnitude of the influence of the modal feature on the final fusion result.

[0088] S260. Perform a weighted operation based on the temporal context state, dialogue modality, visual modality, dialogue reliability, visual reliability, and contribution weight to obtain the multimodal fusion state.

[0089] The gating network is equipped with an attention mask, which is applied to the temporal feature sequence inside the gating network to mask the internal time step positions that represent the time after the current time in the temporal feature sequence. This constrains the model to only call the current and historical time step information in the temporal feature sequence for calculation.

[0090] In this embodiment, weighted operation refers to the weighted modulation of modal features using contribution weights, and then combining this with the temporal context state to complete the aggregation operation. Attention mask is a constraint label applied to the temporal feature sequence to mask future time steps and achieve causal constraints.

[0091] Specifically, the temporal context state hc, dialogue modality hd, visual modality hv, dialogue reliability qd, and visual reliability qv are input into a gating network to perform causal cross-modal fusion processing. The target modality with reduced or missing reliability in dialogue reliability qd and visual reliability qv is identified, and the contribution weight of the target modality is downgraded to obtain the dialogue contribution weight ad and the visual contribution weight av. For example, [ad,av] = softmax(W×[hd,hv,qd,qv]), which means concatenating the dialogue modality features hd, the visual modality features hv, the dialogue reliability qd, and the visual reliability qv to obtain the concatenated vector [hd,hv,qd,qv]. The pre-trained weight matrix W within the gating network is used to perform a linear transformation on the concatenated vector to obtain the original weight scores corresponding to the two modalities. These original weight scores are then fed into the softmax function for normalization. When a decrease in dialogue or visual reliability, or a modality loss, is detected, the original score of the corresponding modality decreases accordingly. After normalization, the adjusted dialogue contribution weights and visual contribution weights are obtained. For instance, when qd decreases (dialogue quality deteriorates), the original score of the dialogue path is suppressed after the W operation. After softmax processing, the final output dialogue contribution weight ad automatically decreases; the visual weight av relatively increases. The magnitude of the weight decrease is given by the mapping relationship learned by the weight matrix W after training convergence.

[0092] A weighted operation is performed based on the temporal context state hc, dialogue modality hd, visual modality hv, dialogue reliability qd, visual reliability qv, and adjusted dialogue contribution weight ad and visual contribution weight av to obtain the multimodal fusion state. For example, f = ad × qd × hd + av × qv × hv + hc, where ad + av = 1.

[0093] Meanwhile, by using the attention mask configured by the gating network to mask the internal time step positions after the current moment in the temporal feature sequence, the model is restricted to using information from the current and historical time steps for computation.

[0094] S270. Input the multimodal fusion state into the future expression prediction head, perform multi-task prediction processing, and output the expression category probability, the expected occurrence time of the expression, and the expression intensity within the preset time period in the future.

[0095] S280. Perform probability calibration on the expression category probability. Based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time, perform confidence gating judgment processing. When the maximum expression category probability, the difference between the maximum and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, generate an early trigger flag. By using the early trigger flag, the robot is triggered to output an expression that matches the maximum expression category probability, the expected expression occurrence time, and the expression intensity.

[0096] Optionally, based on the above embodiments, the probability of the expression category is calibrated. Confidence gating is performed based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time. When the maximum expression category probability, the difference between the maximum and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, a flag allowing early triggering is generated, including:

[0097] A temperature scaling method is used to perform probability calibration on the expression category probabilities to obtain the calibrated expression category probabilities.

[0098] Extract the maximum and second-largest expression category probabilities from the calibrated expression category probabilities, and calculate the probability difference between the maximum and second-largest expression category probabilities.

[0099] If all the following conditions are met: the probability of the largest expression category is greater than or equal to the preset probability threshold, the probability difference is greater than or equal to the preset interval threshold, the overall modality reliability is greater than or equal to the reliability threshold, and the expected appearance time of the expression falls within the executable interval, a flag allowing early triggering is generated.

[0100] In this embodiment, the temperature scaling method is a type of probability calibration technique that scales the original network scores based on temperature parameters and then recalculates the probability distribution. The second-highest expression category probability is the probability corresponding to the expression category with the second-highest value in the calibrated probability distribution.

[0101] Specifically, after performing probability calibration on the expression category probabilities, temperature scaling is used to complete the probability calibration and obtain the calibrated expression category probabilities. The maximum expression category probability and the second largest expression category probability are extracted from the calibrated probability distribution, and the probability difference between the two is calculated. When the maximum expression category probability is greater than or equal to a preset probability threshold, the probability difference is greater than or equal to a preset interval threshold, the modal comprehensive reliability is greater than or equal to a reliability threshold, and the expected appearance time of the expression is within the executable range, an early triggering flag is generated. Calibration can be completed by adjusting the temperature parameters in the verification set through temperature scaling, or other equivalent probability calibration methods can be used to add additional verification conditions as needed to complete the confidence screening.

[0102] By first calibrating the probability of expression categories using temperature scaling, the bias of overconfidence in the model's original prediction probabilities can be corrected, improving the reliability of the confidence level. Based on this, a confidence level gating check is performed by comprehensively considering multiple indicators such as the probability of the largest expression category, the difference between the probability of the second largest expression and the probability of the largest expression, the overall modality reliability, and the predicted time of expression occurrence. Only when multiple conditions are met simultaneously will a flag allowing early triggering be generated. This can filter out prediction results with insufficient confidence, poor modal input quality, low discrimination of the optimal expression category, or unsuitable prediction time for early response, reducing the risk of false triggering caused by the robot outputting expressions prematurely without basis, and improving the accuracy and robustness of predictive response in the human-computer expression interaction process.

[0103] The technical solution of this embodiment performs causal cross-modal fusion by inputting the temporal context state, dialogue modality, visual modality, and corresponding dialogue reliability and visual reliability into a gated network configured with attention masks. The attention mask acts on the temporal feature sequence inside the gated network, masking the internal time step positions representing the current moment, and constraining the model to only call current and historical time step information to participate in the calculation, avoiding the model from using virtual future information inside the temporal sequence to cause inference leakage. At the same time, based on the dialogue reliability and visual reliability, the target modality with reduced or missing reliability is identified and its contribution weight is reduced. Then, the temporal context state, the two types of modalities and their corresponding reliability, and the adjusted contribution weight are combined to complete the weighted operation to obtain the multimodal fusion state. In this way, the influence of degraded modalities can be dynamically suppressed according to the actual data quality of each modality, and the fusion process is guaranteed to strictly follow causal constraints, using only the interaction information that has occurred, thereby improving the accuracy and causal rationality of the multimodal fusion output results.

[0104] Example 3

[0105] Figure 3 This is a flowchart of another user expression prediction and robot proactive response method provided in Embodiment 3 of the present invention. This embodiment is a refinement of the user expression prediction and robot proactive response method in the above embodiments. Figure 3 As shown, the method includes:

[0106] S310. Collect historical dialogue window data and historical face image window data up to the current prediction time, and align the data to obtain multimodal input data;

[0107] S320. Perform dialogue modality extraction and visual modality extraction operations on the multimodal input data to obtain the dialogue modality and visual modality. Calculate the visual reliability based on the multimodal input data and face image quality indicators. Calculate the dialogue reliability based on the multimodal input data and speech recognition and sentence completeness indicators. Calculate the overall modality reliability based on the dialogue reliability and visual reliability.

[0108] S330. If the current prediction time is the first round of prediction, then the preset initial state is used as the historical hidden state, and the multimodal input data is used as the data to be encoded.

[0109] S340. If the current prediction time is not the first round of prediction, then obtain the historical hidden state cached in the previous prediction as well as the encoded dialogue and image vectors. Compare the multimodal input data with the encoded dialogue and image vectors, and filter out the newly added dialogue segments and newly added face image frames as data to be encoded.

[0110] S350. Perform encoding processing on the data to be encoded, combine the historical hidden state to perform incremental temporal state update operation to obtain the updated temporal context state, and input the temporal context state, dialogue modality, visual modality, dialogue reliability and visual reliability into the gating network to perform causal cross-modal fusion processing to obtain the multimodal fusion state.

[0111] S360: Input the multimodal fusion state into the future expression prediction head, perform multi-task prediction processing, and output the expression category probability, the expected occurrence time of the expression, and the expression intensity within the preset time period in the future.

[0112] S370. Perform probability calibration on the expression category probability. Based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time, perform confidence gating judgment processing. When the maximum expression category probability, the difference between the maximum and second largest expression category probabilities, or the modal comprehensive reliability does not meet the corresponding preset conditions, or the expected expression occurrence time is not within the executable range, generate a rejection early trigger flag to maintain the robot's neutral expression through the rejection early trigger flag.

[0113] In this embodiment, the "reject premature triggering" flag is a status marker generated when any triggering condition is not met, used to instruct the robot to abandon the premature expression output. A neutral expression is the baseline expression state maintained by the robot when it has no additional emotional expression.

[0114] Specifically, if at least one of the following conditions is not met: the probability of the largest expression category, the difference between the probabilities of the largest and second-largest expression categories, or the modal comprehensive reliability, or if the expected occurrence time of the expression is outside the executable range, a rejection flag for premature triggering is generated. This flag is used to control the robot to maintain a neutral expression and not to execute an early expression response. For example, the robot can reduce the amplitude of its current expression movement and slowly transition to a neutral transition state.

[0115] S380. When the probability of the largest expression category, the difference between the probabilities of the largest and second largest expression categories, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected appearance time of the expression is within the executable range, generate an early trigger flag. By using the early trigger flag, the robot is triggered to output an expression that matches the probability of the largest expression category, the expected appearance time of the expression, and the expression intensity.

[0116] Optionally, based on the above embodiments, triggering the robot to output an expression that matches the probability of the maximum expression category, the expected occurrence time of the expression, and the expression intensity includes:

[0117] Obtain the robot's communication latency, control latency, and actuator latency, and sum them to obtain the robot's total latency;

[0118] The robot's expression activation time is calculated based on the expected appearance time of the expression and the robot's total latency.

[0119] If the robot's expression activation time is earlier than or equal to the current time, issue an expression-driven command to the robot;

[0120] If the robot's expression activation time is later than the current time, create a timed task, and after running to the robot's expression activation time, issue an expression-driven command to the robot;

[0121] During the process of the robot responding to facial expression-driven commands and executing facial expression output, the robot continuously collects the user's actual facial expressions and extracts the actual expression category, actual occurrence time, and actual expression intensity as actual expression information.

[0122] The actual facial expression information is compared with the facial expression category probability, the expected occurrence time of the expression, and the expression intensity output by the multi-task prediction processing to obtain the facial expression bias;

[0123] When the expression deviation exceeds the preset deviation threshold, the robot's current expression movement amplitude is reduced, or the robot is controlled to revert to a neutral transition state. The current moment is then used as the current prediction moment, and the operation of collecting historical dialogue window data and historical face image window data up to the current prediction moment is performed.

[0124] In this embodiment, communication latency is the time overhead incurred by transmitting instructions between robot control systems. Control latency is the processing time consumed by the controller in parsing and scheduling facial expression instructions. Actuator latency is the inherent motion time of the robot's facial actuator from startup to fully performing an expression. The total robot latency is the time overhead of the entire facial expression driving link obtained by adding communication latency, control latency, and actuator latency.

[0125] The robot's expression activation time is calculated to compensate for system latency, determining the exact moment the robot needs to begin displaying expressions. Expression-driven commands are control instructions sent to the robot's execution unit to drive the robot to output the corresponding expressions. Timed tasks are delayed-triggered tasks set in the background, automatically sending commands upon reaching the expression activation time.

[0126] Actual facial expression information refers to the actual facial expression category, occurrence time, and intensity extracted by the robot during the expression execution process from the user's facial image. Expression deviation is the difference between the user's actual facial expression information and the model's predicted output. The preset deviation threshold is a pre-set threshold for judging whether the predicted result deviates too much from the reality. The neutral transition state is the robot's baseline expressionless state, serving as a transitional state before the expression ends and re-prediction.

[0127] Specifically, the robot's communication latency, control latency, and actuator latency are obtained, summed to obtain the total robot latency, and then the robot's expression initiation time is calculated by combining the expected appearance time of the expression with the total robot latency. For example, t start =t+τ pred -L 总 Where t is the current prediction time, which is the system time at which the system began executing the facial expression prediction calculation in this round. τ pred The predicted occurrence time of the expression (prediction offset) is calculated from the current time t, after τ. pred The user's emoji will appear after a certain time; t+τ pred The absolute moment when the user's expression will occur. L 总 t represents the total latency of the robot's facial expression driving chain, signifying the total time required for the robot to fully display the target facial expression from receiving the instruction. start For the robot to activate its facial expressions, the robot needs to prepare in advance. 总 Only by issuing instructions at specific intervals can the robot's facial expressions be aligned with the user's in terms of timing.

[0128] When the expression is activated earlier than or equal to the current time, i.e., t start ≤t, issue an expression-driven command to the robot; if the expression activation time is later than the current time, t start If >t, a scheduled task is created, which waits until the emoji activation time is reached before issuing the command.

[0129] During the process of the robot performing facial expression output, it continuously collects the user's actual facial expressions, extracts the actual expression category, actual occurrence time, and actual expression intensity as actual expression information, and compares the actual expression information with the expression category probability, expected occurrence time, and expression intensity output by multi-task prediction to obtain the expression deviation.

[0130] When the expression deviation exceeds the preset deviation threshold, the robot's current expression movement amplitude is reduced or the robot is controlled to return to a neutral transition state. The current moment is used as the new prediction starting point, and the steps of collecting historical dialogue window data and historical face image window data are returned to perform expression prediction again. The robot's output expression can be a responsive, soothing or complementary expression rather than a simple mirror image.

[0131] By comprehensively analyzing and summing the latency of various components in the robot's communication, control, and actuator links, the total latency is obtained. Based on this, the robot's expression initiation time is deduced. The inherent system latency is compensated by immediately issuing instructions or scheduling timed tasks, ensuring that the robot's expression output time aligns with the user's expected expression time, thus improving the time synchronization of human-computer expression interaction. Simultaneously, during the robot's expression execution, the user's facial expressions are continuously collected and compared with the prediction results to obtain expression deviations. If the deviation exceeds a preset deviation threshold, the expression amplitude is reduced or the system returns to a neutral state. The expression prediction process is restarted using the latest interaction data. This allows for timely correction of deviations caused by inaccurate predictions, preventing the robot from continuously outputting expressions that do not match the user's true emotions. This effectively improves the dynamic error correction capability of expression prediction response and the smoothness and naturalness of human-computer interaction.

[0132] Furthermore, based on the above embodiments, the method further includes:

[0133] Obtain the original data of human-computer interaction sessions authorized by the user, and unify the time clock of dialogue and face images within the data;

[0134] For each preset sampling time, the dialogue window and face image window before the sampling time are extracted as model input samples, and the user's real expression in the future prediction window at the sampling time is identified. The expression category, the time when the expression first reaches the intensity threshold, and the maximum expression intensity are labeled on the model input samples. If no expression change is identified in the future prediction window at the sampling time, the neutral category is labeled on the model input samples.

[0135] The dataset is divided into training, validation, and test sets according to user categories. Notably, a user's session will not be simultaneously distributed within both the training and test sets.

[0136] Based on the number of samples for each expression category in the training set, a sample balancing process is performed on the samples of different expression categories in the training set to obtain a training sample set for training the future expression prediction head.

[0137] In this embodiment, the raw data of the human-computer interaction conversation consists of dialogue text data generated during the human-computer interaction process and collected facial image data. The sampling time is a reference time point used to capture the model input samples. The dialogue window and facial image window are data segments composed of dialogue fragments and facial image frames collected within a certain time period before the sampling time. The future prediction window is a specified time interval after the sampling time, used to find the user's subsequent true facial expressions.

[0138] The moment when an expression first reaches its intensity threshold is the time point within the future prediction window when the user's expression intensity first reaches the preset intensity threshold. The maximum expression intensity is the highest amplitude value exhibited by the user's expression within the future prediction window. Sample balancing is an operation that reduces the negative impact of sample size differences on model training performance through methods such as class weights, hierarchical sampling, or focus loss. The training sample set is the final set of samples that, after sample balancing, can be fed into the future expression prediction head for training.

[0139] Specifically, the system acquires the original data of the human-computer interaction session authorized by the user, performs unified time clock processing on the dialogue and face images within the session, assigns a unified reference timestamp to the dialogue data and face image frames, and ensures that the time axes of the two types of data are aligned with each other.

[0140] For each preset sampling time, the dialogue window and face image window before the sampling time are extracted as the model input samples. It is forbidden to include dialogue and face images from future time moments in the input samples to avoid leakage of training data.

[0141] Identify the user's actual facial expressions within the future prediction window [t+Δt1, t+Δt2] corresponding to sampling time t, and label the input samples to the model with whether an expression appeared, the expression category, the time when the expression first reached the intensity threshold, and the maximum expression intensity. If no expression change is detected within the future prediction window, label it as no change or neutral.

[0142] The dataset is divided into training, validation, and test sets based on user identities, ensuring that sessions of the same user are not simultaneously distributed across both the training and test sets. This process is used to test the model's generalization ability to new users. For example, the user identity identifier corresponding to each model input sample is extracted first, and all session samples belonging to the same user are grouped into the same user sample group. Using the user sample group as the smallest allocation unit, each user sample group is assigned to the training, validation, or test sets respectively. Splitting session samples of the same user is prohibited, ensuring that all sessions of the same user appear only in one dataset. This ensures that the test set consists entirely of user samples not seen during the training phase, thus completing the dataset partitioning and testing the model's generalization ability to new users.

[0143] Based on the sample quantity of each expression category within the training set, sample balancing is performed on the samples of different expression categories in the training set to obtain a training sample set that can be used for future expression prediction head training. Sample balancing can use class weights, stratified sampling, or focus loss, or other balancing schemes such as oversampling and undersampling. For example, the number of samples corresponding to each expression category in the training sample set is counted to identify the minority class with fewer samples and the majority class with more samples, and any balancing scheme among class weights, stratified sampling, or focus loss is selected for processing; when using class weights, differentiated loss weights are assigned to different expression categories to increase the influence of minority class samples in the loss calculation process; when using stratified sampling, samples are extracted stratified by expression category in each training iteration to ensure a balanced distribution of samples of each category in each batch; when using focus loss, the loss contribution weight of high-confidence samples is reduced to amplify the impact of hard-to-classify samples on parameter updates and alleviate the training bias caused by class sample imbalance.

[0144] By obtaining user authorization before collecting raw data from human-computer interaction conversations and unifying the time clock of dialogues and facial images, the temporal alignment of multimodal samples can be ensured, avoiding temporal misalignment issues. Sampling only from historical window data and using future user expressions as labels prevents future data leakage during training and ensures consistency between training samples and online inference data input logic. Adopting a dataset partitioning method based on user identity avoids the same user's conversation being distributed across training and test sets, allowing for a more realistic test of the model's generalization ability to unfamiliar new users. Furthermore, sample balancing processing is performed to address the differences in the number of expression category samples in the training set, mitigating training bias caused by class imbalance and improving the recognition and prediction accuracy of future expression prediction heads for various expressions, especially those with low sample sizes. Ultimately, this results in a more adaptable and reliable prediction model.

[0145] Furthermore, based on the above embodiments, the method further includes:

[0146] After performing face cropping on the face samples in the training sample set, data augmentation processing is performed on the face samples to adjust brightness, scale, and slight pose.

[0147] Based on the training sample set, a comprehensive training loss function is constructed, which includes future expression category loss, occurrence time regression loss, intensity regression loss, and probability calibration or consistency constraint loss. The loss weights corresponding to each type of loss are determined on the validation set. The prediction model is trained using the comprehensive training loss function, and the trained prediction model is used as the future expression prediction head.

[0148] In this embodiment, face cropping refers to cropping an image containing only the face from the original image containing the user. Data augmentation processing refers to applying operations such as brightness, scale, and slight pose transformations to the face image without changing the image's semantics and expression, thereby expanding the diversity of training samples. Future expression category loss is a loss term used to measure the deviation between the predicted expression category and the actual expression category. Occurrence time regression loss is a loss term used to measure the deviation between the predicted expression occurrence time and the actual annotation time.

[0149] Intensity regression loss measures the deviation between predicted and actual facial expression intensity. Probability calibration or consistency constraint loss constrains the reliability of the model's output probability and improves the temporal consistency of prediction results. The comprehensive training loss function is the sum of the sums of multiple losses multiplied by their respective weights. Loss weights are the weight coefficients assigned to each loss term, which are optimally determined on the validation set.

[0150] Specifically, after performing face cropping on the face samples in the training sample set, brightness shift adjustment, image scale transformation, and slight face head pose shift transformation can be performed on the cropped face image samples respectively. During the transformation operation, only the visual features of the face image are modified, without making any changes to the real expression labels such as expression category, occurrence time, and expression intensity that have been labeled for the face sample.

[0151] Based on the training sample set, a comprehensive training loss function L=λ1×L is constructed, which includes future expression category loss, occurrence time regression loss, intensity regression loss, and probability calibration or consistency constraint loss. cls +λ2×L time +λ3×L intensity +λ4×L calThe loss weights for each loss term are determined on the validation set. Here, L is the overall training loss function, the total loss value that needs to be minimized during model training. λ1 is the loss weight (hyperparameter) corresponding to the expression category loss, used to adjust the proportion of the expression classification task in the total loss. cls This is the future expression category loss, a classification loss term used to measure the deviation between the expression category predicted by the model and the actual expression category of the sample.

[0152] λ² is the loss weight (hyperparameter) corresponding to the expression occurrence time regression loss, used to adjust the proportion of loss in the expression occurrence time regression task. L time It is the time regression loss, which measures the deviation between the predicted time of the expression and the actual time of its occurrence.

[0153] λ3 is the loss weight (hyperparameter) corresponding to the expression intensity regression loss, used to adjust the proportion of loss in the expression intensity regression task. intensity It is the intensity regression loss, a regression loss term used to measure the deviation between the expression intensity predicted by the model and the actual expression intensity labeled in the samples.

[0154] λ4 is the loss weight (hyperparameter) corresponding to the probabilistic calibration or consistency constraint loss, used to adjust the proportion of the calibration constraint term. cal It is a probability calibration or consistency constraint loss, a constraint term loss, used to guide the model to output more reliable probabilities, or to ensure that the time series prediction results are consistent.

[0155] For example, the loss function adapted to the expression classification task is selected as the future expression category loss, the regression loss function is selected as the occurrence time regression loss and intensity regression loss, and the corresponding constraint loss term is selected as the probability calibration or consistency constraint loss. The four loss terms are multiplied by their corresponding loss weights and then added together to form a comprehensive training loss function. Then, multiple sets of different weight combinations are traversed on the validation set to carry out validation experiments. The set of weights with the best overall model performance is selected as the final loss weights corresponding to each loss.

[0156] The prediction model is trained using this comprehensive training loss function, and the trained model is used as the future expression prediction head. For example, the training sample set can be input into the prediction model to obtain prediction results for expression category, expected occurrence time of expression, and expression intensity. The prediction results and the true labels of the samples are substituted into the comprehensive training loss function to calculate the comprehensive loss value. Based on the loss value, backpropagation is used to iteratively update the internal parameters of the prediction model until the model reaches the preset training termination condition. After training is completed, the prediction model is used as the future expression prediction head to output the user's future expression prediction results.

[0157] Furthermore, during online operation, the basic network parameters of the prediction model (future expression prediction head) can be frozen, and frequent updates are not performed on them. During human-computer interaction, based on the expression deviation obtained by comparing the user's actual expression with the expression prediction results, the user-level calibration parameters corresponding to the current interactive user are updated online first. The user-level calibration parameters include thresholds for each expression category, average reaction time, and output temperature parameters. After collecting a sufficient number of feedback samples authorized by the user, the offline training process is started, and the basic network parameters of the prediction model are updated iteratively using the feedback samples. This avoids the risk of online model drift and error accumulation caused by directly updating the basic network parameters online. The user-level calibration parameters are hyperparameters in the inference operation stage, not network weights in the prediction model training stage. After continuously collecting facial expression deviation data through interaction, a set of calibration parameters corresponding to the user is updated online iteratively based on the expression prediction deviation.

[0158] Among them, the threshold for each expression category is a hyperparameter in the inference stage. It is a set of judgment thresholds used in the confidence gating process, such as preset probability threshold, preset interval threshold, and reliability threshold. Each expression category can have its own independent threshold, rather than all expressions sharing a fixed set of thresholds.

[0159] Different users have different habits of expressing emotions; some users have subtle expressions, so the threshold for that expression category needs to be lowered; some users have exaggerated expressions, so the threshold can be raised to reduce false triggers.

[0160] Average reaction time is the average time taken for a specific user from the occurrence of a dialogue stimulus to the appearance of the corresponding facial expression, calculated historically. The predicted facial expression occurrence time τ is used to correct the model's output. pred The model predicts the time of an expression and then adjusts it based on the user's historical average reaction time, making the predicted time more consistent with the user's personal habits and reducing prediction errors.

[0161] The output temperature parameter is a temperature hyperparameter used during temperature scaling calibration. It scales the original logits of the expression categories and calibrates the confidence level of the model's output probabilities. Higher temperatures result in a flatter probability distribution, while lower temperatures lead to a sharper probability distribution. Each user can configure a unique temperature parameter for personalized calibration of their prediction confidence.

[0162] By performing data augmentation processing on facial images after face cropping, including brightness, scale, and slight pose adjustments, the diversity of facial samples can be expanded. This reduces the risk of overfitting, where the model excessively memorizes user facial appearance features while ignoring expression features, and improves the model's adaptability under different imaging conditions. Employing a multi-objective comprehensive training loss function that covers expression category, occurrence time, expression intensity, and probability calibration or consistency constraints, and using a validation set to optimize the weighting of each loss, can balance the training priorities among classification tasks, two regression tasks, and probability constraint objectives. This allows the model to simultaneously learn the predictive capabilities of multiple dimensions, including expression category, occurrence time, and expression intensity, and guides the model to output more reliable prediction results. Ultimately, the trained future expression prediction head can effectively improve the overall accuracy and temporal consistency of multi-task prediction of user future expressions.

[0163] In a specific example, the input of the user's future facial expression prediction and robot proactive response system architecture can be divided into two data streams: dialogue input and image input, as well as a quality assessment data stream. The dialogue input receives text or speech-to-text content with turn numbers and timestamps, and outputs it to the dialogue feature encoding module to extract semantic, emotional cues, and dialogue features corresponding to dialogue behaviors. The image input receives a sequence of face frames with acquisition timestamps, and sends it to the visual feature encoding module to extract visual features corresponding to facial expression action units, key points, and motion changes. The quality assessment module conducts multimodal data quality assessment based on speech recognition confidence, face occlusion, and image blurring.

[0164] The outputs of the dialogue feature encoding module, visual feature encoding module, and quality assessment module are jointly fed into the incremental temporal state update module. This module only encodes newly added dialogues and new image frames, while retaining historical hidden states. The incremental temporal state update module outputs the temporal context state to the causal cross-modal fusion module, which performs reliability gating and prohibits the use of information from future moments. The causal cross-modal fusion module outputs the fused features and sends them to the confidence gating module, which makes judgments based on thresholds and probability intervals, and does not trigger robot preemptive responses in low-confidence scenarios. The output of the confidence gating module is transmitted to the future expression prediction head, which outputs the prediction results of expression category probability, expression start time, and expression intensity. The prediction results are sent to the advance response planning module, which compensates for communication latency and actuator latency, generates a scheduling scheme, and sends it to the robot expression driving and rolling correction module. The robot expression driving and rolling correction module executes the robot expression output, while simultaneously collecting the user's actual expression and sending the actual expression feedback signal back to the incremental temporal state update module, forming a closed-loop feedback process.

[0165] Understandably, after performing probability calibration and confidence gating on the output user expression prediction results, an advance response plan that can be executed by the robot is only generated when the maximum predicted probability of the expression, the probability difference between the largest and second largest candidates, the visual reliability corresponding to the image, and the dialogue reliability corresponding to the dialogue all meet their respective preset judgment conditions. The robot's expression initiation time is calculated by subtracting the robot's communication delay, control delay, and facial actuator delay from the predicted user expression occurrence time to obtain the total delay. If any judgment condition is not met, the robot is controlled to maintain a neutral expression state and continue to wait to receive the next batch of multimodal input data to carry out a new round of prediction, thereby avoiding the problem of the robot making an erroneous pre-response.

[0166] The system's inputs, outputs, and data definitions can be as follows: Dialogue input consists of several rounds of dialogue between the user and the robot up to the current moment, which may include speech-to-text, speaker identification, start or end time of each round, pause duration, question or answer type, and speech recognition confidence level, etc.; Image input consists of a sequence of face image frames captured by the camera, including capture timestamp, face detection confidence level, key points, head pose, sharpness, and occlusion ratio, etc.; The historical observation window represents historical information from tW to t (W represents the observation window duration, and t is the current moment). For example, the image window can be 1.0 to 3.0 seconds, and the dialogue window can be the most recent 1 to 3 valid rounds, with the specific range determined by actual business needs. The future prediction window represents the time period from t+Δt1 to t+Δt2 (Δt1 and Δt2 represent the prediction start offset and prediction end offset, respectively), for example, predicting whether the target expression will appear within 0.3-1.5 seconds. The prediction categories include at least no significant change or neutral, happy, sad, angry, surprised, fearful, and disgusted, and categories can also be combined according to the actual expression requirements. The prediction output includes the probability of the future expression category p(y) (p(y) represents the prediction probability corresponding to each expression category) and the time τ from the occurrence. pred (τ) pred The parameters include the expected time of the expression, the intensity spred (spred represents the predicted intensity of the expression), the overall confidence level C (C represents the overall modal confidence level), and the flag G indicating whether early triggering is allowed (G represents the trigger flag of the confidence-gated output).

[0167] In a specific example, the expression prediction and early response control process can be as follows: Collect data from the dialogue window and face image window up to the current moment, and synchronize the data according to timestamps; extract dialogue semantics, emotional cues, visual expressions, and motion features, while calculating modal reliability; call the cached historical hidden states and perform incremental temporal updates for new data; obtain the multimodal fusion state through causal cross-modal attention operations with reliability gating constraints; predict the user's expression category, expression occurrence time, and expression intensity within the next time period based on the fusion state; perform probability calibration and confidence gating on the predicted expression category probabilities; determine whether the confidence level, modal reliability, and prediction interval have reached the triggering conditions; if the determination result is negative, control the robot to maintain a neutral expression or delay the response, waiting to receive the next batch of data; if the determination result is positive, combine the predicted expression occurrence time with the robot's execution delay to perform calculations and schedule the robot to trigger an early expression response. Collect the user's actual expression information, continuously update the system state, and correct the robot's expression transition effects. After completion, the process starts a new round of iterative judgment.

[0168] In a specific example, during the robot's proactive response control process, the robot's expression planning module stores at least the actuator trajectory, minimum preparation time, maximum speed, and safety margin parameters corresponding to each target expression. After the prediction module outputs the predicted expression category, expected appearance time, and expression intensity, the expression planning module first selects the appropriate response type based on the current dialogue scenario, and then, based on the start time t... start The system calculates the trigger time for the robot's facial expressions. For scenarios predicting a user is about to smile, it can pre-activate a slight smile. When a negative expression like sadness or anger is predicted, it prioritizes outputting attention, convergence, or reassuring expressions, avoiding mechanical replication of the user's negative expressions. To prevent frequent switching of robot expressions, the system is configured with a minimum expression duration parameter and a state transition matrix. When the target expressions output in consecutive preset prediction cycles (e.g., two) are different, switching the robot's current target expression is only allowed if the confidence level of the new prediction result is higher than the original prediction result by a preset margin, or if a significant change in the user's actual facial expression is detected.

[0169] Figure 4 This is an advance response timing diagram based on predicted occurrence time and actuator delay applicable to embodiments of the present invention, such as... Figure 4 As shown, the system predicts the time when the user's future facial expression will appear, subtracts the total system latency from this predicted time, and calculates the robot's facial expression activation time. Before the current time t, the system acquires historical dialogue and image window data and performs prediction calculations. Along the time axis, the robot's facial expression activation time is t+τ.pred -L 总 From this start-up moment, the robot sequentially experiences communication latency, control latency, and actuator latency, after which the robot's target expression enters the effective range; the predicted time of the user's expression appearance is t+τ. pred This position has a tolerance range. When the system detects that the user's actual expression has appeared, it enters the rolling recognition and correction stage, and dynamically corrects the expression that the robot is outputting based on the user's real expression.

[0170] For example, when a user engages in a relaxed conversation with the robot, facial images captured 1.5 seconds prior to the current time t show a gradual upward curve of the user's mouth and increased activity of the orbicularis oculi muscle. Simultaneously, the most recent conversation indicates a positive response, resulting in high confidence in the speech-to-text transcription. After cross-modal fusion of visual and conversational branch data through reliability gating, the system predicts that the user will exhibit a happy or smiling expression within the next 0.6 seconds, with this expression having the highest prediction probability. The predicted time of the expression's appearance is τ. pred Equal to 0.55 seconds; the total delay L obtained from the robot's facial control link detection is 0.55 seconds. 总 The time is 0.28 seconds. Based on this, the system calculates the startup time and then... pred -L 总 The system initiates the smiling motion trajectory at t+0.27 seconds, ensuring that the robot's output expression reaches a perceptible level near the actual time the user's smile occurs. If the facial image is occluded at the same time, reducing visual reliability, the system automatically lowers the contribution weight of visual features in the fusion process. If the dialogue content is also incomplete, and the final overall confidence level does not reach the preset threshold, the system abandons the premature smiling response, maintains the robot's current neutral expression, and waits to receive the next batch of input data. Through this predictable yet proactively non-triggered operating mechanism, the probability of the robot making erroneous pre-response is reduced.

[0171] It is important to emphasize that conventional passive recognition or trend judgment schemes aim to identify current facial expressions that have already appeared, or to judge current emotions and trends. This invention, however, directly predicts the specific facial expressions the user will exhibit within a future time window, the expected time of appearance of those expressions, and the intensity of those expressions. Regarding input constraints, conventional schemes analyze single-frame images or historical results, while this invention uses a dialogue and image history window with timestamps as input, and employs causal masks to ensure that the computation process does not access information from future moments. In terms of response timing, conventional schemes only initiate robot facial expression output after facial expression recognition is complete, while this invention uses the expected time of appearance of the expression minus communication latency, control latency, and actuator latency to pre-plan the initiation time of the robot's facial expressions. In terms of error trigger control, conventional solutions typically provide feedback based on the most probable expression category. This invention employs a joint gating mechanism comprised of probability calibration, category probability interval, and modal reliability. When the prediction confidence is low, it actively refuses to execute an early response. Regarding computation, conventional solutions may repeatedly process complete historical window data in each round of inference. This invention caches historical hidden states and only encodes newly added dialogue segments and image frames, achieving incremental inference computation. In the error correction stage, conventional solutions provide one-time triggers for feedback. This invention continuously reads the user's actual expression information during robot expression execution and performs rolling corrections for deviations between prediction and reality in expression category, occurrence time, and expression intensity.

[0172] Optionally, at least four control experiments can be set up. Group A is a scheme that responds based on the expression recognition of the current frame. Group B is a scheme that uses only the image history window to predict future expressions. Group C is a scheme that integrates dialogue and image features, but repeatedly encodes the complete window data in each round of reasoning. Group D is the technical solution of this invention with reliability gating, incremental multimodal prediction and time delay compensation functions. All control group models use the same training and testing user segmentation method and the same prediction window.

[0173] Evaluation metrics include: Top-1 (maximum) accuracy of future expressions, which is the percentage of samples correctly predicted for the main expression category within the prediction window, with an acceptance target of at least a preset percentage improvement (e.g., 8) compared to the image single-modal baseline; Macro-F1, the macro-average result of the F1 values ​​for each expression category, used to mitigate the impact of sample class imbalance, with an acceptance target of an improvement greater than or equal to a preset value (e.g., 0.08) relative to the image single-modal baseline; Occurrence Time (MAE), representing the average absolute error between the predicted expression occurrence time and the expression occurrence time labeled by humans or sensors, with an acceptance target of the error value not exceeding a preset duration (e.g., 200 milliseconds) and a reduction greater than or equal to a corresponding preset percentage compared to the non-gated fusion scheme (e.g., 30%); Early Hit Rate, referring to the proportion of samples where the robot enters the target expression state before or within the tolerance range of the user's actual expression, with an acceptance target of not less than or equal to a corresponding preset percentage (e.g., 80%); and Error Pre-Trigger Rate. The percentage of samples where the robot triggers a response before the user displays the target expression is the first metric. The acceptance target is less than or equal to the corresponding preset percentage (e.g., 8%), while being greater than or equal to the corresponding preset percentage (e.g., 30%) compared to the no-confidence gating scheme. The single-cycle inference latency is the end-to-end time from new data input to output prediction results. The acceptance target is less than or equal to the corresponding preset time (e.g., 30 milliseconds) on the edge computing platform, and being greater than or equal to the corresponding preset percentage (e.g., 35%) compared to the full-window repeated encoding scheme. The peak memory or computational load represents the peak memory usage or computational load per inference cycle during the online inference phase. The acceptance target is a reduction greater than or equal to the corresponding preset percentage (e.g., 25%) compared to the full-window repeated encoding scheme. The subjective interaction score is the five-point scale score given by the user for the naturalness, synchronicity, and comfort of the human-computer interaction. The acceptance target is an increase greater than or equal to the corresponding preset score (e.g., 0.5 points) compared to the average score of the passive recognition scheme.

[0174] It is understandable that the above values ​​are for reference only, and the specific values ​​can be set according to the actual application.

[0175] During the experiment, five types of experimental data can be recorded simultaneously. The first type is the accuracy, Macro-F1, and mean absolute error of occurrence time under different prediction windows, such as 0.3-0.8 seconds and 0.8-1.5 seconds. The second type is the performance change of the multimodal gating scheme compared to the ungated fusion scheme when face occlusion, lighting conditions, head posture, and speech transcription error rate change. The third type is the average latency, P95 latency (95% of request inference tasks take less than or equal to this latency), peak memory usage, and power consumption data of the incremental inference scheme and the full window repetitive encoding scheme on the same hardware platform. The fourth type is the robot's expression initiation lead time, the time required for the robot's expression to reach a perceptible amplitude, and the time difference between this moment and the moment when the user's expression actually appears. The fifth type is false trigger cases, refusal to trigger cases, and rolling correction cases, and the corresponding operation logs and video timestamps are retained.

[0176] The technical solution of this embodiment generates a rejection flag after the confidence gating judgment process is completed. When the probability of the largest expression category, the difference between the probability of the largest and second largest expression categories, or the modal comprehensive reliability does not meet the corresponding preset conditions, or when the expected appearance time of the expression is not in the executable range, a neutral expression is maintained. Thus, in the case of insufficient prediction confidence or mismatched prediction time, the robot can be actively prevented from executing the pre-response expression output, avoiding the problems of premature expression output under low confidence and inappropriate interaction, thus ensuring the stability of the human-computer interaction process.

[0177] Example 4

[0178] Figure 5 This is a schematic diagram of a user expression prediction and robot active response device provided in Embodiment 4 of the present invention. Figure 5 As shown, the device includes:

[0179] The acquisition and alignment module 510 is used to acquire historical dialogue window data and historical face image window data up to the current prediction time, and perform data alignment to obtain multimodal input data;

[0180] The reliability calculation module 520 is used to perform dialogue modality extraction and visual modality extraction operations on multimodal input data to obtain dialogue modality and visual modality, and calculate visual reliability based on multimodal input data and face image quality indicators, calculate dialogue reliability based on multimodal input data and speech recognition and sentence completeness indicators, and calculate modality comprehensive reliability based on dialogue reliability and visual reliability.

[0181] The first-round module 530 is used to take the preset initial state as the historical hidden state and the multimodal input data as the data to be encoded if the current prediction time is the first-round prediction.

[0182] The non-first-round module 540 is used to obtain the historical hidden state cached in the previous prediction and the encoded dialogue and image vectors if the current prediction time is not the first-round prediction. It compares the multimodal input data with the encoded dialogue and image vectors and filters out the newly added dialogue fragments and newly added face image frames as data to be encoded.

[0183] The modality fusion module 550 is used to perform encoding processing on the data to be encoded, perform incremental temporal state update operation in combination with historical hidden states to obtain the updated temporal context state, and input the temporal context state, dialogue modality, visual modality, dialogue reliability and visual reliability into the gating network to perform causal cross-modal fusion processing to obtain the multimodal fusion state;

[0184] The task prediction module 560 is used to input the multimodal fusion state into the future expression prediction head, perform multi-task prediction processing, and output the probability of expression category, the expected time of expression occurrence, and the intensity of expression within a preset time period in the future.

[0185] The advance triggering module 570 is used to perform probability calibration on the expression category probability. Based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time, confidence gating judgment is performed. When the maximum expression category probability, the difference between the maximum and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, an advance triggering flag is generated. This flag triggers the robot to output an expression that matches the maximum expression category probability, the expected expression occurrence time, and the expression intensity.

[0186] The technical solution of this invention obtains multimodal input data by aligning historical dialogue window data and historical face image window data, and calculates visual reliability and dialogue reliability based on face image quality indicators, speech recognition and sentence completeness indicators, respectively, which can quantitatively characterize the data quality of the two types of modal inputs; by using the first-round or non-first-round prediction differentiation processing logic, the non-first round only selects newly added dialogue segments and newly added face image frames as data to be encoded and performs incremental temporal state updates by combining the cached historical hidden states, without having to repeatedly encode all historical window data, reducing the encoding computation overhead and memory access volume at the edge; and by using a gating network to complete causal cross-modal fusion, combining the various modalities The system dynamically adjusts modal contributions based on its own reliability, reducing the interference of low-quality modalities on the fusion results. The future expression prediction head performs multi-task prediction, outputting expression category probabilities, expected expression occurrence times, and expression intensities to support early responses with latency adaptation. After probability calibration, confidence gating is performed based on probability conditions, overall modal reliability, and expected expression occurrence times. Only when multiple conditions are met simultaneously is an early trigger flag generated to drive the robot to output matching expressions. Under the limited computing power at the edge, this system balances data quality constraints of multimodal inputs with temporal inference efficiency, reduces invalid triggers, and improves the reliability and rationality of the robot's early response to users' future expressions, avoiding delayed or misaligned expression responses.

[0187] Based on the above embodiments, the modal fusion module 550 is specifically used for:

[0188] Based on the dialogue reliability and visual reliability input to the gating network, the target modality with reduced or missing reliability is identified, and the contribution weight corresponding to the target modality is reduced. A weighted operation is performed based on the temporal context state, dialogue modality, visual modality, dialogue reliability, visual reliability, and contribution weight to obtain the multimodal fusion state. The gating network is configured with an attention mask, which is applied to the temporal feature sequence inside the gating network to mask the internal time step positions representing the time after the current moment in the temporal feature sequence, constraining the model to only call the current and historical time step information in the temporal feature sequence to participate in the calculation.

[0189] Based on the above embodiments, the advance module 570 is specifically used for:

[0190] A temperature scaling method is used to perform probability calibration on the expression category probabilities to obtain calibrated expression category probabilities. The maximum expression category probability and the second largest expression category probability are extracted from the calibrated expression category probabilities, and the probability difference between the maximum expression category probability and the second largest expression category probability is calculated. If all the conditions are met, such as the maximum expression category probability being greater than or equal to a preset probability threshold, the probability difference being greater than or equal to a preset interval threshold, the modal comprehensive reliability being greater than or equal to a reliability threshold, and the expected appearance time of the expression falling within the executable interval, an early triggering flag is generated.

[0191] Furthermore, based on the above embodiments, the user expression prediction and robot active response device may further include:

[0192] The neutral expression module is used to generate a rejection early trigger flag after performing confidence gating judgment processing based on the calibrated expression category probability, modal comprehensive reliability and the expected expression occurrence time. If the maximum expression category probability, the difference between the maximum and second largest expression category probabilities or the modal comprehensive reliability do not meet the corresponding preset conditions, or the expected expression occurrence time is not within the executable range, the robot maintains neutral expression by rejecting early trigger flag.

[0193] Based on the above embodiments, the advance module 570 is further configured to:

[0194] The robot's communication latency, control latency, and actuator latency are acquired and summed to obtain the total robot latency. Based on the predicted expression occurrence time and the total robot latency, the robot's expression initiation time is calculated. If the robot's expression initiation time is earlier than or equal to the current time, an expression-driving command is issued to the robot. If the robot's expression initiation time is later than the current time, a timed task is created to issue the expression-driving command to the robot after reaching the expression initiation time. During the robot's response to the expression-driving command and expression output, the user's actual facial expressions are continuously collected, and the actual expression category, actual occurrence time, and actual expression intensity are extracted as actual expression information. The actual expression information is compared with the expression category probability, predicted expression occurrence time, and expression intensity output by the multi-task prediction processing to obtain the expression deviation. When the expression deviation exceeds a preset deviation threshold, the robot's current expression amplitude is reduced, or the robot is controlled to revert to a neutral transition state. The current time is used as the current prediction time, and the process of collecting historical dialogue window data and historical face image window data up to the current prediction time is resumed.

[0195] Furthermore, based on the above embodiments, the user expression prediction and robot active response device may further include:

[0196] The unified clock module is used to acquire the original data of human-computer interaction sessions authorized by the user and to unify the time clock of dialogues and face images within the data.

[0197] The neutral labeling module is used to extract the dialogue window and face image window before each preset sampling time as model input samples, and identify the user's real expression in the future prediction window at the sampling time. The model input samples are labeled with the expression category, the time when the expression first reaches the intensity threshold, and the maximum expression intensity. If no expression change is identified in the future prediction window at the sampling time, the model input samples are labeled with a neutral category.

[0198] The dataset partitioning module is used to perform dataset partitioning on all model input samples according to user category, resulting in training set, validation set and test set respectively; notably, the same user's session will not be distributed in both the training set and the test set at the same time.

[0199] The sample balancing module is used to perform sample balancing processing on samples of different expression categories in the training set according to the number of samples of each expression category, so as to obtain a training sample set for training the future expression prediction head.

[0200] Furthermore, based on the above embodiments, the user expression prediction and robot active response device may further include:

[0201] The data augmentation module is used to perform face cropping on the face samples in the training sample set, and then perform data augmentation processing on the face samples in terms of brightness, scale, and slight pose.

[0202] The model training module is used to construct a comprehensive training loss function based on the training sample set, which includes future expression category loss, occurrence time regression loss, intensity regression loss, and probability calibration or consistency constraint loss. The loss weights corresponding to each type of loss are determined on the validation set. The prediction model is trained using the comprehensive training loss function, and the trained prediction model is used as the future expression prediction head.

[0203] The user expression prediction and robot active response device provided in the embodiments of the present invention can execute the user expression prediction and robot active response method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0204] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0205] In the technical solution disclosed herein, the information collected is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0206] In the technical solution disclosed herein, if automated decision-making is involved, a corresponding operation entry will be provided to the user, allowing the user to choose to agree to or reject the automated decision result; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0207] Example 5

[0208] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0209] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0210] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0211] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as user expression prediction and robot proactive response methods, namely:

[0212] Collect historical dialogue window data and historical face image window data up to the current prediction time, and align the data to obtain multimodal input data;

[0213] Perform dialogue modality extraction and visual modality extraction operations on multimodal input data to obtain dialogue modality and visual modality. Calculate visual reliability based on multimodal input data and face image quality indicators, calculate dialogue reliability based on multimodal input data and speech recognition and sentence completeness indicators, and calculate overall modality reliability based on dialogue reliability and visual reliability.

[0214] If the current prediction time is the first round of prediction, then the preset initial state will be used as the historical hidden state, and the multimodal input data will be used as the data to be encoded.

[0215] If the current prediction time is not the first round of prediction, then obtain the historical hidden state cached from the previous prediction as well as the encoded dialogue and image vectors. Compare the multimodal input data with the encoded dialogue and image vectors, and filter out the new dialogue fragments and new face image frames as the data to be encoded.

[0216] Encoding processing is performed on the data to be encoded, and incremental temporal state update operation is performed in combination with the historical hidden state to obtain the updated temporal context state. The temporal context state, dialogue modality, visual modality, dialogue reliability and visual reliability are then input into the gating network to perform causal cross-modal fusion processing to obtain the multimodal fusion state.

[0217] The multimodal fusion state is input into the future expression prediction head, multi-task prediction processing is performed, and the expression category probability, the expected occurrence time of the expression, and the expression intensity are output within a preset time period in the future.

[0218] The probability of each expression category is calibrated. Based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time, confidence gating is performed. When the maximum expression category probability, the difference between the maximum and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, an early trigger flag is generated. This flag triggers the robot to output an expression that matches the maximum expression category probability, the expected expression occurrence time, and the expression intensity.

[0219] In some embodiments, the user expression prediction and robot proactive response method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the user expression prediction and robot proactive response method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the user expression prediction and robot proactive response method by any other suitable means (e.g., by means of firmware).

[0220] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0221] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0222] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0223] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0224] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0225] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0226] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0227] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for predicting user facial expressions and for a robot to proactively respond, characterized in that, include: Collect historical dialogue window data and historical face image window data up to the current prediction time, and align the data to obtain multimodal input data; Perform dialogue modality extraction and visual modality extraction operations on multimodal input data to obtain dialogue modality and visual modality. Calculate visual reliability based on multimodal input data and face image quality indicators, calculate dialogue reliability based on multimodal input data and speech recognition and sentence completeness indicators, and calculate overall modality reliability based on dialogue reliability and visual reliability. If the current prediction time is the first round of prediction, then the preset initial state will be used as the historical hidden state, and the multimodal input data will be used as the data to be encoded. If the current prediction time is not the first round of prediction, then obtain the historical hidden state cached from the previous prediction as well as the encoded dialogue and image vectors. Compare the multimodal input data with the encoded dialogue and image vectors, and filter out the new dialogue fragments and new face image frames as the data to be encoded. Encoding processing is performed on the data to be encoded, and incremental temporal state update operation is performed in combination with the historical hidden state to obtain the updated temporal context state. The temporal context state, dialogue modality, visual modality, dialogue reliability and visual reliability are then input into the gating network to perform causal cross-modal fusion processing to obtain the multimodal fusion state. The multimodal fusion state is input into the future expression prediction head, multi-task prediction processing is performed, and the expression category probability, the expected occurrence time of the expression, and the expression intensity are output within a preset time period in the future. The probability of each expression category is calibrated. Based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time, confidence gating is performed. When the maximum expression category probability, the difference between the maximum and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, an early trigger flag is generated. This flag triggers the robot to output an expression that matches the maximum expression category probability, the expected expression occurrence time, and the expression intensity.

2. The method according to claim 1, characterized in that, The temporal context state, dialogue modality, visual modality, dialogue reliability, and visual reliability are input into a gating network to perform causal cross-modal fusion processing, resulting in a multimodal fusion state, including: Based on the dialogue reliability and visual reliability input to the gating network, the target modal with reduced or missing reliability is identified, and the contribution weight of the target modal is reduced. A weighted operation is performed based on the temporal context state, dialogue modality, visual modality, dialogue reliability, visual reliability, and contribution weight to obtain the multimodal fusion state; The gating network is equipped with an attention mask, which is applied to the temporal feature sequence inside the gating network to mask the internal time step positions that represent the time after the current time in the temporal feature sequence. This constrains the model to only call the current and historical time step information in the temporal feature sequence for calculation.

3. The method according to claim 1, characterized in that, The probability of each expression category is calibrated. Based on the calibrated expression category probability, modal comprehensive reliability, and expected expression occurrence time, a confidence gating process is performed. When the probability of the largest expression category, the difference between the largest and second-largest expression category probabilities, and the modal comprehensive reliability all meet the corresponding preset conditions, and the expected expression occurrence time is within the executable range, a flag allowing early triggering is generated, including: A temperature scaling method is used to perform probability calibration on the expression category probabilities to obtain the calibrated expression category probabilities. Extract the maximum and second-largest expression category probabilities from the calibrated expression category probabilities, and calculate the probability difference between the maximum and second-largest expression category probabilities. If all the following conditions are met: the probability of the largest expression category is greater than or equal to the preset probability threshold, the probability difference is greater than or equal to the preset interval threshold, the overall modality reliability is greater than or equal to the reliability threshold, and the expected appearance time of the expression falls within the executable interval, a flag allowing early triggering is generated.

4. The method according to claim 1, characterized in that, After performing confidence-gated judgment based on the calibrated expression category probability, modality comprehensive reliability, and expected expression occurrence time, the following steps are also included: If the probability of the largest expression category, the difference between the probabilities of the largest and second largest expression categories, or the modal comprehensive reliability does not meet the corresponding preset conditions, or if the expected occurrence time of the expression is not within the executable range, a rejection early trigger flag is generated to maintain the robot's neutral expression through the rejection early trigger flag.

5. The method according to claim 1, characterized in that, The expressions that trigger the robot's output match the probability of the maximum expression category, the expected time of expression appearance, and the expression intensity, including: Obtain the robot's communication latency, control latency, and actuator latency, and sum them to obtain the robot's total latency; The robot's expression activation time is calculated based on the expected appearance time of the expression and the robot's total latency. If the robot's expression activation time is earlier than or equal to the current time, issue an expression-driven command to the robot; If the robot's expression activation time is later than the current time, create a timed task, and after running to the robot's expression activation time, issue an expression-driven command to the robot; During the process of the robot responding to facial expression-driven commands and executing facial expression output, the robot continuously collects the user's actual facial expressions and extracts the actual expression category, actual occurrence time, and actual expression intensity as actual expression information. The actual facial expression information is compared with the facial expression category probability, the expected occurrence time of the expression, and the expression intensity output by the multi-task prediction processing to obtain the facial expression bias; When the expression deviation exceeds the preset deviation threshold, the robot's current expression movement amplitude is reduced, or the robot is controlled to revert to a neutral transition state. The current moment is then used as the current prediction moment, and the operation of collecting historical dialogue window data and historical face image window data up to the current prediction moment is performed.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the original data of human-computer interaction sessions authorized by the user, and unify the time clock of dialogue and face images within the data; For each preset sampling time, the dialogue window and face image window before the sampling time are extracted as model input samples, and the user's real expression in the future prediction window at the sampling time is identified. The expression category, the time when the expression first reaches the intensity threshold, and the maximum expression intensity are labeled on the model input samples. If no expression change is identified in the future prediction window at the sampling time, the neutral category is labeled on the model input samples. The dataset is divided into training, validation, and test sets according to user categories. Notably, a user's session will not be simultaneously distributed within both the training and test sets. Based on the number of samples for each expression category in the training set, a sample balancing process is performed on the samples of different expression categories in the training set to obtain a training sample set for training the future expression prediction head.

7. The method according to claim 6, characterized in that, The method further includes: After performing face cropping on the face samples in the training sample set, data augmentation processing is performed on the face samples to adjust brightness, scale, and slight pose. Based on the training sample set, a comprehensive training loss function is constructed, which includes future expression category loss, occurrence time regression loss, intensity regression loss, and probability calibration or consistency constraint loss. The loss weights corresponding to each type of loss are determined on the validation set. The prediction model is trained using the comprehensive training loss function, and the trained prediction model is used as the future expression prediction head.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the user expression prediction and robot active response method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the user facial expression prediction and robot active response method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the user expression prediction and robot active response method according to any one of claims 1-7.

Images