Interaction method based on face image, electronic equipment and computer program product

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides an interaction method based on a face image, electronic equipment and a computer program product. The interaction method based on the face image comprises the following steps: predicting an expected position of each tracking object in a current frame based on a face image of a historical frame; performing position matching on the actual position of the target object in the face image of the current frame and the expected position of each tracking object, and binding the target object with the tracking object successfully matched in position; and based on the emotional state and the lip state of the current frame and the emotional state and the lip state of a tracking object bound with the target object in the historical frame, carrying out smoothing processing so as to execute an interaction instruction based on the actual emotional state and the actual lip state obtained through smoothing processing, according to the method, the same object can be subjected to multi-dimensional data fusion through accurate face tracking, so that the reliability of interactive operation is higher, and more natural and humanized interactive response can be generated conveniently.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to an interaction method, electronic device and computer program product based on facial images. Background Technology

[0002] In scenarios such as human-computer interaction, intelligent customer service, and virtual assistants, accurately perceiving and responding naturally to user emotions is crucial for improving user experience. Existing technologies typically employ emotion recognition methods based on the user's current modality, such as emotion analysis based solely on facial expressions or intent recognition based solely on speech. These methods are highly susceptible to misusing random facial movements or emotions in the interaction, affecting the reliability of the interaction and leading to a decline in user experience. Summary of the Invention

[0003] This application provides an interaction method, electronic device, and computer program product based on facial images, which can improve the reliability of the interaction.

[0004] The first aspect of this application provides an interaction method based on face images, comprising: predicting the expected position of each tracked object in the current frame based on face images of historical frames; identifying the emotional state and lip state of the target object in the current frame based on the face image of the current frame, and determining the actual position of the target object in the face image of the current frame; performing position matching between the actual position and the expected position of each tracked object, and binding the target object to the tracked object whose position is successfully matched; performing smoothing processing based on the emotional state and lip state of the current frame, and the emotional state and lip state of the tracked object bound to the target object in historical frames, to obtain the actual emotional state and actual lip state of the target object; and executing an interaction command based on the actual emotional state and actual lip state.

[0005] In some implementations of the first aspect, the actual emotional state of the target object is obtained by smoothing the emotional state of the target object based on the current frame's emotional state and the emotional state of the tracked object bound to the target object in historical frames. This includes: determining the frequency of occurrence of each emotional state in the current frame and historical frames; taking the emotional state with an occurrence frequency greater than a frequency threshold as the actual emotional state; if there is no emotional state with an occurrence frequency greater than the frequency threshold, calculating the comprehensive confidence of each emotional state based on the confidence of the current frame's emotional state and the confidence of the emotional state in historical frames; and taking the emotional state with the highest comprehensive confidence as the actual emotional state.

[0006] In some implementations of the first aspect, a comprehensive confidence score for each emotional state is calculated based on the confidence score of the emotional state in the current frame and the confidence scores of the emotional states in historical frames, including: for each emotional state, weighted fusion of the confidence scores corresponding to each frame, wherein the weight of the confidence score corresponding to each frame is related to the time interval between the frame and the current frame.

[0007] In some implementations of the first aspect, the actual lip state of the target object is obtained by smoothing the lip state of the target object in the historical frames based on the lip state of the current frame and the lip state of the tracked object bound to the target object in the historical frames, including: if the lip state of the current frame is a lip movement state, and the proportion of frames with lip movement states in the historical frames is greater than the proportion threshold, then the lip movement state is taken as the actual lip state; otherwise, the non-lip movement state is taken as the actual lip state.

[0008] In some implementations of the first aspect, identifying the lip state of a target object in the current frame based on a face image of the current frame includes: determining the lip movement amplitude relative to the current frame based on the face image; determining the amplitude difference between the lip movement amplitude of the current frame and the maximum lip movement amplitude in each historical frame; and confirming that the current frame is in a lip movement state if the amplitude difference is greater than a difference threshold.

[0009] In some embodiments of the first aspect, the interaction method further includes: after all objects in the face image of the current frame have been matched, determining the number of consecutive lost frames of the unmatched tracking objects; if the number of consecutive lost frames is greater than a frame number threshold, storing the face features of the unmatched tracking objects into a preset feature library.

[0010] In some implementations of the first aspect, if the target object fails to match with each tracked object, the interaction method further includes: matching the facial features of the target object with the facial features of each tracked object in a preset feature library; and binding the target object with the tracked object whose features are successfully matched.

[0011] In some embodiments of the first aspect, the interaction method further includes: performing feature matching between the facial features of the target object and each known object in the face database, and binding the identity information of the target object with the known object whose features are successfully matched.

[0012] A second aspect of this application provides an interactive device based on face images, comprising: a prediction unit for predicting the expected position of each tracked object in the current frame based on face images of historical frames; a detection unit for identifying the emotional state and lip state of a target object in the current frame based on the face image of the current frame, and determining the actual position of the target object in the face image of the current frame; a matching unit for matching the actual position with the expected position of each tracked object, and binding the target object to the tracked object whose position is successfully matched; a smoothing unit for performing smoothing processing based on the emotional state and lip state of the current frame, and the emotional state and lip state of the tracked object bound to the target object in historical frames, to obtain the actual emotional state and actual lip state of the target object; and an interaction unit for executing interaction commands based on the actual emotional state and actual lip state.

[0013] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described interaction method based on a face image.

[0014] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described interaction method based on a face image.

[0015] The fifth aspect of this application provides a computer program product that, when the computer program is run, causes the above-described face image-based interaction method to be executed.

[0016] In the embodiments of this application, the expected position of each tracked object in the current frame is predicted based on the face image of historical frames; the actual position of the target object in the face image of the current frame is matched with the expected position of each tracked object, and the target object is bound to the tracked object whose position is successfully matched; based on the emotional state and lip state of the current frame, and the emotional state and lip state of the tracked object bound to the target object in the historical frames, smoothing processing is performed, so as to execute the interaction command based on the actual emotional state and actual lip state obtained by the smoothing processing. This enables multi-dimensional data fusion of the same object through accurate face tracking, making the reliability of the interaction operation higher and facilitating the generation of more natural and humanized interaction responses. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating the implementation process of an interaction method based on face images provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram illustrating the specific implementation process of the matching and tracking object provided in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram illustrating the specific implementation process for determining the actual emotional state provided in the embodiments of this application;

[0021] Figure 4 This is a schematic diagram of six key points provided in the embodiments of this application;

[0022] Figure 5 This is a schematic diagram illustrating the specific implementation process for determining the actual lip state provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the modules provided in the embodiments of this application;

[0024] Figure 7 This is a schematic diagram of the structure of an interactive device based on a face image provided in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are protected by this application.

[0027] In scenarios such as human-computer interaction, intelligent customer service, and virtual assistants, accurately perceiving and responding naturally to user emotions is crucial for improving user experience. Existing technologies typically employ emotion recognition methods based on the user's current modality, such as emotion analysis based solely on facial expressions or intent recognition based solely on speech. These methods are highly susceptible to misusing random facial movements or emotions in the interaction, affecting the reliability of the interaction and leading to a decline in user experience.

[0028] In view of this, this application proposes an interaction method based on face images, which can perform multi-dimensional data fusion on the same object through accurate face tracking, thereby making the reliability of the interaction operation higher and facilitating the generation of more natural and humanized interaction responses.

[0029] To illustrate the technical solution of this application, specific embodiments are described below.

[0030] Figure 1 This illustration shows a schematic diagram of the implementation process of an interaction method based on facial images provided in an embodiment of this application. This method can be applied to electronic devices. The aforementioned electronic devices refer to intelligent devices such as computers (including embedded computers), smartphones, robots, and servers, and this application does not impose any limitations on this.

[0031] It should be noted that this application is based on facial images, which can be obtained with user authorization. Each facial image may contain the faces of one or more objects.

[0032] Specifically, the above-mentioned face image-based interaction method may include the following steps S101 to S105.

[0033] Step S101: Based on the face images of historical frames, predict the expected position of each tracked object in the current frame.

[0034] In this context, the tracked object refers to any object that has appeared in the face images of historical frames, used for trajectory tracking between different frames. Specifically, different objects can be identified using the face images of historical frames, and objects from different historical frames can be matched. Successfully matched objects can be considered as the same tracked object; otherwise, they are treated as new tracked objects. By utilizing the positions of each tracked object in each historical frame, the trajectory and movement speed of each tracked object can be predicted, thereby predicting the expected position of each tracked object in the current frame. The expected position is the predicted location of the tracked object in the face image of the current frame.

[0035] Step S102: Based on the face image of the current frame, identify the emotional state and lip state of the target object in the current frame, and determine the actual position of the target object in the face image of the current frame.

[0036] Specifically, when a face image of the current frame is captured, image recognition can be used to identify the target object's emotional state, lip state, and actual position within the face image of the current frame. The target object can be any object appearing in the face image of the current frame. The emotional state represents the target object's emotion, such as happiness, sadness, anxiety, anger, or surprise. The lip state indicates whether the target object is speaking, including lip movement and non-lip movement. Both emotional state and lip state reflect the user's interaction needs. The actual position of the target object in the face image of the current frame can refer to the position of the target object's face detection bounding box, used for matching with the tracked object.

[0037] Step S103: Match the actual location with the expected location of each tracked object, and bind the target object to the tracked object whose location has been successfully matched.

[0038] Specifically, the actual location is matched with the expected location of each tracked object. When the expected location of a tracked object is the same as or close to the actual location, the target object and the tracked object belong to the same object. Therefore, the target object and the tracked object can be considered to be successfully matched. At this time, the target object can be bound to the tracked object that has been successfully matched to associate the information of the current frame and the historical frame.

[0039] This application does not impose any restrictions on the location matching method. For example, location matching can be performed by calculating the distance between the actual location and the expected location of each tracked object, or by calculating the intersection over union (IoU) between the face detection box corresponding to the actual location and the face detection box corresponding to the expected location of each tracked object. The matching algorithm can be implemented using the Hungarian algorithm.

[0040] Step S104: Based on the emotional state and lip state of the current frame, and the emotional state and lip state of the tracked object bound to the target object in the historical frames, smoothing is performed to obtain the actual emotional state and actual lip state of the target object.

[0041] In other words, by using the target object's emotional state and lip state in the current frame, and its emotional state and lip state in historical frames, a smoothing process can be performed to obtain the target object's current actual emotional state and lip state. This smoothing process avoids the reliability of subsequent interactive operations being affected by random facial movements or emotions in the current frame's emotional state and lip state.

[0042] Step S105: Execute the interaction command based on the actual emotional state and the actual lip state.

[0043] Specifically, based on the actual emotional state and actual lip movements, a combined decision can be made to analyze the user's current need for interaction, generate and execute interaction commands to interact with the user. These interaction commands can include voice output commands, robot action commands, interface update commands, data recording commands, etc.

[0044] For example, if the emotion of "happiness" is detected and the user is in a lip-moving state, the system will proactively trigger the voice output command, "You look very happy, do you need help?" If the emotion of "sadness" is detected and the user is not in a lip-moving state, the system will trigger comfort-related commands, such as playing soothing music or changing the interface color. In a lip-moving state, the voice recognition module can also be activated to prepare for receiving user commands.

[0045] In the embodiments of this application, the expected position of each tracked object in the current frame is predicted based on the face image of historical frames; the actual position of the target object in the face image of the current frame is matched with the expected position of each tracked object, and the target object is bound to the tracked object whose position is successfully matched; based on the emotional state and lip state of the current frame, and the emotional state and lip state of the tracked object bound to the target object in the historical frames, smoothing processing is performed, so as to execute the interaction command based on the actual emotional state and actual lip state obtained by the smoothing processing. This enables multi-dimensional data fusion of the same object through accurate face tracking, making the reliability of the interaction operation higher and facilitating the generation of more natural and humanized interaction responses.

[0046] In some embodiments of this application, the electronic device can continuously capture facial images using a camera to obtain facial images of historical frames and the current frame. The camera can be an RGB camera, a depth camera, or other types of cameras.

[0047] In some embodiments of this application, the electronic device can obtain the position of each frame through a face detection and recognition model, extract the emotional state using an emotion recognition model, and obtain lip feature points using a face key point detection model, so as to determine the lip state using the lip feature points.

[0048] This application does not impose any limitations on face detection and recognition models, emotion recognition models, or facial landmark detection models. For example, a face detection model can employ a lightweight face detector based on YOLOv6n, whose output includes the face bounding box coordinates (x, y, w, h) and confidence scores, providing basic input for subsequent processing. A face recognition model can employ a feature extractor based on the ResNet100 architecture, which can extract a highly discriminative 512-dimensional feature vector from detected face regions for identity matching and re-identification tasks. An emotion recognition model can use RepVGG as the classifier for the backbone network, achieving a good balance between accuracy and efficiency. Its output covers probability distributions of various emotional states, providing a semantic understanding foundation for emotional interaction. A facial landmark detection model can employ a lightweight landmark detection network based on an improvement of MobileNetV2, optimizing predictions for 68 facial landmarks, with particular attention to landmarks in the lip region, providing accurate geometric information for lip movement detection.

[0049] To ensure the continuity of the trajectory and the consistency of the identity, the actual location can be matched with the expected location of each tracked object, and the target object can be bound to the tracked object whose location has been successfully matched.

[0050] Specifically, face images from the current frame and historical frames can be fed into the face detection model, which outputs the location and confidence score of all face detection boxes. To improve matching efficiency, low-confidence face detection boxes can be filtered out based on a preset threshold, retaining only reliable ones.

[0051] like Figure 2 As shown, for the face detection bounding boxes detected after face detection in a face image, Kalman filtering is first used to predict the position and velocity of each tracked object in historical frames, estimating the expected position of each tracked object in the current frame. Then, the intersection-over-union ratio (IoU) of the predicted position and the face detection bounding box in the current frame is calculated. The Hungarian algorithm is used to complete position matching, binding the target object to the successfully matched tracked object to establish the correspondence between the face detection bounding box in the current frame and the existing trajectory.

[0052] For a target object that is bound to a tracking object, the target object's data (such as the current frame's emotional state and lip state, and actual location) can be added to the tracking object's dataset to update the tracking object's motion trajectory, making it easier to predict the expected location in the next frame.

[0053] Correspondingly, the interaction method may also include: after all objects in the face image of the current frame have been matched, determining the number of consecutive lost frames of the unmatched tracking objects; if the number of consecutive lost frames is greater than the frame number threshold, storing the face features of the unmatched tracking objects into a preset feature library.

[0054] Specifically, such as Figure 2 As shown, if the object being tracked in the face image of the current frame is not matched, it can be considered that the object being tracked has been lost in the current frame. By judging the number of consecutive lost frames, if the number of consecutive lost frames is greater than the frame number threshold, it is moved into the preset feature library, so that the object being tracked enters the re-retrieval state.

[0055] If the target object fails to match any of the tracked objects, the interaction method may further include: matching the facial features of the target object with the facial features of each tracked object in the preset feature library, and binding the target object to the tracked object whose features are successfully matched.

[0056] In other words, such as Figure 2 As shown, if the target object does not match the tracked object, the image region of the target object in the face image can be cropped and input into the face recognition model to extract its face feature vector. This vector is then compared with the similarity of each tracked object in the preset feature library that has entered the re-retrieval state. If the similarity exceeds a preset threshold (e.g., 0.85), the tracked object is restored, and the target object is bound to the tracked object that has successfully matched the features, using the same identifier to avoid the identifier switching problem when the target object briefly leaves the screen and then returns.

[0057] For each tracked object in a pre-defined feature library that has been retrieved in a different state, a duration threshold can be set. For example... Figure 3 As shown, for each tracked object in the re-retrieval state, if the time elapsed between the data in the preset feature library and the current frame exceeds the time threshold, the expired data is cleared to avoid infinite memory growth, while maintaining a sufficient time window for re-identification.

[0058] Ultimately, a closed loop of "detection, prediction, association, and updating" can be formed, ensuring accurate and stable face tracking even in multi-target and dynamically changing scenarios. The introduction of a pre-defined feature library and feature matching strategy improves the stability of face tracking in multi-person scenarios, providing a reliable guarantee for continuous identity recognition.

[0059] In some embodiments of this application, such as Figure 3 As shown, the actual emotional state of the target object is obtained by smoothing the emotional state of the target object based on the current frame's emotional state and the emotional state of the tracked object bound to the target object in the historical frames. This can include steps S301 to S304.

[0060] Step S301: Determine the frequency of occurrence of each emotional state in the current frame and the emotional states of historical frames.

[0061] Specifically, the image region within the face detection bounding box of each frame is input into the emotion recognition model, which outputs the category confidence distribution of the emotion state. The emotion state with the highest confidence is taken as the emotion state of the corresponding frame. A fixed-length emotion history queue is maintained for each identifier, storing the most recently identified emotion category and its corresponding confidence. For a target object, the frequency of each emotion state of the target object in the emotion history queue can be counted through its identifier.

[0062] Step S302: Emotional states that occur more frequently than a frequency threshold are taken as actual emotional states.

[0063] When the frequency of a certain emotional state in the emotional history queue exceeds a frequency threshold (e.g., 60%), the emotional state can be considered a stable emotional outcome for the user and can be taken as the actual emotional state.

[0064] Step S303: If there is no emotional state with a frequency greater than the frequency threshold, calculate the comprehensive confidence of each emotional state based on the confidence of the emotional state in the current frame and the confidence of the emotional state in the historical frames.

[0065] Step S304: The emotional state with the highest overall confidence level is taken as the actual emotional state.

[0066] When no emotional state occurs more frequently than a frequency threshold, it indicates that there is no clearly dominant emotion. In this case, a comprehensive confidence score can be calculated for each emotional state, and the emotional state with the highest comprehensive confidence score can be taken as the actual emotional state. The comprehensive confidence score characterizes the degree of credibility that the target object is in that emotional state.

[0067] In some embodiments of this application, calculating the comprehensive confidence of each emotional state based on the confidence of the emotional state of the current frame and the confidence of the emotional state of historical frames may include: for each emotional state, weighted fusion of the confidence corresponding to each frame.

[0068] The weight of the confidence level for each frame is related to the time interval between that frame and the current frame.

[0069] Specifically, the overall confidence level = Σ(confidence level_i × decay factor^t), where confidence level_i represents the confidence level of a certain emotional state in the i-th frame, decay factor^t is the weight, t represents the time interval between the current frame and the current frame, and the decay factor can be a constant between 0 and 1, for example, 0.9.

[0070] It should be noted that when a valid emotional state cannot be obtained for several consecutive frames (such as when the face is severely obscured or the face is in profile), the most recent valid emotional state can be maintained to avoid drastic fluctuations in output.

[0071] In some embodiments of this application, identifying the lip state of a target object in the current frame based on the face image of the current frame may include: determining the lip movement amplitude relative to the current frame based on the face image; determining the amplitude difference between the lip movement amplitude of the current frame and the maximum lip movement amplitude in each historical frame; and confirming that the current frame is in a lip movement state if the amplitude difference is greater than the difference threshold.

[0072] Specifically, based on a facial landmark detection model, six key points in the lip region can be accurately located, such as... Figure 4 As shown, these six key points can be divided into three pairs, each pair including an upper lip key point and a lower lip key point. The extracted key point coordinates are normalized to eliminate the influence of facial distance and scale variations, ensuring cross-user detection consistency. The lip movement amplitude can be calculated using the Euclidean distance between each pair of key points. Specifically, the difference in key point distances between the current frame and the previous frame can be calculated, and the average of the absolute values ​​is taken as the lip movement amplitude of the current frame.

[0073] At this point, the lip movement amplitudes of the most recent M frames (e.g., M=10) are recorded to form a queue. The absolute difference between the lip movement amplitude of the current frame and the maximum lip movement amplitude in the queue is calculated. If the difference is greater than the difference threshold (e.g., 0.1), the current frame is determined to be in a lip movement state.

[0074] like Figure 5 As shown, the actual lip state of the target object is obtained by smoothing the lip state of the target object based on the current frame's lip state and the lip state of the tracked object bound to the target object in the historical frames, which may include steps S501 to S502.

[0075] Step S501: If the lip state of the current frame is a lip movement state, and the proportion of frames with lip movement states in the historical frames is greater than the proportion threshold, then the lip movement state is taken as the actual lip state.

[0076] Step S502, otherwise, the non-lip movement state is taken as the actual lip state.

[0077] Specifically, the initial lip movement states of the most recent N frames (e.g., N=10) are recorded to form a queue. If the current frame is in a lip movement state, and the proportion of frames in the queue that are in a lip movement state exceeds a threshold (e.g., 30%), then the final output lip movement state is a lip movement state; otherwise, the output lip movement state is a non-lip movement state.

[0078] Thus, by using an innovative dual-queue smoothing mechanism for two-layer filtering, the system effectively distinguishes between real speech lip movements and random facial movements, significantly reducing the misjudgment rate caused by environmental interference and providing reliable technical support for the accurate triggering of voice interaction.

[0079] In addition, the interaction method may also include: matching the facial features of the target object with the features of each known object in the face database, and binding the identity information of the target object with the known object whose features are successfully matched.

[0080] Specifically, facial features of known objects can be pre-collected to form a face database. Known objects are those with clearly defined identities, such as those whose names are known. The face database does not need to be dynamically updated. The extracted facial features of the target object are compared with the facial features of each known object in the face database to calculate similarity. If the similarity is higher than a threshold, the identity information of the target object and that known object with a similarity higher than the threshold is bound; otherwise, it is marked as a stranger.

[0081] At this point, once the target object is bound to the tracking object and assigned a tracking identifier of "5", the identity information "Zhang San" can be returned for the object with the identifier "5" through identity information binding.

[0082] Thus, as Figure 6 As shown, facial images can be acquired through a data acquisition module. A face detection and tracking module is used to detect and match target objects. A face registration and recognition module is used to bind identity information. An emotion perception module is used to determine the actual emotional state. A lip movement detection module is used to determine the actual lip state. This allows the interaction decision module to comprehensively understand user intent by integrating information from three dimensions: identity information, actual emotional state, and actual lip state. This avoids the limitations of a single modality and makes the interaction response more closely match the user's emotional state. The technical solution of this application can be flexibly deployed in multiple fields such as intelligent customer service, virtual assistants, medical companionship, and intelligent navigation. Figure 6 The modular design shown facilitates customization according to different scenario requirements, and has broad application prospects and commercial value.

[0083] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders.

[0084] like Figure 7 The diagram shown is a structural schematic of an interactive device 700 based on a face image provided in an embodiment of this application. The interactive device 700 based on a face image is configured on an electronic device.

[0085] Specifically, the face image-based interactive device 700 may include:

[0086] The prediction unit 701 is used to predict the expected position of each tracked object in the current frame based on the face images of historical frames;

[0087] The detection unit 702 is used to identify the emotional state and lip state of the target object in the current frame based on the face image of the current frame, and to determine the actual position of the target object in the face image of the current frame.

[0088] The matching unit 703 is used to match the actual position with the expected position of each tracked object, and bind the target object to the tracked object whose position is successfully matched.

[0089] The smoothing processing unit 704 is used to perform smoothing processing based on the emotional state and lip state of the current frame, and the emotional state and lip state of the tracked object bound to the target object in the historical frames, to obtain the actual emotional state and actual lip state of the target object.

[0090] The interaction unit 705 is used to execute interaction commands based on the actual emotional state and the actual lip state.

[0091] In some embodiments of this application, the smoothing processing unit 704 is specifically used to: determine the frequency of occurrence of each emotional state in the current frame and the emotional states in the historical frames; take the emotional states with an occurrence frequency greater than a frequency threshold as the actual emotional states; if there are no emotional states with an occurrence frequency greater than a frequency threshold, calculate the comprehensive confidence of each emotional state based on the confidence of the emotional state in the current frame and the confidence of the emotional states in the historical frames; and take the emotional state with the highest comprehensive confidence as the actual emotional state.

[0092] In some embodiments of this application, the smoothing processing unit 704 is specifically used to: for each emotional state, perform weighted fusion of the confidence scores corresponding to each frame, wherein the weight of the confidence scores corresponding to each frame is related to the time interval between the frame and the current frame.

[0093] In some embodiments of this application, the smoothing processing unit 704 is specifically used to: if the lip state of the current frame is a lip movement state, and the proportion of frames with lip movement states in the historical frames is greater than the proportion threshold, then the lip movement state is taken as the actual lip state; otherwise, the non-lip movement state is taken as the actual lip state.

[0094] In some embodiments of this application, the detection unit 702 is specifically used to: determine the lip movement amplitude of the current frame based on the face image; determine the amplitude difference between the lip movement amplitude of the current frame and the maximum lip movement amplitude in each historical frame; and if the amplitude difference is greater than the difference threshold, confirm that the current frame is in a lip movement state.

[0095] In some embodiments of this application, the matching unit 703 is further configured to: determine the number of consecutive lost frames of the unmatched tracking object after all objects in the face image of the current frame have been matched; if the number of consecutive lost frames is greater than the frame number threshold, store the face features of the unmatched tracking object in a preset feature library.

[0096] In some embodiments of this application, the matching unit 703 is further configured to: if the target object fails to match with each tracking object, match the facial features of the target object with the facial features of each tracking object in the preset feature library; and bind the target object to the tracking object whose features are successfully matched.

[0097] In some embodiments of this application, the face image-based interactive device 700 further includes an identity binding unit, specifically used for: performing feature matching between the face features of the target object and each known object in the face database, and binding the identity information of the target object with the known object whose features are successfully matched.

[0098] It should be noted that, for the sake of convenience and brevity, the specific working process of the above-mentioned face image-based interactive device 700 can be found in the following reference: Figures 1 to 6 The corresponding process of the method will not be described in detail here.

[0099] like Figure 8 The diagram shown is a schematic representation of an electronic device provided in an embodiment of this application. Specifically, the electronic device 8 may include: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as an interactive program based on a face image. When the processor 80 executes the computer program 82, it implements the steps in the various face image-based interactive method embodiments described above, for example... Figure 1 Steps S101 to S105 are shown. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 7 The functions of the prediction unit 701, detection unit 702, matching unit 703, smoothing unit 704, and interaction unit 705 shown are illustrated.

[0100] The computer program can be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0101] For example, the computer program can be divided into: a prediction unit, used to predict the expected position of each tracked object in the current frame based on the face images of historical frames; a detection unit, used to identify the emotional state and lip state of the target object in the current frame based on the face image of the current frame, and determine the actual position of the target object in the face image of the current frame; a matching unit, used to perform position matching between the actual position and the expected position of each tracked object, and bind the target object to the tracked object whose position is successfully matched; a smoothing processing unit, used to perform smoothing processing based on the emotional state and lip state of the current frame, and the emotional state and lip state of the tracked object bound to the target object in historical frames, to obtain the actual emotional state and actual lip state of the target object; and an interaction unit, used to execute interaction instructions based on the actual emotional state and actual lip state.

[0102] The electronic device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0103] The processor 80 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0104] The memory 81 can be an internal storage unit of the electronic device, such as a hard drive or memory. The memory 81 can also be an external storage device of the electronic device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 81 can include both internal and external storage units. The memory 81 is used to store the computer program and other programs and data required by the electronic device. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0105] It should be noted that, for the sake of convenience and brevity, the structure of the above-mentioned electronic device can also be referred to the specific description of the structure in the method embodiment, which will not be repeated here.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for various specific applications, but such implementations should not be considered beyond the scope of this application.

[0109] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0113] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A face image-based interaction method, characterized by, The method comprises the following steps: predicting expected positions of each tracking object in a current frame based on face images of historical frames; identifying an emotional state and a lip state of a target object in the current frame based on face images of the current frame, and determining an actual position of the target object in the face image of the current frame; position matching the actual position with the expected positions of each tracking object, and binding the target object with a tracking object that is successfully position matched; performing smoothing processing on the emotional state and the lip state of the current frame, and the emotional state and the lip state of the historical frame of the tracking object bound with the target object, to obtain an actual emotional state and an actual lip state of the target object; executing an interaction instruction based on the actual emotional state and the actual lip state. 2.The face image-based interaction method of claim 1, wherein, The method for smoothing processing the emotional state of the current frame and the emotional state of the historical frame of the tracking object bound with the target object to obtain an actual emotional state of the target object comprises: determining the frequency of each emotional state in the emotional state of the current frame and the emotional state of the historical frame; regarding an emotional state with a frequency greater than a frequency threshold as the actual emotional state; if there is no emotional state with a frequency greater than the frequency threshold, calculating a comprehensive confidence of each emotional state based on the confidence of the emotional state of the current frame and the confidence of the emotional state of the historical frame; regarding an emotional state with the highest comprehensive confidence as the actual emotional state. 3.The face image-based interaction method of claim 2, wherein, The method for calculating a comprehensive confidence of each emotional state based on the confidence of the emotional state of the current frame and the confidence of the emotional state of the historical frame comprises: for each emotional state, performing weighted fusion on the confidence of each frame, and the weight of the confidence of each frame is related to the time interval between the frame and the current frame. 4.The face image-based interaction method of claim 1, wherein, The method for smoothing processing the lip state of the current frame and the lip state of the historical frame of the tracking object bound with the target object to obtain an actual lip state of the target object comprises: if the lip state of the current frame is a lip movement state, and the proportion of the number of frames of the lip movement state in the historical frame is greater than a proportion threshold, regarding the lip movement state as the actual lip state; otherwise, regarding a non-lip movement state as the actual lip state. 5.The face image-based interaction method of claim 4, wherein, The method for identifying the lip state of the target object in the current frame based on the face image of the current frame comprises: determining a lip movement amplitude of the current frame based on the face image; determining an amplitude difference between the lip movement amplitude of the current frame and the maximum lip movement amplitude in each of the historical frames; if the amplitude difference is greater than a difference threshold, confirming that the current frame is in a lip movement state.

6. The face-image-based interaction method according to any one of claims 1 to 5, wherein The interaction method further comprises: after all objects in the face image of the current frame are matched, determining the number of consecutive lost frames of a tracking object that is not successfully matched; if the number of consecutive lost frames is greater than a frame number threshold, storing the face feature of the tracking object that is not successfully matched in a preset feature library. 7.The face image-based interaction method of claim 6, wherein, if the target object fails to be matched with each tracking object, the interaction method further comprises: matching the face feature of the target object with the face features of each tracking object in the preset feature library; binding the target object with a tracking object which is successfully matched with the feature.

8. The face-image-based interaction method according to any one of claims 1 to 5, wherein The interaction method further comprises: performing feature matching between the facial feature of the target object and each known object in a face library, and binding the target object with identity information of a known object which is successfully matched with the feature.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the interaction method based on a face image according to any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer program is executed to cause the interaction method based on a face image according to any one of claims 1 to 7 to be performed.

Images