A facial expression change anomaly recognition method based on time sequence feature modeling

Abstract

The application discloses a facial expression change abnormality recognition method based on time sequence feature modeling, and comprises the following steps: acquiring a continuous video sequence containing a face, generating a face key point time sequence; counting the displacement of each key point, determining an expression change starting area, and generating an expression shock source node set; constructing an expression shock source propagation time sequence, and generating an expression shock source propagation time field; connecting the intersection nodes of the shock source propagation path in the face key point space to form a multi-shock source interference time sequence structure; generating an interference stable time sequence structure from the interference area with unchanged node connection relationship; detecting the area with changed node connection relationship, and generating a minimum abnormal interference substructure; generating an expression abnormality mode, and outputting a facial expression change abnormality recognition result. The application utilizes the time sequence feature modeling method, realizes expression change structured abnormality recognition, and has the advantages of accurate abnormality positioning and high recognition stability.

Description

Technical Field

[0001] This invention relates to the field of facial expression anomaly recognition, and more particularly to a method for facial expression anomaly recognition based on temporal feature modeling. Background Technology

[0002] With the development of computer vision and face analysis technology, facial expression recognition methods based on video sequences have been gradually applied to fields such as behavior analysis, human-computer interaction, and security monitoring. Existing technologies typically acquire facial video sequences and extract facial key points, perform statistical analysis on the positional changes of key points in continuous time frames, and identify and classify expression changes through key point displacement features, expression feature vectors, or deep learning models. These methods usually rely on the magnitude of key point displacement, local region motion features, or overall feature vector changes to determine the expression state, thereby achieving the detection and recognition of facial expression changes.

[0003] Most existing methods focus on the analysis of changes in single key points or local features, lacking the ability to model the spatiotemporal propagation relationship and structural interference features between multiple key points during facial expression changes. They are unable to reflect the propagation path and structural evolution of facial expression changes in the facial key point space. At the same time, when there are local anomalies or complex interferences in facial expression changes, traditional methods are unable to accurately locate the abnormal change areas, resulting in insufficient accuracy in abnormal expression recognition. Therefore, how to model the propagation structure of facial expression changes in continuous time frames and extract abnormal interference features from structural evolution to improve the accuracy of abnormal expression recognition has become a technical problem that needs to be solved. Summary of the Invention

[0004] One objective of this invention is to propose a method for identifying facial expression anomalies based on temporal feature modeling. This invention utilizes temporal feature modeling to achieve structured anomaly identification of facial expression changes, and has the advantages of accurate anomaly localization and high identification stability.

[0005] A method for identifying facial expression anomalies based on temporal feature modeling according to an embodiment of the present invention includes the following steps: Obtain a continuous video sequence containing faces, and preprocess it to generate a temporal sequence of facial key points; The displacement of each key point in the temporal sequence of facial key points between adjacent time frames is calculated. The displacement of each key point is counted within a preset time window. The key point region with a displacement greater than a preset displacement threshold is determined as the expression change starting region. A set of expression source nodes is generated from the expression change starting region. Construct an expression source propagation time sequence based on the displacement sequence of the source nodes in the expression source node set in continuous time frames, and generate an expression source propagation time field based on the expression source propagation time sequence. Under the same time index, the source propagation paths in the facial source propagation time sequence field are superimposed, and the intersection nodes of the source propagation paths in the facial key point space are connected to form a multi-source interference time sequence structure. Within a continuous time window, the node connection relationships in the multi-source interferometric time sequence structure are statistically analyzed, and an interferometric stable time sequence structure is generated from the interferometric region where the node connection relationships remain unchanged. In the interferometric stable temporal structure, the region where the node connection relationship changes is detected, and the minimum anomalous interferometric substructure is generated from this region; Anomaly patterns in facial expressions are generated based on the node evolution sequence of the minimum anomaly interferometer substructure in continuous time frames, and the results of facial expression change anomaly recognition are output.

[0006] Optionally, the preprocessing specifically includes video frame extraction, face detection, face region cropping, face pose correction, image normalization, illumination normalization, facial key point localization, and time index sorting.

[0007] Optionally, the generation of the expression source node set specifically includes: The temporal sequence of facial key points is obtained. The two-dimensional coordinates of each facial key point are obtained in each time frame. The coordinate data of the same key point in consecutive time frames are extracted in chronological order. The coordinate difference of the same key point in adjacent time frames is calculated to generate the key point displacement sequence of each facial key point in consecutive time frames. The key point displacement sequence is cumulatively calculated within a preset time window. The displacement of each facial key point within the time window is summed to generate a cumulative key point displacement sequence within the time window. The displacement of each key point in the cumulative sequence of key point displacements is compared with a preset displacement threshold one by one. Key points with displacements greater than the preset displacement threshold are marked as expression change key points. The expression change key points are then connected according to the adjacency relationship in the facial key point space to generate the expression change starting region. Read the key point displacement sequence in the starting region of the facial expression change, calculate the displacement difference between key points in adjacent time frames within consecutive time frames, and determine key points whose displacement difference is less than a preset stable threshold as continuous key points of facial expression change, and generate a set of continuous key points of facial expression change. Based on the positional relationship of the set of continuous key points of facial expression changes in the facial key point space, adjacent continuous key points are divided into the same key point region to generate a stable region of facial expression change. All key points in the stable region of facial expression change are identified as facial expression source nodes, and a set of facial expression source nodes is formed from these nodes.

[0008] Optionally, the generation of the expression source propagation time series field specifically includes: The two-dimensional coordinates of the corresponding key points in the facial key point time sequence are arranged in the order of time frames according to the set of facial expression source nodes. The coordinates of each facial expression source node in the continuous time frame are organized to generate a time sequence set of source node coordinates. The coordinate difference of each facial expression source node in the time series set of source node coordinates is calculated in adjacent time frames, and the displacement change of each facial expression source node in consecutive time frames is recorded in the order of time frames to generate a time series sequence of source node displacement. The displacement amplitude of each source node in the source node displacement time series is calculated in continuous time frames, and the displacement amplitude changes in each time frame are recorded in chronological order to generate a source propagation intensity time series. Within the same time frame, spatial distances are calculated for the facial source nodes in the time series set of source node coordinates, and connection relationships between source nodes are established according to the spatial distances between nodes, generating a set of spatial connection relationships between source nodes; The source propagation intensity time series and the source node spatial connection relationship set are combined according to the time frame index. In each time frame, the source node propagation intensity and node connection relationship are uniformly organized to generate the source propagation time series. The propagation time sequence of facial expression sources is spatiotemporally organized in the space of facial key points according to the time frame order to generate the propagation time sequence field of facial expression sources.

[0009] Optionally, the generation of the multi-source interferometric time sequence structure specifically includes: In each time frame, the propagation trajectory of the facial expression source is constructed according to the spatial connection relationship between the source nodes. The spatial coordinates of the same facial expression source node in consecutive time frames are connected in time order to form the node propagation trajectory. The node propagation trajectories are organized according to the source node number to generate a set of source propagation trajectories. In the same time frame, the spatial distance of each source propagation trajectory in the source propagation trajectory set is calculated. The propagation trajectory of the source propagation trajectory whose Euclidean distance between the node coordinates in the same time frame is less than the preset propagation distance threshold is established as a propagation adjacency relationship. The adjacency relationship is recorded according to the node number to generate a source propagation adjacency relationship set. The set of adjacency relationships of earthquake source propagation and the set of earthquake source propagation trajectories are combined according to the time frame index. In the same time frame, the earthquake source propagation trajectories with adjacency relationships are overlaid. The earthquake source propagation trajectories participating in the overlay are recorded uniformly according to the node number to generate an earthquake source propagation path overlay set. In the set of superimposed propagation paths of earthquake sources, the spatial positions of nodes between propagation trajectories of each earthquake source are compared. Nodes with the same spatial coordinates or a spatial distance of less than a preset intersection threshold in the same time frame of different earthquake source propagation trajectories are identified as propagation intersection nodes. The propagation intersection nodes are numbered and organized according to the time frame order to generate a set of propagation intersection nodes. Record the node connection relationships of each node in the propagation intersection node set in consecutive time frames, and connect the intersection nodes that maintain the node connection relationship unchanged in consecutive time frames according to the spatial adjacency relationship to generate the interference node connection relationship set; The interference node connection relationships are organized in the facial key point space according to the time frame order to generate a multi-source interference time sequence structure.

[0010] Optionally, the generation of the interference-stable time sequence structure specifically includes: The multi-source interferometric time-series structure is unfolded in the order of time frames within a continuous time window, and the connection relationship of the interferometric nodes in each time frame is arranged according to the node number to generate a sequence of interferometric node connection relationships. The node connection relationships in the interference node connection relationship sequence are compared frame by frame between adjacent time frames. The node connection relationships that exist simultaneously in adjacent time frames are determined as the maintained connection relationships and recorded in the order of time frames to generate the node connection maintained sequence. Within a continuous time window, the node connection relationships in the node connection maintenance sequence are cumulatively recorded. The node connection relationships that exist in each time frame within the continuous time window are organized according to the node number to generate a stable set of node connection relationships. The node connection relationships in the stable node connection relationship set are organized according to the spatial position of the nodes in the facial key point space. Nodes that have node connection relationships and belong to the stable node connection relationship set are combined to generate a stable interference node region set. The stable interference node regions are arranged in the order of time frames in a continuous time frame. The spatial position of the node connection relationship in the facial key point space is combined with the time frame index to generate an interference stable temporal structure.

[0011] Optionally, the generation of the minimum anomalous interference substructure specifically includes: The stable temporal structure of the interference is unfolded in the order of time frames, and the node connection relationship in each time frame is arranged according to the node number to generate a stable interference node connection sequence. The node connection relationships in the stable interference node connection sequence are compared frame by frame between adjacent time frames. Node connection relationships with the same node number that exist in adjacent time frames are matched. Node connection relationships that exist in the current time frame but not in the previous time frame, as well as node connection relationships that exist in the previous time frame but not in the current time frame, are recorded to generate a node connection change sequence. The node connection relationships in the node connection change sequence are organized according to the node number. The node connection relationships that show node connection changes within a continuous time window are summarized to generate a set of abnormal node connection relationships. Arrange the node connection relationships in the abnormal node connection relationship set according to the spatial position of the nodes in the facial key point space, and combine the node connection relationships with the same node number to generate an abnormal node connection region set. The node connection relationships in the set of abnormal node connection regions are sorted according to the number of nodes. The node combination with the fewest nodes and the node connection change relationship is determined as the abnormal node connection substructure, and the minimum abnormal interference substructure is generated.

[0012] Optionally, the generation of the facial expression anomaly recognition result specifically includes: The smallest anomalous interference substructure is expanded in the order of time frames, and the connection relationship of the anomalous nodes in each time frame is arranged according to the node number to generate an anomalous interference node connection sequence. The node connection relationships in the abnormal interference node connection sequence are arranged in chronological order according to the node number in consecutive time frames, and the node connection relationships of the same node number in each time frame are recorded to generate an abnormal node evolution sequence. The node connection relationships in the abnormal node evolution sequence are compared frame by frame between adjacent time frames. Nodes with different node connection relationships in the current time frame and the previous time frame are recorded and arranged in the order of time frames to generate a node evolution change sequence. The nodes in the node evolution sequence are organized according to their node numbers. Nodes that show changes in node connections within a continuous time window are summarized to generate an abnormal node set. The nodes in the abnormal node set are arranged according to the spatial position of the nodes in the facial key point space. The nodes with spatial coordinate differences less than the preset spatial distance threshold are combined according to the node number to generate an abnormal expression pattern. The nodes in the abnormal expression pattern are arranged in the order of time frames, and the spatial coordinates of each node in the corresponding time frame are recorded to generate the abnormal expression change recognition result.

[0013] The beneficial effects of this invention are: This invention proposes a method for identifying facial expression anomalies based on temporal feature modeling. By processing continuous video sequences containing faces, the method extracts temporal sequences of facial key points and analyzes the displacement changes of key points in continuous time frames. This constructs a set of expression source nodes and a temporal field for expression source propagation, thereby achieving spatiotemporal modeling of the expression change propagation process. Compared with traditional methods that rely solely on single key point displacement or local region features for expression recognition, this invention starts from the propagation process of expression changes. By establishing the propagation path of expression sources and constructing a multi-source interference temporal structure in the facial key point space, the propagation relationship of expression changes in both time and space can be fully described. This allows for a more accurate reflection of the structural features in the real expression change process, improving the ability to depict complex expression change processes.

[0014] This invention, based on multi-source interferometric temporal structures, generates stable interferometric temporal structures by statistically analyzing the stability of node connections within continuous time windows. It then detects changes in node connections and extracts the smallest anomalous interferometric substructure from a structural evolution perspective, enabling precise localization of anomalous facial expression regions. By analyzing the evolutionary relationships of anomalous nodes across continuous time frames, it generates anomalous facial expression patterns and outputs anomalous facial expression recognition results. This allows anomalous facial expression recognition to not only rely on local motion features but also to comprehensively assess the interferometric relationships between facial expression propagation structures and changes in stable structures. Compared to existing technologies, this invention can more stably identify anomalous change regions in complex facial expression scenarios, improving the accuracy and stability of anomalous recognition. It also enhances the ability to express the structural features of facial expression changes, thereby improving the reliability and practicality of facial expression anomaly recognition. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a method for identifying abnormal facial expression changes based on temporal feature modeling proposed in this invention; Figure 2 This is the minimum anomaly interference substructure extraction diagram of a facial expression change anomaly recognition method based on temporal feature modeling proposed in this invention; Figure 3 This is a multi-source interference temporal structure diagram of a facial expression anomaly recognition method based on temporal feature modeling proposed in this invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0017] refer to Figures 1-3 A method for identifying facial expression anomalies based on temporal feature modeling includes the following steps: Obtain a continuous video sequence containing faces, and preprocess it to generate a temporal sequence of facial key points; The displacement of each key point in the temporal sequence of facial key points between adjacent time frames is calculated. The displacement of each key point is counted within a preset time window. The key point region with a displacement greater than a preset displacement threshold is determined as the expression change starting region. A set of expression source nodes is generated from the expression change starting region. Construct an expression source propagation time sequence based on the displacement sequence of the source nodes in the expression source node set in continuous time frames, and generate an expression source propagation time field based on the expression source propagation time sequence. Under the same time index, the source propagation paths in the facial source propagation time sequence field are superimposed, and the intersection nodes of the source propagation paths in the facial key point space are connected to form a multi-source interference time sequence structure. Within a continuous time window, the node connection relationships in the multi-source interferometric time sequence structure are statistically analyzed, and an interferometric stable time sequence structure is generated from the interferometric region where the node connection relationships remain unchanged. In the interferometric stable temporal structure, the region where the node connection relationship changes is detected, and the minimum anomalous interferometric substructure is generated from this region; Anomaly patterns in facial expressions are generated based on the node evolution sequence of the minimum anomaly interferometer substructure in continuous time frames, and the results of facial expression change anomaly recognition are output.

[0018] In this embodiment, the preprocessing specifically includes video frame extraction, face detection, face region cropping, face pose correction, image normalization, illumination normalization, facial key point localization, and time index sorting.

[0019] In this embodiment, the generation of the expression source node set specifically includes: The temporal sequence of facial key points is obtained. The two-dimensional coordinates of each facial key point are obtained in each time frame. The coordinate data of the same key point in consecutive time frames are extracted in chronological order. The coordinate difference of the same key point in adjacent time frames is calculated to generate the key point displacement sequence of each facial key point in consecutive time frames. The key point displacement sequence is cumulatively calculated within a preset time window. The displacement of each facial key point within the time window is summed to generate a cumulative key point displacement sequence within the time window. The displacement of each key point in the cumulative sequence of key point displacements is compared with a preset displacement threshold one by one. Key points with displacements greater than the preset displacement threshold are marked as expression change key points. The expression change key points are then connected according to the adjacency relationship in the facial key point space to generate the expression change starting region. Read the key point displacement sequence in the starting region of the facial expression change, calculate the displacement difference between key points in adjacent time frames within consecutive time frames, and determine key points whose displacement difference is less than a preset stable threshold as continuous key points of facial expression change, and generate a set of continuous key points of facial expression change. Based on the positional relationship of the set of continuous key points of facial expression changes in the facial key point space, adjacent continuous key points are divided into the same key point region to generate a stable region of facial expression change. All key points in the stable region of facial expression change are identified as facial expression source nodes, and a set of facial expression source nodes is formed from these nodes.

[0020] In this embodiment, the generation of the expression source propagation time series field specifically includes: The two-dimensional coordinates of the corresponding key points in the facial key point time sequence are arranged in the order of time frames according to the set of facial expression source nodes. The coordinates of each facial expression source node in the continuous time frame are organized to generate a time sequence set of source node coordinates. The coordinate difference of each facial expression source node in the time series set of source node coordinates is calculated in adjacent time frames, and the displacement change of each facial expression source node in consecutive time frames is recorded in the order of time frames to generate a time series sequence of source node displacement. The displacement amplitude of each source node in the source node displacement time series is calculated in continuous time frames, and the displacement amplitude changes in each time frame are recorded in chronological order to generate a source propagation intensity time series. Within the same time frame, spatial distances are calculated for the facial source nodes in the time series set of source node coordinates, and connection relationships between source nodes are established according to the spatial distances between nodes, generating a set of spatial connection relationships between source nodes; The source propagation intensity time series and the source node spatial connection relationship set are combined according to the time frame index. In each time frame, the source node propagation intensity and node connection relationship are uniformly organized to generate the source propagation time series. The propagation time sequence of facial expression sources is spatiotemporally organized in the space of facial key points according to the time frame order to generate the propagation time sequence field of facial expression sources.

[0021] In this embodiment, the generation of the multi-source interference time sequence structure specifically includes: In each time frame, the propagation trajectory of the facial expression source is constructed according to the spatial connection relationship between the source nodes. The spatial coordinates of the same facial expression source node in consecutive time frames are connected in time order to form the node propagation trajectory. The node propagation trajectories are organized according to the source node number to generate a set of source propagation trajectories. In the same time frame, the spatial distance of each source propagation trajectory in the source propagation trajectory set is calculated. The propagation trajectory of the source propagation trajectory whose Euclidean distance between the node coordinates in the same time frame is less than the preset propagation distance threshold is established as a propagation adjacency relationship. The adjacency relationship is recorded according to the node number to generate a source propagation adjacency relationship set. The set of adjacency relationships of earthquake source propagation and the set of earthquake source propagation trajectories are combined according to the time frame index. In the same time frame, the earthquake source propagation trajectories with adjacency relationships are overlaid. The earthquake source propagation trajectories participating in the overlay are recorded uniformly according to the node number to generate an earthquake source propagation path overlay set. In the set of superimposed propagation paths of earthquake sources, the spatial positions of nodes between propagation trajectories of each earthquake source are compared. Nodes with the same spatial coordinates or a spatial distance of less than a preset intersection threshold in the same time frame of different earthquake source propagation trajectories are identified as propagation intersection nodes. The propagation intersection nodes are numbered and organized according to the time frame order to generate a set of propagation intersection nodes. The generation of the propagation intersection node set specifically includes: The source propagation path superposition set is grouped according to time frame index. Source propagation trajectory nodes in the same time frame are arranged according to node number, generating a time frame node arrangement sequence. Within the same time frame, the spatial coordinates of nodes in the time frame node arrangement sequence are compared pairwise. For nodes with different node numbers, the difference in their spatial coordinates is calculated, and node pairs with a difference less than a preset intersection threshold are recorded, generating a candidate intersection node pair set. The node pairs in the candidate intersection node pair set are organized according to node number, and node pairs appearing in the same time frame are arranged according to time frame order, generating a time frame intersection node pair sequence. The node pairs in the time frame intersection node pair sequence are merged according to node number, and nodes with the same node number are uniformly numbered, generating an intersection node number sequence. The intersection node number sequence is organized according to time frame order, and nodes with the same number in the same time frame are grouped and recorded, generating a propagation intersection node set. Record the node connection relationships of each node in the propagation intersection node set in consecutive time frames, and connect the intersection nodes that maintain the node connection relationship unchanged in consecutive time frames according to the spatial adjacency relationship to generate the interference node connection relationship set; The generation of the set of interference node connection relationships specifically includes: The propagation intersection node set is expanded according to the time frame index, and the intersection nodes in the same time frame are arranged according to the node number to generate a time frame intersection node sequence. The node numbers in the time frame intersection node sequence are matched frame by frame between adjacent time frames. Intersection nodes that exist simultaneously in the current time frame and the previous time frame and have the same node number are recorded to generate an intersection node preservation sequence. The intersection nodes in the intersection node preservation sequence are organized according to the node number, and intersection nodes that exist in consecutive time frames are summarized to generate a stable intersection node set. The nodes in the stable intersection node set are paired according to their spatial positions in the facial keypoint space. Node pairs with a spatial distance less than a preset adjacency distance threshold are recorded to generate a candidate adjacency node pair set. The node pairs in the candidate adjacency node pair set are arranged according to the node number and organized according to the time frame order to generate an interference node connection relationship set. The interference node connection relationships are organized in the facial key point space according to the time frame order to generate a multi-source interference temporal structure; The generation of multi-source interferometric time-series structures specifically includes: The set of interferometric node connections is expanded according to the time frame index, and the node connections in the same time frame are arranged according to the node number to generate a time frame node connection sequence. The node connections in the time frame node connection sequence are grouped according to the node number, and the node connections with the same node number are arranged in chronological order in consecutive time frames to generate a node connection time sequence. The node connections in the node connection time sequence are combined in chronological order, and the node connections with the same node connection in adjacent time frames are continuously recorded to generate a continuous node connection sequence. The continuous node connection sequence is organized according to the node number, and the spatial positions of nodes with the same node connection in consecutive time frames are recorded to generate a node connection spatiotemporal sequence. The node connection spatiotemporal sequence is uniformly arranged in chronological order in the facial key point space, and the node connections are recorded in correspondence between spatial position and time frame index to generate a multi-source interferometric time sequence structure.

[0022] In this embodiment, the generation of the interference-stable time sequence structure specifically includes: The multi-source interferometric time-series structure is unfolded in the order of time frames within a continuous time window, and the connection relationship of the interferometric nodes in each time frame is arranged according to the node number to generate a sequence of interferometric node connection relationships. The node connection relationships in the interference node connection relationship sequence are compared frame by frame between adjacent time frames. The node connection relationships that exist simultaneously in adjacent time frames are determined as the maintained connection relationships and recorded in the order of time frames to generate the node connection maintained sequence. Within a continuous time window, the node connection relationships in the node connection maintenance sequence are cumulatively recorded. The node connection relationships that exist in each time frame within the continuous time window are organized according to the node number to generate a stable set of node connection relationships. The node connection relationships in the stable node connection relationship set are organized according to the spatial position of the nodes in the facial key point space. Nodes that have node connection relationships and belong to the stable node connection relationship set are combined to generate a stable interference node region set. The generation of the stable interference node region set specifically includes: The set of stable node connections is expanded according to node numbers. The two nodes in each connection are arranged according to their spatial coordinates in the facial keypoint space, generating a node connection coordinate sequence. In this sequence, the spatial coordinates of each node are compared pairwise. Connections where the Euclidean distance between their spatial coordinates is less than a preset adjacency distance threshold are recorded, generating a candidate set of adjacent node connections. The connections in this candidate set are then organized according to node numbers, and connections with common node numbers are combined, generating a node connection combination sequence. Within this combination sequence, nodes are grouped according to their spatial coordinates, and combinations of spatially continuous connections are merged, generating a node connection region sequence. Finally, the node connection region sequence is organized according to node numbers, and connections belonging to the same region are grouped and recorded, generating a set of stable interference node regions. The stable interference node regions are arranged in the order of time frames in a continuous time frame. The spatial position of the node connection relationship in the facial key point space is combined with the time frame index to generate an interference stable temporal structure. The generation of interferometric stable time series structures specifically includes: The set of stable interferometric node regions is expanded according to the time frame index. Stable interferometric node regions in the same time frame are arranged according to their region numbers to generate a time frame region sequence. Stable interferometric node regions in the time frame region sequence are matched in consecutive time frames according to their region numbers. Stable interferometric node regions that exist simultaneously in adjacent time frames and have the same region number are recorded to generate a region-preserving sequence. Stable interferometric node regions in the region-preserving sequence are arranged according to the time frame order. Stable interferometric node regions that maintain the same region number in consecutive time frames are organized to generate a stable interferometric region temporal sequence. Each stable interferometric node region in the stable interferometric region temporal sequence is recorded according to its node spatial coordinates. The node spatial coordinates are combined with the corresponding time frame index to generate a region spatiotemporal sequence. The region spatiotemporal sequence is organized uniformly according to the time frame order. The spatial positions of stable interferometric node regions in the facial key point space are matched with the time frame indexes and recorded to generate an interferometric stable temporal structure.

[0023] In this embodiment, the generation of the minimum anomalous interference substructure specifically includes: The stable temporal structure of the interference is unfolded in the order of time frames, and the node connection relationship in each time frame is arranged according to the node number to generate a stable interference node connection sequence. The node connection relationships in the stable interference node connection sequence are compared frame by frame between adjacent time frames. Node connection relationships with the same node number that exist in adjacent time frames are matched. Node connection relationships that exist in the current time frame but not in the previous time frame, as well as node connection relationships that exist in the previous time frame but not in the current time frame, are recorded to generate a node connection change sequence. The node connection relationships in the node connection change sequence are organized according to the node number. The node connection relationships that show node connection changes within a continuous time window are summarized to generate a set of abnormal node connection relationships. Arrange the node connection relationships in the abnormal node connection relationship set according to the spatial position of the nodes in the facial key point space, and combine the node connection relationships with the same node number to generate an abnormal node connection region set. The generation of the abnormal node connection region set specifically includes: The abnormal node connection set is expanded according to node number. Each node number in the connection is then matched with the corresponding facial key point spatial coordinates. The spatial coordinates of each node in the connection are recorded, generating an abnormal node connection coordinate sequence. Within this sequence, the spatial coordinates of each connection are compared one by one. Connections where the spatial distance between node coordinates is less than a preset adjacency distance threshold are recorded. Connections meeting the adjacency distance condition are then organized by node number, generating a candidate abnormal adjacent node connection set. The connections within this candidate abnormal adjacent node connection set are then... Relationships are categorized by node number. Node connections with the same node number are merged. Node connections with the same node number and adjacency distance are combined to generate an abnormal node connection combination sequence. Within this abnormal node connection combination sequence, nodes are spatially sorted by their spatial coordinates. Node connections that are spatially continuous and have node connections are merged according to their spatial adjacency to generate an abnormal node connection region sequence. Finally, the abnormal node connection region sequence is aggregated and organized by node number. Node connections belonging to the same spatial region and maintaining node connections are recorded uniformly to generate an abnormal node connection region set. The node connection relationships in the abnormal node connection region set are sorted according to the number of nodes. The node combination with the fewest nodes and the node connection change relationship is determined as the abnormal node connection substructure, and the minimum abnormal interference substructure is generated. The generation of the minimum anomalous interferometric substructure specifically includes: The set of abnormal node connection regions is expanded according to region numbers. The node connections within each abnormal node connection region are arranged according to node numbers. The node numbers and corresponding node connections in each connection relationship are recorded uniformly, generating an abnormal node connection relationship sequence. Within this sequence, the connections are counted according to node numbers. The number of nodes contained in each abnormal node connection region is recorded, and the node count information is recorded according to the region number, generating an abnormal node count record sequence. The abnormal node connection regions in this sequence are then sorted according to the number of nodes. The quantities are arranged in ascending order and recorded according to the region number to generate an abnormal node quantity sorting sequence. The node connection relationships in the abnormal node quantity sorting sequence are checked one by one. The node connection region with the fewest nodes that maintain the node connection change relationship is marked, and the node numbers and node connection relationships in the node connection region are organized to generate candidate abnormal node connection substructures. The node connection relationships in the candidate abnormal node connection substructures are uniformly organized according to the node number, and the spatial position of the node connection relationship in the facial key point space is matched with the corresponding node number to generate the minimum abnormal interference substructure.

[0024] In this embodiment, the generation of facial expression anomaly recognition results specifically includes: The smallest anomalous interference substructure is expanded in the order of time frames, and the connection relationship of the anomalous nodes in each time frame is arranged according to the node number to generate an anomalous interference node connection sequence. The node connection relationships in the abnormal interference node connection sequence are arranged in chronological order according to the node number in consecutive time frames, and the node connection relationships of the same node number in each time frame are recorded to generate an abnormal node evolution sequence. The node connection relationships in the abnormal node evolution sequence are compared frame by frame between adjacent time frames. Nodes with different node connection relationships in the current time frame and the previous time frame are recorded and arranged in the order of time frames to generate a node evolution change sequence. The nodes in the node evolution sequence are organized according to their node numbers. Nodes that show changes in node connections within a continuous time window are summarized to generate an abnormal node set. The nodes in the abnormal node set are arranged according to the spatial position of the nodes in the facial key point space. The nodes with spatial coordinate differences less than the preset spatial distance threshold are combined according to the node number to generate an abnormal expression pattern. The nodes in the abnormal expression pattern are arranged in the order of time frames, and the spatial coordinates of each node in the corresponding time frame are recorded to generate the abnormal expression change recognition result.

[0025] Example 1: To verify the feasibility of this invention in practice, it was applied to the process of recognizing abnormal facial expression changes in a smart interactive scenario. This scenario is set in a service environment with human-computer interaction capabilities. During daily operation, staff need to continuously monitor the facial expression changes of people entering the interaction area using video equipment to identify abnormal emotional changes or behavioral trends. This environment is located in the interaction area of ​​a city's public service hall. Fixed-position video capture devices are deployed on-site to continuously capture facial video sequences of people entering the interaction area. The video capture devices are installed at the hall entrance and interaction window areas, covering the range of facial movement and continuously recording continuous video footage. The video capture time covers the entire day's operation. During the video capture process, people will experience varying degrees of facial expression changes while queuing, communicating with staff, and confirming information. Traditional methods typically rely on key point displacement amplitude or expression classification models to recognize expressions. When facial expression changes are complex or local abnormal changes occur, unstable recognition or difficulty in accurately locating abnormal changes can easily occur. Therefore, a method is needed that can model the propagation structure of facial expression changes and identify abnormal structural changes.

[0026] In this scenario, a continuous video sequence containing faces is first acquired using a video capture device. The video sequence is then preprocessed to generate a temporal sequence of facial key points. During preprocessing, video frame extraction is performed, dividing the continuous video into image frames arranged chronologically. Within each image frame, face regions are located using face detection methods. The detected face regions are then cropped, and face pose correction is performed to ensure consistent face orientation within the image. Subsequently, image normalization and illumination normalization are applied to the cropped face images to reduce the impact of ambient lighting variations on subsequent key point localization. After image processing, facial key point localization is performed on the face regions within each time frame, obtaining the two-dimensional coordinates of each key point in the image. These key point coordinates are then sorted chronologically to generate the temporal sequence of facial key points.

[0027] After obtaining the temporal sequence of facial key points, the positional changes of each key point in the sequence between adjacent time frames are calculated. By comparing the coordinate differences of key points in consecutive time frames, a key point displacement sequence is generated. The key point displacements are statistically analyzed within a continuous time window. By accumulating the displacement changes of each key point, key points with significant displacement changes are identified as expression change key points. Based on the spatial adjacency relationship between facial key points, these key points are connected to form the expression change initiation region. In the further analysis of key point displacement changes, the stability of displacement changes in consecutive time frames is judged. Key point regions that maintain stable displacement changes in consecutive time frames are identified as expression change stable regions, and key points in these regions are identified as expression source nodes, thereby generating a set of expression source nodes.

[0028] After generating the set of facial expression source nodes, the spatial position changes of the source nodes in continuous time frames are organized, the coordinate changes of each source node in the time series are recorded and a source node displacement change sequence is formed, the connection relationship between nodes is established according to the positional relationship of the source nodes in the facial key point space, and the source node displacement change information and node spatial connection relationship are uniformly organized to form a facial expression source propagation time sequence. By organizing this sequence in the facial key point space, a facial expression source propagation time sequence field can be obtained. In this time sequence field, the propagation path of each source node in the time dimension and the spatial relationship between different source nodes can be clearly represented, thus reflecting the propagation process of facial expression changes in facial space.

[0029] Based on the temporal field of facial expression source propagation, spatial analysis is performed on the source propagation trajectories in the same time frame. Adjacency relationships are established for propagation trajectories that meet the spatial distance conditions, and propagation path superposition is performed within the time frame. When different source propagation trajectories intersect in the facial key point space, the intersecting nodes are recorded and their connection relationships in consecutive time frames are analyzed. Intersecting nodes with stable connection relationships are connected to form a set of interference node connection relationships. By organizing these connection relationships in the facial key point space according to the time sequence, a multi-source interference temporal structure can be constructed, enabling the interference relationship between multiple facial expression change sources to be expressed in both time and space dimensions.

[0030] After obtaining the multi-source interferometric time-series structure, the structure is unfolded within a continuous time window, and the changes in the node connection relationships in different time frames are analyzed. By comparing the node connection relationships in the continuous time frames, the node connection relationships that maintain stable connections are recorded and accumulated within the time window to generate a set of stable node connection relationships. Subsequently, the stable node connection relationships are organized according to the node positions in the facial key point space, and the node connection relationships belonging to the same spatial region are combined to form a set of stable interferometric node regions. This set is further organized in the time dimension to generate an interferometric stable time-series structure.

[0031] In the interferometric stable temporal structure, the changes in node connectivity in continuous time frames are further analyzed. Node connectivity with changes is recorded and organized to generate a set of anomalous node connectivity. By organizing the positions of anomalous node connectivity in the facial key point space, node connectivity with common node numbers is combined to form a set of anomalous node connectivity regions. Subsequently, the number of nodes in each anomalous node connectivity region is counted and sorted. The node combination with the fewest number of nodes and maintaining the node connectivity changes is determined as the anomalous node connectivity substructure, and thus the minimum anomalous interferometric substructure is generated.

[0032] After generating the minimum abnormal interference substructure, the node connection relationship of the substructure in continuous time frames is expanded. By recording the connection changes of each node in the time series, an abnormal node evolution sequence is formed. The connection changes in the node evolution sequence are further analyzed. The nodes with connection changes in the continuous time window are organized and an abnormal node set is generated. Then, based on the spatial position relationship of the nodes in the facial key point space, the spatially adjacent nodes are combined to form an expression abnormality pattern. Finally, by recording the spatial position changes and temporal order relationship of these nodes in continuous time frames, a complete expression change abnormality recognition result is generated.

[0033] In the above application, video data of the interactive area of ​​a city's public service hall was continuously collected at different times to identify and analyze the facial expression changes of people entering the area. During actual operation, various emotional change scenarios were recorded. By processing these continuous video data, the propagation path of facial expression changes in the key facial points and the interference structure formed between multiple sources can be observed. When abnormal emotional changes occur, the node connection relationship will change significantly in continuous time frames and form a minimum abnormal interference substructure in the stable interference structure, thereby accurately locating the abnormal change area. By processing video data in continuous time periods and recording the recognition process, the propagation structure of facial expression changes and the formation process of abnormal interference structures can be clearly observed, thus proving that the method of the present invention can stably identify abnormal facial expression changes in complex facial expression change scenarios and accurately locate the abnormal change area, demonstrating good application results.

[0034] Table 1. Performance Comparison Statistics of Facial Expression Anomaly Recognition Methods

[0035] As shown in Table 1, when different methods were statistically analyzed under the same video acquisition environment, the method of this invention showed more stable recognition results in multiple key indicators. In terms of anomaly recognition accuracy, the traditional key point displacement threshold method mainly relies on the displacement amplitude of key points for judgment, with an accuracy of 73.8%. The traditional method based on expression classification judges the expression as a whole through feature vectors, with an accuracy of 76.5%. After introducing temporal key point feature analysis, the recognition accuracy improved to 79.2%. In contrast, the method of this invention constructs a temporal field of expression source propagation and further establishes a multi-source interference temporal structure. During the recognition process, it can comprehensively analyze the propagation relationship of key points in the time and space dimensions, thereby improving the anomaly recognition accuracy to 84.6%. This result shows that introducing an expression change propagation structure during the recognition process can effectively improve the anomaly recognition capability.

[0036] Regarding the accuracy of abnormal area localization, traditional methods typically rely on the displacement amplitude of key points or changes in local areas. Therefore, when facial expressions involve multiple propagation areas or complex local changes, the localization results are prone to deviation. The data in the table shows that the accuracy of the key point displacement threshold method in abnormal area localization is 68.4%, the traditional facial expression classification method is 71.2%, and the temporal key point feature method is 74.9%. In contrast, the method of this invention achieves an accuracy of 82.3% in abnormal area localization under the same environmental conditions. This improvement is mainly due to the multi-source interference temporal structure constructed in this invention during the recognition process. By analyzing the intersection relationship of different source propagation paths, a stable interference region structure can be formed in the facial key point space, thereby more accurately locating the location of abnormal changes.

[0037] In terms of anomaly recognition stability, the method of this invention also shows a significant advantage. Traditional methods are easily affected by instantaneous facial expression changes or local key point jitter in continuous video sequences, causing the recognition results to fluctuate between different time frames. The data in the table shows that the recognition stability of the key point displacement threshold method is 70.6%, the traditional facial expression classification method is 72.8%, and the temporal key point feature method is 75.7%. The method of this invention achieves a stability of 81.5% under the same environment. This result is related to the introduction of an interference-stabilized temporal structure in this invention. The node connection relationship is statistically analyzed within a continuous time window, and a stable interference region is formed by stabilizing the node connection relationship, making the recognition process less susceptible to interference from instantaneous changes.

[0038] In terms of success rate in recognizing complex facial expressions, the method of this invention also demonstrates higher recognition capability. Complex facial expressions typically involve multiple facial regions changing simultaneously, such as the eyebrows, mouth, and eyes moving at the same time. Traditional methods struggle to accurately distinguish the relationships between different changing regions. The data in the table shows that the key point displacement threshold method achieves a recognition success rate of 66.9% in complex scenes, the traditional expression classification method achieves 69.4%, the temporal key point feature method achieves 72.6%, while the method of this invention reaches 80.1%. This improvement is mainly due to the fact that this invention forms a multi-source interference structure through propagation path superposition and propagation intersection node analysis during the recognition process, enabling the propagation relationship between multiple changing regions to be fully recorded, thereby enhancing the recognition capability in complex scenes.

[0039] Regarding the false recognition rate, the method of this invention also exhibits a lower probability of misjudgment. Traditional keypoint displacement methods are more sensitive to local changes, so they are prone to misjudgment when there are non-expression changes in the action, with a false recognition rate of 12.7%. Traditional expression classification methods judge based on overall features, with a false recognition rate of 11.3%, while the temporal keypoint feature method reduces it to 9.6%. The method of this invention analyzes the changes in node connection relationships and extracts the smallest abnormal interference substructure, so that the abnormal judgment is based on structural changes, thus reducing the false recognition rate to 7.8%.

[0040] In terms of continuous frame structure consistency rate, the method of the present invention also has obvious advantages. The continuous frame structure consistency rate reflects the stability of the structural relationship in the time series during the recognition process. Traditional methods usually lack structural modeling capabilities, so the indicators are 69.5%, 71.7% and 74.3%, respectively. The method of the present invention constructs the propagation time series field of the facial expression source and further forms an interference stable time series structure, so that the structural relationship between key points remains stable in continuous time frames, and thus the indicator reaches 83.2%.

[0041] As can be seen from the data in Table 1, the method of the present invention outperforms traditional methods in terms of anomaly recognition accuracy, anomaly region localization capability, recognition stability, and complex facial expression scene recognition capability. This performance improvement mainly stems from the present invention's approach of starting with the propagation structure of facial expression changes. By establishing a propagation time-series field of facial expression sources, a multi-source interference time-series structure, and an interference stability time-series structure, the propagation relationship of facial expression changes can be modeled in both time and space dimensions. Based on this, by extracting the minimum anomaly interference substructure, anomaly change regions can be identified from the perspective of structural evolution, thereby achieving more stable and accurate facial expression anomaly recognition.

[0042] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying facial expression anomalies based on temporal feature modeling, characterized in that, Includes the following steps: Obtain a continuous video sequence containing faces, and preprocess it to generate a temporal sequence of facial key points; The displacement of each key point in the temporal sequence of facial key points between adjacent time frames is calculated. The displacement of each key point is counted within a preset time window. The key point region with a displacement greater than a preset displacement threshold is determined as the expression change starting region. A set of expression source nodes is generated from the expression change starting region. Construct an expression source propagation time sequence based on the displacement sequence of the source nodes in the expression source node set in continuous time frames, and generate an expression source propagation time field based on the expression source propagation time sequence. Under the same time index, the source propagation paths in the facial source propagation time sequence field are superimposed, and the intersection nodes of the source propagation paths in the facial key point space are connected to form a multi-source interference time sequence structure. Within a continuous time window, the node connection relationships in the multi-source interferometric time sequence structure are statistically analyzed, and an interferometric stable time sequence structure is generated from the interferometric region where the node connection relationships remain unchanged. In the interferometric stable temporal structure, the region where the node connection relationship changes is detected, and the minimum anomalous interferometric substructure is generated from this region; Anomaly patterns in facial expressions are generated based on the node evolution sequence of the minimum anomaly interferometer substructure in continuous time frames, and the results of facial expression change anomaly recognition are output.

2. The method for identifying facial expression anomalies based on temporal feature modeling according to claim 1, characterized in that, The preprocessing specifically includes video frame extraction, face detection, face region cropping, face pose correction, image normalization, illumination normalization, facial key point localization, and time index sorting.

3. The method for identifying facial expression anomalies based on temporal feature modeling according to claim 1, characterized in that, The generation of the set of expression source nodes specifically includes: The temporal sequence of facial key points is obtained. The two-dimensional coordinates of each facial key point are obtained in each time frame. The coordinate data of the same key point in consecutive time frames are extracted in chronological order. The coordinate difference of the same key point in adjacent time frames is calculated to generate the key point displacement sequence of each facial key point in consecutive time frames. The key point displacement sequence is cumulatively calculated within a preset time window. The displacement of each facial key point within the time window is summed to generate a cumulative key point displacement sequence within the time window. The displacement of each key point in the cumulative sequence of key point displacements is compared with a preset displacement threshold one by one. Key points with displacements greater than the preset displacement threshold are marked as expression change key points. The expression change key points are then connected according to the adjacency relationship in the facial key point space to generate the expression change starting region. Read the key point displacement sequence in the starting region of the facial expression change, calculate the displacement difference between key points in adjacent time frames within consecutive time frames, and determine key points whose displacement difference is less than a preset stable threshold as continuous key points of facial expression change, and generate a set of continuous key points of facial expression change. Based on the positional relationship of the set of continuous key points of facial expression changes in the facial key point space, adjacent continuous key points are divided into the same key point region to generate a stable region of facial expression change. All key points in the stable region of facial expression change are identified as facial expression source nodes, and a set of facial expression source nodes is formed from these nodes.

4. The method for identifying facial expression anomalies based on temporal feature modeling according to claim 1, characterized in that, The generation of the expression source propagation time series field specifically includes: The two-dimensional coordinates of the corresponding key points in the facial key point time sequence are arranged in the order of time frames according to the set of facial expression source nodes. The coordinates of each facial expression source node in the continuous time frame are organized to generate a time sequence set of source node coordinates. The coordinate difference of each facial expression source node in the time series set of source node coordinates is calculated in adjacent time frames, and the displacement change of each facial expression source node in consecutive time frames is recorded in the order of time frames to generate a time series sequence of source node displacement. The displacement amplitude of each source node in the source node displacement time series is calculated in continuous time frames, and the displacement amplitude changes in each time frame are recorded in chronological order to generate a source propagation intensity time series. Within the same time frame, spatial distances are calculated for the facial source nodes in the time series set of source node coordinates, and connection relationships between source nodes are established according to the spatial distances between nodes, generating a set of spatial connection relationships between source nodes; The source propagation intensity time series and the source node spatial connection relationship set are combined according to the time frame index. In each time frame, the source node propagation intensity and node connection relationship are uniformly organized to generate the source propagation time series. The propagation time sequence of facial expression sources is spatiotemporally organized in the space of facial key points according to the time frame order to generate the propagation time sequence field of facial expression sources.

5. The method for identifying facial expression anomalies based on temporal feature modeling according to claim 1, characterized in that, The generation of the multi-source interferometric time-series structure specifically includes: In each time frame, the propagation trajectory of the facial expression source is constructed according to the spatial connection relationship between the source nodes. The spatial coordinates of the same facial expression source node in consecutive time frames are connected in time order to form the node propagation trajectory. The node propagation trajectories are organized according to the source node number to generate a set of source propagation trajectories. In the same time frame, the spatial distance of each source propagation trajectory in the source propagation trajectory set is calculated. The propagation trajectory of the source propagation trajectory whose Euclidean distance between the node coordinates in the same time frame is less than the preset propagation distance threshold is established as a propagation adjacency relationship. The adjacency relationship is recorded according to the node number to generate a source propagation adjacency relationship set. The set of adjacency relationships of earthquake source propagation and the set of earthquake source propagation trajectories are combined according to the time frame index. In the same time frame, the earthquake source propagation trajectories with adjacency relationships are overlaid. The earthquake source propagation trajectories participating in the overlay are recorded uniformly according to the node number to generate an earthquake source propagation path overlay set. In the set of superimposed propagation paths of earthquake sources, the spatial positions of nodes between propagation trajectories of each earthquake source are compared. Nodes with the same spatial coordinates or a spatial distance of less than a preset intersection threshold in the same time frame of different earthquake source propagation trajectories are identified as propagation intersection nodes. The propagation intersection nodes are numbered and organized according to the time frame order to generate a set of propagation intersection nodes. Record the node connection relationships of each node in the propagation intersection node set in consecutive time frames, and connect the intersection nodes that maintain the node connection relationship unchanged in consecutive time frames according to the spatial adjacency relationship to generate the interference node connection relationship set; The interference node connection relationships are organized in the facial key point space according to the time frame order to generate a multi-source interference time sequence structure.

6. The method for identifying facial expression anomalies based on temporal feature modeling according to claim 1, characterized in that, The generation of the interference-stable temporal structure specifically includes: The multi-source interferometric time-series structure is unfolded in the order of time frames within a continuous time window, and the connection relationship of the interferometric nodes in each time frame is arranged according to the node number to generate a sequence of interferometric node connection relationships. The node connection relationships in the interference node connection relationship sequence are compared frame by frame between adjacent time frames. The node connection relationships that exist simultaneously in adjacent time frames are determined as the maintained connection relationships and recorded in the order of time frames to generate the node connection maintained sequence. Within a continuous time window, the node connection relationships in the node connection maintenance sequence are cumulatively recorded. The node connection relationships that exist in each time frame within the continuous time window are organized according to the node number to generate a stable set of node connection relationships. The node connection relationships in the stable node connection relationship set are organized according to the spatial position of the nodes in the facial key point space. Nodes that have node connection relationships and belong to the stable node connection relationship set are combined to generate a stable interference node region set. The stable interference node regions are arranged in the order of time frames in a continuous time frame. The spatial position of the node connection relationship in the facial key point space is combined with the time frame index to generate an interference stable temporal structure.

7. The method for identifying facial expression anomalies based on temporal feature modeling according to claim 1, characterized in that, The generation of the minimum anomalous interference substructure specifically includes: The stable temporal structure of the interference is unfolded in the order of time frames, and the node connection relationship in each time frame is arranged according to the node number to generate a stable interference node connection sequence. The node connection relationships in the stable interference node connection sequence are compared frame by frame between adjacent time frames. Node connection relationships with the same node number that exist in adjacent time frames are matched. Node connection relationships that exist in the current time frame but not in the previous time frame, as well as node connection relationships that exist in the previous time frame but not in the current time frame, are recorded to generate a node connection change sequence. The node connection relationships in the node connection change sequence are organized according to the node number. The node connection relationships that show node connection changes within a continuous time window are summarized to generate a set of abnormal node connection relationships. Arrange the node connection relationships in the abnormal node connection relationship set according to the spatial position of the nodes in the facial key point space, and combine the node connection relationships with the same node number to generate an abnormal node connection region set. The node connection relationships in the set of abnormal node connection regions are sorted according to the number of nodes. The node combination with the fewest nodes and the node connection change relationship is determined as the abnormal node connection substructure, and the minimum abnormal interference substructure is generated.

8. The method for identifying facial expression anomalies based on temporal feature modeling according to claim 1, characterized in that, The generation of the facial expression anomaly recognition result specifically includes: The smallest anomalous interference substructure is expanded in the order of time frames, and the connection relationship of the anomalous nodes in each time frame is arranged according to the node number to generate an anomalous interference node connection sequence. The node connection relationships in the abnormal interference node connection sequence are arranged in chronological order according to the node number in consecutive time frames, and the node connection relationships of the same node number in each time frame are recorded to generate an abnormal node evolution sequence. The node connection relationships in the abnormal node evolution sequence are compared frame by frame between adjacent time frames. Nodes with different node connection relationships in the current time frame and the previous time frame are recorded and arranged in the order of time frames to generate a node evolution change sequence. The nodes in the node evolution sequence are organized according to their node numbers. Nodes that show changes in node connections within a continuous time window are summarized to generate an abnormal node set. The nodes in the abnormal node set are arranged according to the spatial position of the nodes in the facial key point space. The nodes with spatial coordinate differences less than the preset spatial distance threshold are combined according to the node number to generate an abnormal expression pattern. The nodes in the abnormal expression pattern are arranged in the order of time frames, and the spatial coordinates of each node in the corresponding time frame are recorded to generate the abnormal expression change recognition result.

Images