Expression information acquisition and expression recognition system

Abstract

The invention relates to the technical field of expression recognition, in particular to an expression information acquisition and expression recognition system, which comprises a motion field construction module, an expression manifold fitting module, a structure track construction module, a semantic region division module, a track standardization module and an expression reconstruction module. Pixel motion vectors are obtained, and a multi-resolution space-time motion field is constructed; performing curved surface fitting on the high-dimensional feature points to obtain a personalized expression manifold; projecting the time sequence change sequence to the personalized expression manifold to obtain a structured expression trajectory of the target object; performing region division on the personalized expression manifold to obtain an expression semantic region; performing bidirectional mapping on the structured expression trajectory and the personalized expression manifold to obtain a standard expression trajectory; performing iterative optimization on the personalized expression manifold, and reconstructing a structured expression trajectory; according to the invention, the accuracy of expression information collection and expression recognition can be improved.

Description

Technical Field

[0001] This invention relates to the field of facial expression recognition technology, and in particular to a facial expression information acquisition and facial expression recognition system. Background Technology

[0002] Existing facial expression information acquisition and recognition technologies struggle to accurately capture multi-scale features of facial movement when processing facial image sequences. They are unable to effectively separate changes in the overall facial contour, local muscle group movements, and subtle texture disturbances, resulting in insufficient completeness and accuracy of the extracted pixel motion vectors. This makes subsequent expression analysis based on motion features lack reliable data support, making it difficult to fully reflect the true dynamic process of expressions, and thus affecting the basic accuracy of expression recognition.

[0003] Meanwhile, existing technologies lack personalized adaptation mechanisms for individual facial features, and mostly use generalized expression models for analysis, which cannot adapt to the differences in facial geometric features of different individuals. Furthermore, in the process of constructing and standardizing expression trajectories, the distribution characteristics of expression semantic regions are not fully combined for accurate mapping and optimization, resulting in poor consistency and standardization of expression trajectories. They are easily affected by individual differences, environmental noise and other factors, ultimately causing unstable accuracy of expression recognition, which is difficult to meet the needs of high-precision application scenarios. Therefore, how to improve the accuracy of expression recognition has become an urgent problem to be solved. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides an expression information acquisition and expression recognition system, characterized in that the system includes a motion field construction module, an expression manifold fitting module, a structural trajectory construction module, a semantic region segmentation module, a trajectory standardization module, and an expression reconstruction module, wherein: The motion field construction module is used to perform dense optical flow analysis on the facial image sequence of the target object to obtain the pixel motion vector of the target object, and decompose the pixel motion vector into multi-scale motion components to construct the multi-resolution spatiotemporal motion field of the target object. The facial expression manifold fitting module is used to perform surface fitting on high-dimensional feature points in the multi-resolution spatiotemporal motion field to obtain the personalized facial expression manifold of the target object. The structure trajectory construction module is used to project the temporal change sequence in the multi-resolution spatiotemporal motion field onto the personalized expression manifold to obtain the projection point sequence of the target object, and connect the projection point sequence to obtain the structured expression trajectory of the target object; The semantic region segmentation module is used to segment the personalized expression manifold into regions based on the geometric feature distribution of the personalized expression manifold, thereby obtaining the expression semantic region of the personalized expression manifold. The trajectory standardization module is used to map the structured facial expression trajectory onto the facial expression semantic region to obtain the semantic projection path of the target object, and to back-map the semantic projection path onto the personalized facial expression manifold to obtain the standard facial expression trajectory of the target object; The expression reconstruction module is used to iteratively optimize the personalized expression manifold based on the geometric consistency between the standard expression trajectory and the structured expression trajectory, and reconstruct the structured expression trajectory according to the optimized expression manifold.

[0005] In a preferred embodiment, when the motion field construction module performs dense optical flow analysis on the facial image sequence of the target object to obtain the pixel motion vector of the target object, and decomposes the pixel motion vector into multi-scale motion components to construct the multi-resolution spatiotemporal motion field of the target object, it is specifically used for: A sequence of facial images of the target object is acquired, and global motion compensation is performed on consecutive image frames in the facial image sequence to obtain a stable image sequence of the target object. Extract the pixel displacement distribution from the stable image sequence to obtain the initial two-dimensional displacement field of the target object; The initial two-dimensional displacement field is decomposed into multiple levels to obtain the overall facial contour changes, local muscle group movements, and subtle texture perturbations of the target object. By integrating the overall facial contour changes, the local muscle group movements, and the subtle texture perturbations, the motion components of the target object are obtained. The motion components are subjected to temporal smoothing to obtain the spatiotemporal motion representation of the target object; The motion components in the spatiotemporal motion representation are hierarchically constructed to obtain the multi-resolution spatiotemporal motion field of the target object.

[0006] In a preferred embodiment, when the expression manifold fitting module performs surface fitting on high-dimensional feature points in the multi-resolution spatiotemporal motion field to obtain the personalized expression manifold of the target object, it is specifically used for: Geometric decomposition of the multi-resolution spatiotemporal motion field yields a high-dimensional set of motion feature points of the multi-resolution spatiotemporal motion field. The high-dimensional motion feature point set is reconstructed to obtain the local neighborhood structure of the high-dimensional motion feature point set; Principal component analysis is performed on the local neighborhood structure to obtain the local tangent space of the high-dimensional motion feature point set; Based on the positional relationships between feature points in the high-dimensional motion feature point set, the local tangent space is stitched together to form the initial global manifold of the target object; Based on the projection distribution of the high-dimensional motion feature point set onto the initial global manifold, the curvature and dimension of the initial global manifold are adaptively adjusted to obtain the personalized expression manifold of the target object.

[0007] In a preferred embodiment, when the structure trajectory construction module projects the temporal change sequence in the multi-resolution spatiotemporal motion field onto the personalized facial expression manifold to obtain the projection point sequence of the target object, and connects the projection point sequence to obtain the structured facial expression trajectory of the target object, it is specifically used for: The multi-resolution spatiotemporal motion field is segmented by a sliding window to obtain temporal motion segments of the multi-resolution spatiotemporal motion field; Extract the local manifold region corresponding to the overall motion pattern in the temporal motion segment from the personalized expression manifold; The temporal motion segment is mapped onto the center point of the local region of the manifold to obtain the initial projection point of the target object; The initial projection points are sequentially arranged to obtain the projection point sequence of the target object; Based on the affine connection structure of the personalized expression manifold, the projection points in the projection point sequence are geometrically connected to obtain the geodesic line segment of the target object; The geodesic segments are smoothly spliced ​​together to obtain the initial facial expression trajectory of the target object; By applying geometric constraints to the initial facial expression trajectory, a structured facial expression trajectory of the target object is obtained.

[0008] In a preferred embodiment, when the semantic region segmentation module performs region segmentation on the personalized expression manifold based on the geometric feature distribution of the personalized expression manifold to obtain the expression semantic region of the personalized expression manifold, it is specifically used for: The curvature distribution, normal vector direction, and local neighborhood shape of the personalized expression manifold are used as feature descriptions of the personalized expression manifold. Based on the spatial distribution described by the features, hierarchical clustering is performed on the personalized expression manifold to obtain the initial geometric region of the personalized expression manifold; Boundary consistency analysis is performed on the initial geometric region to obtain the candidate semantic regions of the personalized expression manifold; Assign expression category identifiers to the candidate semantic regions to obtain the expression semantic regions of the personalized expression manifold.

[0009] In a preferred embodiment, when the trajectory standardization module performs the mapping of the structured facial expression trajectory to the facial expression semantic region to obtain the semantic projection path of the target object, it is specifically used for: Obtain the local geometric descriptor of the sampling points on the structured facial expression trajectory; Based on the local geometric descriptor, a geometric similarity assessment is performed on the structured facial expression trajectory and the facial expression semantic region to obtain the degree of matching between the structured facial expression trajectory and the facial expression semantic region; Based on the matching degree, a corresponding facial expression semantic region is assigned to the collection point to construct a point-region attribution mapping table for the target object; By connecting the semantic region center points in the point-region attribution mapping table in chronological order, the semantic state transition path of the target object is obtained. Abnormal states are removed from the semantic state transition path to obtain the semantic projection path of the target object.

[0010] In a preferred embodiment, when the trajectory standardization module performs the operation of back-mapping the semantic projection path onto the personalized expression manifold to obtain the standard expression trajectory of the target object, it is specifically used for: Based on the continuous semantic state segments on the semantic projection path, corresponding feature points are selected from the facial expression semantic region to obtain a reference feature point sequence for the facial expression semantic region; Based on the personalized expression manifold, trajectory fitting is performed on the reference feature point sequence to obtain the smooth parametric curve of the target object; By applying dynamic constraints to the smooth parameter curve, the optimized parameter curve of the target object is obtained; The optimized parameter curve is reparameterized to obtain the standard facial expression trajectory of the target object.

[0011] In a preferred embodiment, when the expression reconstruction module performs iterative optimization of the personalized expression manifold based on the geometric consistency between the standard expression trajectory and the structured expression trajectory, and reconstructs the structured expression trajectory according to the optimized expression manifold, it is specifically used for: Based on the personalized facial expression manifold, the standard facial expression trajectory and the structured facial expression trajectory are sampled synchronously to obtain the trajectory point pair set of the target object; Geometric attribute comparison is performed on the trajectory point pairs in the trajectory point pair set to obtain the scalar deviation of the trajectory point pair set; Based on the spatial distribution of the scalar deviation, construct the deviation field of the trajectory point pair set; Sub-regions with high deviation values ​​and spatial continuity are identified from the deviation field and used as the geometric misalignment regions of the target object. A correlation analysis is performed on the geometrically misaligned region and the structured facial expression trajectory to obtain the degree of influence of the geometrically misaligned region on the structured facial expression trajectory; Based on the degree of influence, the personalized facial expression manifold is collaboratively adjusted to obtain the adjusted facial expression manifold of the target object; Based on the adjusted facial expression manifold, the geometric properties of the standard facial expression trajectory and the structured facial expression trajectory are re-evaluated to construct a new deviation field for the target object; By comparing and analyzing the deviation intensity of the deviation field and the deviation intensity of the new deviation field, the changes in the deviation field and the new deviation field are obtained. When the change is less than a preset stability threshold, the adjusted facial expression manifold is determined to be the optimized facial expression manifold of the target object.

[0012] In a preferred embodiment, the formula for calculating the scalar deviation is: ; in, This indicates the scalar deviation. Let A represent the first curvature of the trajectory point A. Let A represent the second curvature of the trajectory point A. This represents the first curvature of the trajectory point B. This represents the second curvature of the trajectory point B. This represents the unit normal vector of trajectory point A. This represents the unit normal vector of trajectory point B. This represents the geodesic distance between trajectory point A and trajectory point B. This indicates the preset curvature difference adjustment parameter. This represents the preset weight of the normal vector difference term. This indicates the degree of nonlinearity of the preset curvature difference.

[0013] In a preferred embodiment, when the expression reconstruction module reconstructs the structured expression trajectory based on the optimized expression manifold, it is specifically used for: The semantic projection path of the target object is mapped onto the optimized facial expression manifold to obtain the optimized projection path of the target object; Dense sampling is performed on the optimized projection path to obtain the key point sequence of the optimized projection path; Based on the geometric neighborhood information of the optimized facial expression manifold, local structure analysis is performed on the key point sequence to obtain the original motion components of the key point sequence. The original motion components are serialized and integrated to obtain the motion sequence of the target object; Based on the geometric features of the structured facial expression trajectory, the motion sequence is spatially reconstructed to obtain the reconstructed structured facial expression trajectory of the target object.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention significantly improves the accuracy of facial expression information acquisition through multi-module collaborative operation; the motion field construction module, through dense optical flow analysis and multi-scale motion component decomposition, can comprehensively capture changes in the overall facial contour, local muscle group movements, and subtle texture disturbances. Combined with the multi-resolution spatiotemporal motion field constructed by temporal smoothing processing, it provides a rich and accurate spatiotemporal motion foundation for facial expression information acquisition; the facial expression manifold fitting module, through surface fitting and adaptive adjustment of high-dimensional feature points, forms a personalized facial expression manifold that fits the target object, ensuring that the acquired facial expression features are specific and complete, thus improving the quality of facial expression information acquisition from the source.

[0015] 2. This invention projects a temporal change sequence onto a personalized facial expression manifold to form a structured facial expression trajectory. Combined with the semantic region segmentation module's precise partitioning based on geometric feature distribution, it establishes a clear semantic reference for facial expression recognition. The bidirectional mapping of the trajectory standardization module and the iterative optimization of the facial expression reconstruction module, through geometric consistency verification and deviation correction, continuously optimize the facial expression manifold and trajectory structure, making the reconstructed facial expression trajectory more closely match the real facial expression change patterns, thus significantly improving the reliability and accuracy of facial expression recognition. Attached Figure Description

[0016] Figure 1 This is a system architecture diagram of an expression information acquisition and expression recognition system provided in an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments belong to some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “the” and “the” as used in the embodiments of this invention are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0019] Depending on the context, the word "if" or "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0020] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0021] In practice, the server-side equipment deployed in an expression information collection and recognition system may consist of one or more devices. This expression information collection and recognition system can be implemented as a business instance, a virtual machine, or hardware devices. For example, it can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, it can be understood as software deployed on a cloud node, providing expression information collection and recognition services to various user terminals. Alternatively, it can be implemented as a virtual machine deployed on one or more devices in a cloud node, with application software installed to manage each user terminal. Or, it can also be implemented as a server composed of numerous identical or different types of hardware devices, with one or more hardware devices configured to provide expression information collection and recognition services to various user terminals.

[0022] In terms of implementation, the facial expression information collection and recognition system and the user terminal are mutually compatible. That is, if the facial expression information collection and recognition system is implemented as an application installed on a cloud service platform, then the user terminal is a client that establishes a communication connection with the application; or if the facial expression information collection and recognition system is implemented as a website, then the user terminal is implemented as a webpage; or if the facial expression information collection and recognition system is implemented as a cloud service platform, then the user terminal is implemented as a mini-program in an instant messaging application.

[0023] like Figure 1 The diagram shown is a system architecture diagram of an expression information acquisition and expression recognition system provided in an embodiment of the present invention.

[0024] The facial expression information acquisition and recognition system 100 described in this invention can be located on a cloud server. In terms of implementation, it can be used as one or more service devices, or as an application installed on the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed as a website. Depending on the functions implemented, the facial expression information acquisition and recognition system 100 may include a motion field construction module 101, an facial expression manifold fitting module 102, a structural trajectory construction module 103, a semantic region segmentation module 104, a trajectory standardization module 105, and an facial expression reconstruction module 106. The modules described in this invention can also be called units, referring to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0025] In this embodiment of the invention, in an expression information acquisition and recognition system, each of the above-mentioned modules can be implemented independently and can call other modules. Here, "calling" can be understood as a module connecting to multiple modules of another type and providing corresponding services to those connected modules. In the expression information acquisition and recognition system provided by this embodiment of the invention, without modifying the program code, the applicability of the expression information acquisition and recognition system architecture can be adjusted by adding modules and directly calling them, achieving cluster-based horizontal expansion to quickly and flexibly expand the expression information acquisition and recognition system. In practical applications, the above modules can be set in the same device or different devices, or they can be set in a virtual device, such as a service instance in a cloud server.

[0026] The following describes, with reference to specific embodiments, each component and specific workflow of an expression information acquisition and expression recognition system: The motion field construction module 101 is used to perform dense optical flow analysis on the facial image sequence of the target object to obtain the pixel motion vector of the target object, and decompose the pixel motion vector into multi-scale motion components to construct the multi-resolution spatiotemporal motion field of the target object. In this embodiment of the invention, when the motion field construction module performs dense optical flow analysis on the facial image sequence of the target object to obtain the pixel motion vector of the target object, and decomposes the pixel motion vector into multi-scale motion components to construct the multi-resolution spatiotemporal motion field of the target object, it is specifically used for: A sequence of facial images of the target object is acquired, and global motion compensation is performed on consecutive image frames in the facial image sequence to obtain a stable image sequence of the target object. Extract the pixel displacement distribution from the stable image sequence to obtain the initial two-dimensional displacement field of the target object; The initial two-dimensional displacement field is decomposed into multiple levels to obtain the overall facial contour changes, local muscle group movements, and subtle texture perturbations of the target object. By integrating the overall facial contour changes, the local muscle group movements, and the subtle texture perturbations, the motion components of the target object are obtained. The motion components are subjected to temporal smoothing to obtain the spatiotemporal motion representation of the target object; The motion components in the spatiotemporal motion representation are hierarchically constructed to obtain the multi-resolution spatiotemporal motion field of the target object.

[0027] The target subject's face is continuously photographed using a high-definition camera, capturing all image frames during the subject's natural facial expression changes. These image frames are arranged in chronological order of shooting time to form a complete facial image sequence. The global displacement caused by factors such as head shaking and slight movement of the shooting device in the continuous image frames is analyzed. The positional changes of the background and the overall face in the image are compared frame by frame to determine the offset of each image relative to a preset reference frame. Based on these offsets, each image is adjusted by reverse translation, rotation, or scaling to eliminate interference caused by global motion and ensure that the position of the facial area remains stable in the image sequence. Finally, a stable image sequence with no additional displacement interference in the facial area and clear facial expression changes is obtained.

[0028] The positions of all pixels in two adjacent frames of a stable image sequence are compared one by one. Based on the pixel coordinates of each frame, the position difference of each pixel in the previous frame and the position in the next frame is recorded. The horizontal and vertical movement distance and direction of each pixel are determined. All these movement information of all pixels are organized and arranged according to their coordinate positions in the image to form an initial two-dimensional displacement field that can completely reflect the displacement of each pixel in the two-dimensional plane.

[0029] First, large-scale displacement information affecting the entire facial region is separated from the initial two-dimensional displacement field. This displacement information covers all areas of the face and corresponds to changes in the overall facial contour caused by rotation and translation. Then, medium-scale displacement information concentrated in specific areas such as the eyes, corners of the mouth, and eyebrows is extracted from the remaining displacement information. This displacement information only acts on specific local areas and corresponds to the movement caused by the contraction or relaxation of local muscle groups. Finally, displacement information with extremely small amplitude and scattered distribution is filtered from the remaining displacement information. This displacement information corresponds to the subtle textural disturbances such as the movement of fine lines on the facial skin caused by changes in facial expressions.

[0030] The displacement information corresponding to the overall facial contour changes, the displacement information corresponding to the local muscle group movements, and the displacement information corresponding to the subtle texture disturbances are comprehensively summarized. All displacement data corresponding to each type of movement are checked one by one to ensure that no pixel movement information is missed. The three types of displacement information are fused into a complete set, and finally a motion component containing displacement data corresponding to all types of facial movements is formed.

[0031] Following the shooting time sequence of facial image sequences, the changes in the displacement information of each pixel in the motion component are analyzed sequentially between consecutive frames. The fluctuations of the displacement data of each pixel in the time dimension are observed. For those abnormal displacement data that suddenly appear and do not conform to the displacement change pattern of the preceding and following frames, the abnormal data are corrected by referring to the displacement situation in the adjacent frames before and after the pixel, so that the displacement data of each pixel presents a continuous and stable change trend in the time dimension, while fully preserving the reasonable displacement fluctuations brought about by the expression change itself. Finally, a spatiotemporal motion representation that can simultaneously reflect the spatial distribution and temporal change pattern of pixel displacement is obtained.

[0032] Based on the scale and amplitude of the motion components, the motion components in the spatiotemporal motion representation are divided into three different levels. The motion components corresponding to changes in the overall facial contour are classified as the first level, reflecting the large-scale motion of the face; the motion components corresponding to the motion of local muscle groups are classified as the second level, reflecting the motion of local facial areas; and the motion components corresponding to subtle texture perturbations are classified as the third level, reflecting the motion of subtle facial details. The motion components of these three levels are organically combined according to their respective motion characteristics and spatiotemporal distribution patterns, so that the motion information of different levels complements each other and does not interfere with each other, ultimately forming a multi-resolution spatiotemporal motion field that can comprehensively reflect the facial motion of the target object from different scales and levels.

[0033] The beneficial effects of this implementation process are that it ensures the acquisition of stable image sequences through precise image acquisition and global motion compensation, laying the foundation for subsequent data processing. By comparing pixels one by one and decomposing hierarchically, it accurately extracts different types of facial motion information. The integrated motion components are comprehensive and accurate, and temporal smoothing ensures the continuity and rationality of motion data. The hierarchically constructed multi-resolution spatiotemporal motion field fully presents the motion of each level of the face, providing accurate, comprehensive, and stable motion data support for subsequent modules such as expression manifold fitting and structured expression trajectory construction. This effectively improves the completeness and accuracy of expression information acquisition, thus providing a reliable guarantee for improving the accuracy of expression recognition.

[0034] The facial expression manifold fitting module 102 is used to perform surface fitting on high-dimensional feature points in the multi-resolution spatiotemporal motion field to obtain the personalized facial expression manifold of the target object. In this embodiment of the invention, when the expression manifold fitting module performs surface fitting on high-dimensional feature points in the multi-resolution spatiotemporal motion field to obtain the personalized expression manifold of the target object, it is specifically used for: Geometric decomposition of the multi-resolution spatiotemporal motion field yields a high-dimensional set of motion feature points of the multi-resolution spatiotemporal motion field. The high-dimensional motion feature point set is reconstructed to obtain the local neighborhood structure of the high-dimensional motion feature point set; Principal component analysis is performed on the local neighborhood structure to obtain the local tangent space of the high-dimensional motion feature point set; Based on the positional relationships between feature points in the high-dimensional motion feature point set, the local tangent space is stitched together to form the initial global manifold of the target object; Based on the projection distribution of the high-dimensional motion feature point set onto the initial global manifold, the curvature and dimension of the initial global manifold are adaptively adjusted to obtain the personalized expression manifold of the target object.

[0035] The motion information corresponding to the overall facial contour changes, local muscle group movements, and subtle texture perturbations contained in the multi-resolution spatiotemporal motion field is comprehensively decomposed. Key position points that can represent the core features of each type of motion are extracted one by one. These key position points include the turning points of the facial contour, the core action points of local muscle group movements, and the significant feature points of subtle texture perturbations. All extracted key position points are classified and organized according to their attributes in motion to form a high-dimensional motion feature point set that can completely cover all the core motion features of the multi-resolution spatiotemporal motion field.

[0036] For each feature point in the high-dimensional motion feature point set, a fixed spatial range is defined with that feature point as the center. All other feature points located within this spatial range are selected from the high-dimensional motion feature point set and identified as the neighboring feature points of the central feature point. The linear distance, relative orientation, and other positional relationships between the central feature point and each neighboring feature point are recorded in detail, as well as the mutual positional associations between these neighboring feature points. Based on the positional relationships and associations of each central feature point and its neighboring feature points, a local domain structure is constructed with a single feature point as the core and including surrounding associated feature points, ensuring that each feature point has a corresponding local domain structure to reflect its local spatial distribution.

[0037] For each local domain structure, a comprehensive analysis is conducted on the spatial distribution of all feature points within it. The main directions that dominate the distribution trend of feature points in that local domain are identified. These directions are those where the differences in feature point distribution are most obvious and contain the richest information. Based on these main directions, a low-dimensional spatial framework is constructed. This framework can best fit the actual distribution of all feature points in that local domain and accurately reflect the spatial structural characteristics of the local domain. This low-dimensional spatial framework that fits the local domain structure is called the local tangent space.

[0038] A comprehensive analysis of the overall spatial relationships of all feature points in the high-dimensional motion feature point set is conducted, clarifying the relative positions, adjacent boundaries, and overlapping regions of the local tangent spaces corresponding to different feature points in the overall space. Based on these clear positional relationships, the various local tangent spaces are connected accordingly. For the overlapping regions of adjacent local tangent spaces, it is ensured that the structures of the two spaces can transition smoothly without any breaks or conflicts. All interconnected local tangent spaces are gradually integrated to form a global spatial form that can completely accommodate all high-dimensional motion feature points and has a continuous and unified spatial structure. This global spatial form is the initial global manifold.

[0039] Each feature point in the high-dimensional motion feature point set is projected onto the initial global manifold according to its actual spatial location. The projection position of each feature point is recorded in detail, and the distribution of all projected points on the initial global manifold is analyzed. If the projected points in some regions are found to be too dense, it indicates that the curvature of the region is too high, and the curvature of the region needs to be appropriately adjusted to reduce the curvature of the region, so that the distribution of projected points is more uniform. If the projected points in some regions are scattered and cannot accurately reflect the actual relationship between feature points, it indicates that the dimension of the region is insufficient to support the distribution requirements of feature points, and the corresponding dimension needs to be increased to improve the fitting ability of the manifold to feature points. The curvature and dimension of the initial global manifold are continuously adjusted region by region to address the problems in the projection distribution, until the projection of all high-dimensional motion feature points can accurately match their actual spatial location and motion characteristics, and finally a personalized expression manifold that perfectly fits the facial motion characteristics of the target object is formed.

[0040] The beneficial effects of this implementation process are that it extracts a high-dimensional motion feature point set through precise geometric decomposition, providing comprehensive and core data support for subsequent manifold fitting. Domain reconstruction and principal component analysis ensure that the local tangent space can accurately reflect the local spatial characteristics of each feature point. The reasonable splicing of the local tangent spaces forms an initial global manifold that is comprehensive and structurally continuous. The adaptive adjustment based on the projection distribution allows the manifold to accurately fit the personalized facial motion features of the target object. The whole process is progressive and logically rigorous. The final personalized expression manifold can realistically restore the facial motion patterns of the target object, laying a precise and reliable foundation for the subsequent construction of structured expression trajectories and expression recognition, and effectively improving the accuracy of expression information collection and recognition.

[0041] The structure trajectory construction module 103 is used to project the temporal change sequence in the multi-resolution spatiotemporal motion field onto the personalized expression manifold to obtain the projection point sequence of the target object, and connect the projection point sequence to obtain the structured expression trajectory of the target object; In this embodiment of the invention, when the structure trajectory construction module projects the temporal change sequence in the multi-resolution spatiotemporal motion field onto the personalized expression manifold to obtain the projection point sequence of the target object, and connects the projection point sequence to obtain the structured expression trajectory of the target object, it is specifically used for: The multi-resolution spatiotemporal motion field is segmented by a sliding window to obtain temporal motion segments of the multi-resolution spatiotemporal motion field; Extract the local manifold region corresponding to the overall motion pattern in the temporal motion segment from the personalized expression manifold; The temporal motion segment is mapped onto the center point of the local region of the manifold to obtain the initial projection point of the target object; The initial projection points are sequentially arranged to obtain the projection point sequence of the target object; Based on the affine connection structure of the personalized expression manifold, the projection points in the projection point sequence are geometrically connected to obtain the geodesic line segment of the target object; The geodesic segments are smoothly spliced ​​together to obtain the initial facial expression trajectory of the target object; By applying geometric constraints to the initial facial expression trajectory, a structured facial expression trajectory of the target object is obtained.

[0042] A fixed-duration time window is selected, and the multi-resolution spatiotemporal motion field is continuously slid at uniform time intervals. After each slid, all motion information contained in the window is captured. This motion information covers complete data on the overall facial contour changes, local muscle group movements, and subtle texture perturbations within that time period. By sliding the window successively, the entire multi-resolution spatiotemporal motion field is divided into multiple continuous and non-overlapping motion information segments in chronological order. Each segment corresponds to the facial movement within a specific time period, and finally, the temporal motion segments of the multi-resolution spatiotemporal motion field are obtained.

[0043] Analyze the overall characteristics of facial movements in each temporal motion segment, including the range of facial areas involved in the movement, the coordination relationship of the movements in each area, and the rules of change in the intensity and direction of the movement, to form the overall motion pattern of the temporal motion segment. By comparing the motion feature descriptions of each area on the personalized expression manifold, identify the local areas on the manifold that perfectly match the overall motion pattern in terms of movement area, coordination relationship, intensity and direction. These local areas can accurately correspond to the facial movement state reflected by the temporal motion segment, thereby extracting the local areas of the manifold from the personalized expression manifold.

[0044] The geometric center point of the extracted local region of the manifold is determined. This center point is obtained by calculating the average coordinates of all feature points in the local region and can represent the core position of the entire local region of the manifold. The overall motion features contained in the temporal motion segment are summarized and refined to obtain the core motion information of the segment. This core motion information is then correlated with the center point of the local region of the manifold, so that the motion features of the temporal motion segment are concentrated and mapped onto the center point. This center point corresponding to the temporal motion segment is the initial projection point of the target object.

[0045] According to the temporal order of the sliding window segmentation of the temporal motion segments, all the obtained initial projection points are sorted. The initial projection point corresponding to the earliest segmented temporal motion segment is placed at the beginning, and then the subsequent initial projection points are arranged in the order of segmentation. The initial projection point corresponding to the last sliding window is placed at the end, forming a projection point sequence that strictly follows the temporal order. This sequence can reflect the continuous changes of facial movement in the time dimension.

[0046] By deeply analyzing the affine connection structure of the personalized expression manifold, the association rules and connection methods between points at different positions on the manifold in geometric space are clarified. This structure ensures that the connection between points on the manifold conforms to the spatial logic of facial movement. Based on this affine connection structure, the shortest path between two adjacent projection points in the sequence of projection points is found on the personalized expression manifold. This path can accurately reflect the geometric relationship between two adjacent projection points in the manifold space. Such a shortest path is formed between each pair of adjacent projection points, which is the geodesic segment of the target object.

[0047] Connect the geodesic segments corresponding to all adjacent projection points in the projection point sequence in the order of the projection points. At the connection point of two adjacent geodesic segments, analyze the endpoint curvature and extension direction of the two segments. By adjusting the curvature of the segments near the endpoints, the end of the previous geodesic segment smoothly transitions to the starting point of the next geodesic segment, eliminating the sharp corners and breaks at the connection points. This allows all geodesic segments to be integrated into a continuous, seamless trajectory, which is the initial expression trajectory of the target object.

[0048] By combining the objective laws of facial physiological structure and facial expression movements, the constraints such as the range of motion of each facial muscle group, the fixed positional relationship of facial organs, and the reasonable trajectory of facial expression movements are clarified. In accordance with these constraints, each point and line segment on the initial facial expression trajectory is checked one by one. Parts of the trajectory that exceed the range of muscle movement, conflict with the position of facial organs, or do not conform to the laws of facial expression movements are eliminated. Line segments that deviate from the reasonable shape in the trajectory are corrected so that the trajectory can accurately fit the actual movement logic of facial expressions. Finally, a structured facial expression trajectory that is well-organized and conforms to physiological characteristics and movement laws is obtained.

[0049] The beneficial effects are that this implementation process accurately captures facial motion information at different time periods through sliding window segmentation, ensuring the integrity and continuity of temporal motion segments. Extracting the corresponding local regions of the manifold ensures the accuracy of the initial projection points. The temporal arrangement allows the sequence of projection points to reflect the temporal attributes of the motion. Geometric connections and smooth splicing based on affine connection structures form a continuous and smooth initial expression trajectory, while geometric constraints further standardize the rationality of the trajectory. The entire process is progressive, and the final structured expression trajectory can realistically, accurately, and completely reflect the facial expression motion process of the target object. This provides high-quality basic data for subsequent semantic region segmentation, trajectory standardization, and other modules, effectively improving the accuracy of expression information collection and expression recognition.

[0050] The semantic region segmentation module 104 is used to segment the personalized expression manifold into regions based on the geometric feature distribution of the personalized expression manifold, so as to obtain the expression semantic region of the personalized expression manifold. In this embodiment of the invention, when the semantic region segmentation module performs region segmentation on the personalized expression manifold based on the geometric feature distribution of the personalized expression manifold to obtain the expression semantic region of the personalized expression manifold, it is specifically used for: The curvature distribution, normal vector direction, and local neighborhood shape of the personalized expression manifold are used as feature descriptions of the personalized expression manifold. Based on the spatial distribution described by the features, hierarchical clustering is performed on the personalized expression manifold to obtain the initial geometric region of the personalized expression manifold; Boundary consistency analysis is performed on the initial geometric region to obtain the candidate semantic regions of the personalized expression manifold; Assign expression category identifiers to the candidate semantic regions to obtain the expression semantic regions of the personalized expression manifold.

[0051] A comprehensive analysis of the geometric characteristics of personalized facial expression manifolds is conducted, with point-by-point detection of the curvature of the manifold surface. The curvature intensity and trend of each location are recorded to form a complete curvature distribution. The direction perpendicular to the surface at each point is determined, and the direction of the normal vector at each point is clarified using a unified spatial coordinate system as a reference. The set of normal vector directions is then compiled. The arrangement, distribution density, and interconnection relationships of the surrounding feature points are observed, and different types of local spatial morphologies are summarized. The curvature distribution, the set of normal vector directions, and the description of local spatial morphology are integrated to form a feature description that can fully reflect the geometric characteristics of personalized facial expression manifolds.

[0052] Based on the similarity of curvature distribution, normal vector direction, and local neighborhood shape in the feature descriptions, and combined with the spatial positional relationship of each feature point on the personalized expression manifold, hierarchical clustering operations are carried out. First, feature points with highly similar feature descriptions and adjacent spatial positions are grouped into small local clusters. Then, the feature associations and spatial distribution associations between these small clusters are analyzed. Small clusters with high feature description fit and spatial interconnection are gradually merged to form larger regions. After multiple rounds of such hierarchical merging and division, multiple initial geometric regions with consistent internal features, clear boundaries, and mutual independence are finally obtained.

[0053] Each initial geometric region's boundary contour is examined one by one. The feature descriptions of all points on the boundary are analyzed to confirm whether the curvature distribution, normal vector direction, and local neighborhood shape of these boundary points are consistent with the feature descriptions inside the region, avoiding abrupt feature changes at the boundary points. At the same time, it is checked whether the boundaries of adjacent initial geometric regions are smoothly connected, without overlapping, crossing, or breaking. Initial geometric regions with inconsistent boundary point features or poor boundary connections are adjusted by trimming the boundary parts that do not match the features and correcting the region contours so that the boundaries of each region are both complete and consistent with adjacent regions, ultimately resulting in candidate semantic regions with regular boundaries and unified internal features.

[0054] By deeply analyzing the facial movement patterns associated with the geometric feature descriptions corresponding to each candidate semantic region, and combining the facial movement patterns of common expressions, a correspondence between geometric feature descriptions and expression categories is established. For example, if the feature description of a candidate semantic region corresponds to the movement characteristics of the corners of the mouth rising and the muscles on both sides of the face contracting, then the region is assigned the expression category label of "smiling". If the feature description of a candidate semantic region corresponds to the movement characteristics of the eyebrows falling and the forehead lines deepening, then it is assigned the expression category label of "frowning". This ensures that each candidate semantic region obtains a unique and accurate expression category label, and finally forms a personalized expression manifold expression semantic region with clear expression semantics.

[0055] The beneficial effects of this implementation process are that by comprehensively extracting the multi-dimensional geometric features of personalized facial expression manifolds as feature descriptions, it provides a precise basis for region division. Hierarchical clustering operation achieves reasonable division of the initial geometric region based on feature similarity and spatial correlation. Boundary consistency analysis further optimizes the regularity and rationality of the region boundaries. The assignment of expression category labels gives the divided regions clear semantic value. The entire process is progressive, ensuring that the semantic regions of facial expressions can accurately correspond to the actual situation of facial expression movements. This provides a reliable foundation for the semantic mapping and standardization of subsequent structured expression trajectories, effectively improving the accuracy of expression information collection and recognition.

[0056] The trajectory standardization module 105 is used to map the structured facial expression trajectory to the facial expression semantic region to obtain the semantic projection path of the target object, and to back-map the semantic projection path to the personalized facial expression manifold to obtain the standard facial expression trajectory of the target object; In this embodiment of the invention, when the trajectory standardization module performs the mapping of the structured facial expression trajectory to the facial expression semantic region to obtain the semantic projection path of the target object, it is specifically used for: Obtain the local geometric descriptor of the sampling points on the structured facial expression trajectory; Based on the local geometric descriptor, a geometric similarity assessment is performed on the structured facial expression trajectory and the facial expression semantic region to obtain the degree of matching between the structured facial expression trajectory and the facial expression semantic region; Based on the matching degree, a corresponding facial expression semantic region is assigned to the collection point to construct a point-region attribution mapping table for the target object; By connecting the semantic region center points in the point-region attribution mapping table in chronological order, the semantic state transition path of the target object is obtained. Abnormal states are removed from the semantic state transition path to obtain the semantic projection path of the target object.

[0057] When the trajectory standardization module performs the inverse mapping of the semantic projection path onto the personalized expression manifold to obtain the standard expression trajectory of the target object, it is specifically used for: Based on the continuous semantic state segments on the semantic projection path, corresponding feature points are selected from the facial expression semantic region to obtain a reference feature point sequence for the facial expression semantic region; Based on the personalized expression manifold, trajectory fitting is performed on the reference feature point sequence to obtain the smooth parametric curve of the target object; By applying dynamic constraints to the smooth parameter curve, the optimized parameter curve of the target object is obtained; The optimized parameter curve is reparameterized to obtain the standard facial expression trajectory of the target object.

[0058] Local geometric information of all collection points on the structured facial expression trajectory is extracted one by one. The distribution density and relative positional relationship of adjacent collection points around each collection point are carefully observed. The curvature of the location of the collection point is measured. The angle and line segment length formed by the connection between the point and the surrounding points are recorded. This information about the local spatial morphology and associated features of the collection points is integrated and summarized to form a local geometric descriptor that can uniquely represent the local geometric characteristics of each collection point.

[0059] The local geometric descriptor of each acquisition point is compared one by one with the geometric features of all facial expression semantic regions. The focus is on analyzing the curvature features and local point distribution patterns contained in the descriptor and their fit with the curvature distribution and local domain shape of the facial expression semantic region. The number of overlaps and similarity between the two in the core geometric features are counted. Based on these statistical results, a quantitative matching value is given. This value is the degree of matching between the acquisition points on the structured facial expression trajectory and each facial expression semantic region.

[0060] Sort the data according to the matching degree value of each data collection point, select the facial expression semantic region with the highest matching degree as the corresponding region of the data collection point, and match the unique identifier and spatial coordinates of each data collection point with the corresponding facial expression semantic region identifier to form a clear mapping table, so as to ensure that each data collection point has a clear region record, and finally construct a complete point-region attribution mapping table.

[0061] Extract the geometric center point of each facial expression semantic region from the point-region attribution mapping table. This center point is determined by calculating the average spatial coordinates of all feature points in the region, which can represent the core location of the region. Strictly follow the original time order of the collection points on the structured facial expression trajectory, find the center point of the facial expression semantic region corresponding to each collection point, and connect these center points sequentially with smooth line segments to form a path that can reflect the continuous change of facial expression semantic state over time. This path is the semantic state transition path.

[0062] A comprehensive analysis of each semantic state along the semantic state transition path is conducted. Combining the movement patterns of common facial expressions and the limitations of facial physiological structure, it is determined whether each state conforms to the continuous logic of facial expression changes. Abnormal states that do not have a reasonable connection with the preceding and following semantic states, deviate from the normal facial expression movement trajectory, or do not conform to the range of facial muscle movement are identified. The path nodes corresponding to these abnormal states are removed, and the removed paths are smoothed to ensure that each semantic state of the remaining paths can transition continuously and reasonably, ultimately resulting in a complete and abnormal semantic projection path.

[0063] Based on the continuous semantic state segments presented on the semantic projection path, the facial expression semantic region corresponding to each state segment is identified. Within each facial expression semantic region, feature points that can accurately reflect the core geometric features and motion attributes of the region are selected. These feature points must be evenly distributed within the region and cover the key motion positions of the region. According to the temporal order of semantic state transitions, all selected feature points are arranged sequentially to form an ordered sequence of reference feature points.

[0064] Based on the geometric structure of the personalized expression manifold, each feature point in the reference feature point sequence is used as a key node of the path. A curve is drawn in the manifold space. This curve must accurately pass through all key nodes. At the same time, the direction of the curve is adjusted according to the curvature distribution and spatial shape of the manifold to ensure that the transition of the curve between nodes is natural and smooth, without abrupt turns or breaks. Finally, a smooth parametric curve that fits the geometric characteristics of the personalized expression manifold is formed.

[0065] By combining the dynamic laws of facial muscle movement, the reasonable range of speed and acceleration during facial expression movements, as well as the mechanical constraints of muscle contraction and relaxation, are determined. In accordance with these constraints, each segment of the smooth parametric curve is checked one by one. Parts of the curve that are too fast or too slow, have accelerations that exceed the reasonable range, or do not conform to the principles of muscle mechanics are eliminated. The curvature and trend of the curve are adjusted so that the movement state of the curve fully matches the dynamic laws of facial expression movements, thus obtaining the optimized parametric curve.

[0066] The parameter standards for the optimized parameter curves are unified. The curves are reset according to fixed time intervals and spatial scales, so that each segment of the curve has consistent parameter specifications, ensuring that the measurement standards of the curves are consistent in both time and spatial dimensions. Through this standardized parameter adjustment, the inconsistency of parameters caused by differences in the original data in the curves is eliminated, and finally a standardized expression trajectory with unified specifications is formed.

[0067] The beneficial effects are as follows: This implementation process ensures the accuracy of matching the collection points with the semantic regions of facial expressions by accurately extracting local geometric descriptors of the collection points and performing similarity assessments; the construction of point-region attribution mapping tables and semantic state transition paths enables the semantic transformation of facial expression trajectories; and the elimination of abnormal states ensures the reliability of semantic projection paths. Furthermore, the selection of reference feature point sequences, trajectory fitting, dynamic constraints, and reparameterization processes ensure that the standard facial expression trajectory not only conforms to the geometric characteristics of personalized facial expression manifolds but also conforms to the objective laws of facial movements. The entire process is progressive and logically rigorous, and the final standard facial expression trajectory has a high degree of accuracy and standardization, providing high-quality data support for subsequent facial expression reconstruction and recognition, and effectively improving the overall accuracy of facial expression information collection and recognition.

[0068] The expression reconstruction module 106 is used to iteratively optimize the personalized expression manifold based on the geometric consistency between the standard expression trajectory and the structured expression trajectory, and reconstruct the structured expression trajectory according to the optimized expression manifold.

[0069] In this embodiment of the invention, when the expression reconstruction module performs iterative optimization of the personalized expression manifold based on the geometric consistency between the standard expression trajectory and the structured expression trajectory, and reconstructs the structured expression trajectory according to the optimized expression manifold, it is specifically used for: Based on the personalized facial expression manifold, the standard facial expression trajectory and the structured facial expression trajectory are sampled synchronously to obtain the trajectory point pair set of the target object; Geometric attribute comparison is performed on the trajectory point pairs in the trajectory point pair set to obtain the scalar deviation of the trajectory point pair set; Based on the spatial distribution of the scalar deviation, construct the deviation field of the trajectory point pair set; Sub-regions with high deviation values ​​and spatial continuity are identified from the deviation field and used as the geometric misalignment regions of the target object. A correlation analysis is performed on the geometrically misaligned region and the structured facial expression trajectory to obtain the degree of influence of the geometrically misaligned region on the structured facial expression trajectory; Based on the degree of influence, the personalized facial expression manifold is collaboratively adjusted to obtain the adjusted facial expression manifold of the target object; Based on the adjusted facial expression manifold, the geometric properties of the standard facial expression trajectory and the structured facial expression trajectory are re-evaluated to construct a new deviation field for the target object; By comparing and analyzing the deviation intensity of the deviation field and the deviation intensity of the new deviation field, the changes in the deviation field and the new deviation field are obtained. When the change is less than a preset stability threshold, the adjusted facial expression manifold is determined to be the optimized facial expression manifold of the target object.

[0070] The formula for calculating the scalar deviation is: ; in, This indicates the scalar deviation. Let A represent the first curvature of the trajectory point A. Let A represent the second curvature of the trajectory point A. This represents the first curvature of the trajectory point B. This represents the second curvature of the trajectory point B. This represents the unit normal vector of trajectory point A. This represents the unit normal vector of trajectory point B. This represents the geodesic distance between trajectory point A and trajectory point B. This indicates the preset curvature difference adjustment parameter. This represents the preset weight of the normal vector difference term. This indicates the degree of nonlinearity of the preset curvature difference.

[0071] When the expression reconstruction module reconstructs the structured expression trajectory based on the optimized expression manifold, it is specifically used for: The semantic projection path of the target object is mapped onto the optimized facial expression manifold to obtain the optimized projection path of the target object; Dense sampling is performed on the optimized projection path to obtain the key point sequence of the optimized projection path; Based on the geometric neighborhood information of the optimized facial expression manifold, local structure analysis is performed on the key point sequence to obtain the original motion components of the key point sequence. The original motion components are serialized and integrated to obtain the motion sequence of the target object; Based on the geometric features of the structured facial expression trajectory, the motion sequence is spatially reconstructed to obtain the reconstructed structured facial expression trajectory of the target object.

[0072] Based on the spatial coordinate system and time dimension benchmark of personalized facial expression manifold, corresponding sampling points are selected on the standard facial expression trajectory and the structured facial expression trajectory, with the same time interval as the standard. The selection of each sampling point strictly matches the facial movement state at the same time node, ensuring that each set of sampling points consists of a point on the standard facial expression trajectory and a point on the structured facial expression trajectory at the same time. All such point pairs are arranged in chronological order to form the trajectory point pair set of the target object.

[0073] The geometric attributes of each pair of trajectory points in the trajectory point set are analyzed one by one, including the curvature of each trajectory point, the direction perpendicular to the trajectory surface, and the shortest path length between the two points. The curvature of each pair of trajectory points is directly compared, the difference between the two is recorded, the degree of fit of the two points in the vertical direction is judged, the degree of inconsistency of the directions is statistically analyzed, and the influence of the shortest path length between the two points on the difference is combined. By summarizing the differences of these geometric attributes, a scalar deviation that can quantify the geometric inconsistency of each pair of trajectory points is formed.

[0074] The scalar deviation of each pair of trajectory points is precisely mapped to its corresponding spatial coordinate position on the personalized expression manifold. Each scalar deviation value is used as the attribute information of that spatial position. All spatial positions and their corresponding scalar deviation values ​​are integrated to form a field that can comprehensively and continuously reflect the geometric deviation distribution of each region on the personalized expression manifold. This field is the deviation field of the trajectory point pair set.

[0075] By iterating through all scalar deviation values ​​in the deviation field and setting a clear deviation threshold, all manifold spatial locations with deviation values ​​exceeding the threshold are selected. Then, the spatial correlation of these high-deviation locations is analyzed, and high-deviation locations that are adjacent to each other and can form a continuous range are integrated into a complete sub-region. These sub-regions with high deviation and spatial continuity are the geometric misalignment regions of the target object.

[0076] Each geometrically inaccurate region is analyzed to determine its specific location and extent on the personalized facial expression manifold. By comparing the path direction, key node distribution, and geometric features of the structured facial expression trajectory, it is determined which segments of the structured facial expression trajectory have spatial overlap or direct correlation with the inaccurate region. The number of sampling points and the proportion of trajectory length affected by the inaccurate region in the structured facial expression trajectory are counted. Combining the breadth and depth of the influence, the degree of influence of each geometrically inaccurate region on the structured facial expression trajectory is comprehensively determined.

[0077] Based on the degree of influence of each geometrically inaccurate region, priority is given to adjusting the regions with the highest degree of influence. During the adjustment process, the curvature and spatial dimension of the inaccurate region are changed to make the geometric structure of the region more consistent with the actual geometric characteristics of the standard expression trajectory and the structured expression trajectory. At the same time, the overall coherence of the manifold is taken into account to ensure that the adjusted inaccurate region is smoothly connected with other parts of the personalized expression manifold without structural breaks, overlaps or conflicts. Finally, the adjusted expression manifold of the target object is obtained.

[0078] The standard facial expression trajectory and the structured facial expression trajectory are remapped onto the adjusted facial expression manifold. Following the same geometric attribute comparison method as before, the differences in curvature, vertical direction, and shortest path length between each pair of corresponding sampling points are analyzed one by one. These differences are combined to form a new scalar bias. The new scalar bias is distributed on the adjusted facial expression manifold according to its corresponding spatial position to construct a new bias field for the target object.

[0079] Calculate the average value of all scalar deviation values ​​in the deviation field and the new deviation field respectively, and perform a difference operation on the two average values. The result of this operation is the change in the deviation field and the new deviation field. Continuously compare this change with the preset stability threshold to monitor the relationship between the two.

[0080] When the calculated change is less than the preset stability threshold, it means that the geometric structure of the adjusted facial expression manifold has been able to achieve an ideal stable state of geometric consistency between the standard facial expression trajectory and the structured facial expression trajectory, and no further adjustments are needed. At this point, the adjusted facial expression manifold is directly determined as the optimized facial expression manifold of the target object.

[0081] Referring to the correspondence between each semantic state and the facial expression semantic region in the semantic projection path, each node on the semantic projection path is accurately mapped to the corresponding facial expression semantic region on the optimized facial expression manifold based on its own semantic attributes and spatial location information. This ensures that each mapped node not only meets the original semantic attribute requirements but also perfectly matches the geometric structure of the optimized facial expression manifold. All mapped nodes are arranged sequentially according to the time order of the original semantic projection path to form the optimized projection path of the target object.

[0082] Continuous and dense sampling is performed on the optimized projection path at very small spatial intervals. The number of sampling points must be sufficient to cover all key turning points and subtle change areas on the path, ensuring that every subtle movement change on the path can be captured by the corresponding sampling point. All collected sampling points are sorted and arranged according to the direction of the optimized projection path to obtain the key point sequence of the optimized projection path of the target object.

[0083] Geometric information such as the distribution density of surrounding feature points, their positional relationships, and the curvature characteristics of the region are extracted from each keypoint on the optimized facial expression manifold. This information is then used to conduct an in-depth analysis of the motion trends and spatial constraints of each keypoint on the optimized facial expression manifold, thereby reconstructing the facial motion features corresponding to each keypoint, including changes in the overall facial contour, local muscle group movements, and subtle texture perturbations. The set of these reconstructed motion features is the original motion component of the keypoint sequence.

[0084] According to the time sequence of key points, the original motion components corresponding to each key point are arranged sequentially to ensure that the arrangement order of the motion components is completely consistent with the time process of facial expression movements. At the same time, the motion components of adjacent key points are smoothed to eliminate abrupt differences between the motion components of different key points, so that all original motion components are integrated to form a continuous, complete and logically coherent motion sequence of the target object.

[0085] Geometric features such as the curvature distribution, overall path direction, and spatial location of key nodes of the structured facial expression trajectory are extracted. These geometric features are used as core constraints to adjust and optimize the spatial dimension of the motion sequence, so that the geometric shape of the motion sequence is accurately matched with the geometric features of the structured facial expression trajectory. At the same time, the geometric structure characteristics of the optimized facial expression manifold are fully combined to ensure that the spatial distribution of the adjusted motion sequence on the optimized facial expression manifold is reasonable and conforms to the laws of facial movement. Finally, a reconstructed structured facial expression trajectory that can accurately and completely reflect the facial expression movement process of the target object is formed.

[0086] The first and second curvatures of the trajectory points, as well as the unit normal vector, are obtained from the trajectory point pair set obtained by the expression reconstruction module through synchronous sampling of standard and structured expression trajectories. This is achieved by performing geometric attribute analysis on each trajectory point in the pair. The geometric attribute analysis process involves first extracting the local geometric features of a single trajectory point on its corresponding trajectory, and then determining the relevant attributes of the trajectory point using curve geometry calculation methods. These methods analyze the spatial distribution of adjacent trajectory points, build a local curve model, derive the curvature based on the degree of curvature of the local curve model, and determine the unit normal vector based on the direction of the tangent to the local curve model being perpendicular to and pointing outwards from the curve. The geodesic distance between trajectory points is calculated by measuring the distance between two trajectory points in the pair. The shortest path length on the personalized facial expression manifold is obtained by using the geodesic distance calculation method based on the geometric structure of the personalized facial expression manifold. Two trajectory points are used as the start and end points, respectively, and the shortest path connecting these two points is found on the manifold surface. The length of this path is the required geodesic distance. The preset curvature difference term adjustment parameters, preset normal vector difference term weights, and preset curvature difference nonlinearity are fixed values ​​determined by the system before scalar deviation calculation, based on the accuracy requirements of facial expression recognition and historical data verification results. The determination process involves collecting a large amount of standard and structured facial expression trajectory data corresponding to different facial expressions, conducting multiple experiments on different values, observing their impact on the accuracy of the final scalar deviation calculation, and selecting the value that provides the most accurate recognition results for geometrically inaccurate areas as the fixed setting.

[0087] The value ranges of the above three preset parameters are not arbitrarily set, but are determined based on facial physiological structure characteristics, facial expression movement patterns, and verification using a large amount of experimental data. Their core objective is to accurately quantify the geometric consistency deviation between standard and structured facial expression trajectories, providing a reliable basis for iterative optimization of personalized facial expression manifolds. The specific value ranges and setting logic are as follows: The curvature difference adjustment parameter ranges from 0.6 to 0.9. In the geometric changes of facial expressions, curvature difference is a core indicator reflecting the consistency of trajectory morphology, such as the curvature of the corners of the mouth when smiling and the curvature change of forehead wrinkles when frowning. This was experimentally verified using 1200 test samples of different ages, genders, and facial features. When the curvature difference has insufficient weight in the scalar deviation calculation, it will underestimate the core differences in trajectory morphology, leading to inaccurate identification of geometrically inaccurate regions; when At times, it may overemphasize the impact of curvature differences, misjudging subtle, non-expression-related curvature fluctuations in the face as geometric inaccuracies, thus interfering with the optimization direction; when When the value is in the range of 0.6-0.9, it can accurately match the curvature change pattern of facial expression trajectory, ensuring the dominant role and rationality of curvature difference in deviation calculation.

[0088] The weight of the normal vector difference term ranges from 0.3 to 0.5. The normal vector difference is used to supplement the evaluation of the spatial orientation consistency of the trajectory, such as whether the orientation of the trajectory in facial facial movements conforms to the spatial logic of facial muscle contraction or relaxation. Its weight needs to be balanced with the curvature difference term to avoid bias caused by a single-dimensional evaluation. Experimental verification shows that when... At this time, the influence of normal vector differences is excessively weakened, making it impossible to identify inaccurate areas where the trajectory direction does not conform to the facial movement logic, such as the abnormal orientation of the eyebrow trajectory normal vector when frowning; when When the weight of the normal vector difference exceeds a reasonable range, it will obscure the core role of the curvature difference. The core geometric differences in facial expressions stem from curvature rather than direction, leading to distorted bias calculations. When it is in the range of 0.3-0.5, it can be compared with... A "primary and secondary synergy" weighting system is formed to fully cover the dual geometric attributes of trajectory shape and direction, ensuring the completeness of deviation assessment.

[0089] The nonlinearity of curvature difference ranges from 1.2 to 1.8. The curvature difference in facial expression trajectories is not linear; for example, when smiling, the curvature change of the corners of the mouth from their natural state to their maximum amplitude exhibits a gradual, nonlinear increase. Its function is to fit this nonlinear feature by analyzing the trajectory curvature change curves of different facial expressions: when At times, the near-linear mapping cannot accurately capture the non-linear gradual change of curvature. For example, the difference in the rate of curvature change of the drooping corners of the mouth when sad causes the deviation calculation to fail to reflect the true geometric inconsistency; when At times, excessive nonlinear enhancement can distort the true distribution of curvature differences, such as amplifying small curvature fluctuations into significant deviations, leading to misjudgments of the optimization direction; when When the value is in the range of 1.2-1.8, it can accurately fit the nonlinear curvature changes caused by facial muscle movements, ensuring that the quantitative results of curvature differences are consistent with the real laws of facial expression movements.

[0090] This calculation process quantifies the geometric consistency difference between standard and structured facial expression trajectories by comprehensively analyzing the geometric properties of corresponding trajectory points. Specifically, it first calculates the first and second curvature differences between the two trajectory points, then calculates the directional difference of the unit normal vectors of the two trajectory points, and then adjusts the overall difference by combining the geodesic distance between the two trajectory points. The final result is the scalar deviation, which directly reflects the degree of inconsistency in the geometric properties of the two trajectory points. This provides specific difference data for each pair of trajectory points for the subsequent construction of the deviation field. The deviation field is the basis for identifying geometrically inaccurate regions, and the identification of geometrically inaccurate regions clarifies the target area for the collaborative adjustment of personalized facial expression manifolds, thereby achieving iterative optimization of personalized facial expression manifolds.

[0091] When the first or second curvature difference between two trajectory points increases, the corresponding difference calculation result increases, leading to an increase in the overall scalar deviation. When the difference in the unit normal vector direction between two trajectory points increases, the dot product of the unit normal vectors decreases, the corresponding normal vector difference calculation result increases, and the overall scalar deviation also increases. When the geodesic distance between two trajectory points increases, the denominator value used to adjust the overall difference increases, and the overall scalar deviation decreases while the difference calculation result in the numerator remains unchanged. When the preset curvature difference adjustment value increases, the influence of the curvature difference on the overall scalar deviation strengthens, and the overall scalar deviation increases under the same curvature difference. When the preset normal vector difference weight value increases, the influence of the normal vector difference on the overall scalar deviation strengthens, and the overall scalar deviation increases under the same normal vector difference. When the preset curvature difference nonlinearity value increases, the calculation sensitivity of the curvature difference increases, and larger curvature differences are further amplified, making the overall scalar deviation change more significantly when the curvature difference is large.

[0092] The beneficial effects are that this implementation process, through simultaneous sampling and geometric attribute comparison of standard and structured facial expression trajectories, accurately locates the geometrically inaccurate areas of personalized facial expression manifolds. After multiple rounds of iterative optimization, the optimized facial expression manifold has higher geometric consistency and accuracy. The structured facial expression trajectory reconstruction process based on the optimized facial expression manifold, through semantic mapping, dense sampling, and motion component extraction and integration, ensures that the reconstructed trajectory can fit the personalized facial expression features and facial movement patterns of the target object. The entire process is progressive and logically rigorous, and the final reconstructed structured facial expression trajectory has extremely high accuracy and completeness, providing the most reliable core data support for facial expression recognition and significantly improving the overall accuracy of facial expression information collection and recognition.

[0093] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0094] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An expression information collection and expression recognition system, characterized by, The system comprises a motion field construction module, an expression manifold fitting module, a structure trajectory construction module, a semantic region division module, a trajectory standardization module and an expression reconstruction module, wherein: The motion field construction module performs dense optical flow analysis on a face image sequence of a target object to obtain pixel motion vectors of the target object, decomposes the pixel motion vectors into multi-scale motion components to construct a multi-resolution spatiotemporal motion field of the target object, and comprises: A face image sequence of a target object is collected, and global motion compensation is performed on consecutive image frames in the face image sequence to obtain a stable image sequence of the target object; Pixel displacement distribution in the stable image sequence is extracted to obtain an initial two-dimensional displacement field of the target object; The initial two-dimensional displacement field is decomposed at multiple levels to obtain overall contour changes of the face of the target object, local muscle group movements and subtle texture disturbances; The overall contour changes of the face, the local muscle group movements and the subtle texture disturbances are integrated to obtain motion components of the target object; The motion components are subjected to temporal smoothing processing to obtain a spatiotemporal motion representation of the target object; The motion components in the spatiotemporal motion representation are hierarchically constructed to obtain a multi-resolution spatiotemporal motion field of the target object; The expression manifold fitting module performs surface fitting on high-dimensional feature points in the multi-resolution spatiotemporal motion field to obtain a personalized expression manifold of the target object; The structure trajectory construction module projects a time-series change sequence in the multi-resolution spatiotemporal motion field onto the personalized expression manifold to obtain a sequence of projection points, and connects the sequence of projection points to obtain a structured expression trajectory of the target object; The semantic region division module divides the personalized expression manifold based on the geometric feature distribution of the personalized expression manifold to obtain expression semantic regions of the personalized expression manifold; The trajectory standardization module maps the structured expression trajectory into the expression semantic regions to obtain a semantic projection path of the target object, and inversely maps the semantic projection path onto the personalized expression manifold to obtain a standard expression trajectory of the target object, comprising: Local geometric descriptors of collection points on the structured expression trajectory are obtained; Based on the local geometric descriptors, geometric similarity of the structured expression trajectory and the expression semantic regions is evaluated to obtain a matching degree of the structured expression trajectory and the expression semantic regions; According to the matching degree, corresponding expression semantic regions are assigned to the collection points to construct a point-region attribution mapping table of the target object; In time sequence, semantic region center points in the point-region attribution mapping table are connected to obtain a semantic state transition path of the target object; The semantic state transition path is subjected to abnormal state elimination to obtain a semantic projection path of the target object; According to consecutive semantic state segments on the semantic projection path, corresponding feature points are selected from the expression semantic regions to obtain a reference feature point sequence of the expression semantic regions; fitting a trajectory to the reference feature point sequence based on the personalized expression manifold, to obtain a smooth parameter curve of the target object; performing dynamics constraint on the smooth parameter curve, to obtain an optimized parameter curve of the target object; performing re-parameterization on the optimized parameter curve, to obtain a standard expression trajectory of the target object; the expression reconstruction module is configured to perform iterative optimization on the personalized expression manifold based on geometric consistency between the standard expression trajectory and the structured expression trajectory, and reconstruct the structured expression trajectory according to the optimized expression manifold, and the expression reconstruction module comprises: synchronously sampling the standard expression trajectory and the structured expression trajectory based on the personalized expression manifold, to obtain a set of trajectory point pairs of the target object; performing geometric attribute comparison on the trajectory point pairs in the set of trajectory point pairs, to obtain a scalar deviation of the set of trajectory point pairs, wherein a calculation formula of the scalar deviation is: ； wherein, denotes the scalar deviation, denotes a first curvature of the trajectory point A, denotes a second curvature of the trajectory point A, denotes a first curvature of the trajectory point B, denotes a second curvature of the trajectory point B, denotes a unit normal vector of the trajectory point A, denotes a unit normal vector of the trajectory point B, denotes a geodesic distance between the trajectory point A and the trajectory point B, denotes a preset curvature difference term adjustment parameter, denotes a preset normal vector difference term weight, denotes a preset curvature difference nonlinearity degree; constructing a deviation field of the set of trajectory point pairs according to a spatial distribution of the scalar deviation; identifying a sub-region with high deviation value and spatial continuity from the deviation field as a geometric misalignment region of the target object; performing correlation analysis on the geometric misalignment region and the structured expression trajectory, to obtain an influence degree of the geometric misalignment region on the structured expression trajectory; performing collaborative adjustment on the personalized expression manifold based on the influence degree, to obtain an adjusted expression manifold of the target object; performing geometric attribute re-evaluation on the standard expression trajectory and the structured expression trajectory based on the adjusted expression manifold, to construct a new deviation field of the target object; performing comparison analysis on deviation intensities of the deviation field and the new deviation field, to obtain a change amount of the deviation field and the new deviation field; when the change amount is less than a preset stable threshold, determining that the adjusted expression manifold is an optimized expression manifold of the target object; mapping a semantic projection path of the target object to the optimized expression manifold, to obtain an optimized projection path of the target object; performing dense sampling on the optimized projection path, to obtain a key point sequence of the optimized projection path; performing local structure analysis on the key point sequence based on geometric field information of the optimized expression manifold, to obtain an original motion component of the key point sequence; serializing and integrating the original motion component, to obtain a motion sequence of the target object; performing spatial reconstruction on the motion sequence based on geometric features of the structured expression trajectory, to obtain a reconstructed structured expression trajectory of the target object.

2. The facial information acquisition and facial recognition system of claim 1, wherein, In the process of fitting a surface to the high-dimensional feature points in the multi-resolution spatio-temporal motion field to obtain the personalized expression manifold of the target object, the expression manifold fitting module is specifically configured to: perform geometric decomposition on the multi-resolution spatio-temporal motion field, to obtain a set of high-dimensional motion feature points of the multi-resolution spatio-temporal motion field; perform field reconstruction on the set of high-dimensional motion feature points, to obtain a local field structure of the set of high-dimensional motion feature points; perform principal component analysis on the local field structure to obtain a local tangent space of the high-dimensional motion feature point set; based on a position relationship between feature points in the high-dimensional motion feature point set, splice the local tangent space into an initial global manifold of the target object; based on a projection distribution of the high-dimensional motion feature point set on the initial global manifold, adaptively adjust a curvature and a dimension of the initial global manifold to obtain a personalized expression manifold of the target object.

3. The facial information acquisition and facial recognition system of claim 1, wherein, When the structure trajectory construction module performs projection of a time sequence in the multi-resolution spatiotemporal motion field to the personalized expression manifold to obtain a projection point sequence of the target object, and connects the projection point sequence to obtain a structured expression trajectory of the target object, the structure trajectory construction module is specifically configured to: perform sliding window segmentation on the multi-resolution spatiotemporal motion field to obtain a time sequence motion segment of the multi-resolution spatiotemporal motion field; extract a local region of the personalized expression manifold corresponding to an overall motion mode in the time sequence motion segment from the personalized expression manifold; map the time sequence motion segment to a center point of the local region of the personalized expression manifold to obtain an initial projection point of the target object; perform time sequence arrangement on the initial projection point to obtain the projection point sequence of the target object; based on an affine connection structure of the personalized expression manifold, perform geometric connection on the projection points in the projection point sequence to obtain a geodesic segment of the target object; perform smooth splicing on the geodesic segment to obtain an initial expression trajectory of the target object; perform geometric constraint on the initial expression trajectory to obtain the structured expression trajectory of the target object.

4. The facial information acquisition and facial recognition system of claim 1, wherein, When the semantic region division module performs region division on the personalized expression manifold based on a geometric feature distribution of the personalized expression manifold to obtain an expression semantic region of the personalized expression manifold, the semantic region division module is specifically configured to: take a curvature distribution, a normal vector direction, and a local field shape of the personalized expression manifold as a feature description of the personalized expression manifold; based on a spatial distribution of the feature description, perform hierarchical clustering on the personalized expression manifold to obtain an initial geometric region of the personalized expression manifold; perform boundary consistency analysis on the initial geometric region to obtain a candidate semantic region of the personalized expression manifold; assign an expression category identifier to the candidate semantic region to obtain the expression semantic region of the personalized expression manifold.

Images