Distributed real-time face feature extraction method and system based on edge computing

Abstract

The application provides a kind of based on edge computing's distributed real-time face feature extraction method and system, it is related to face recognition technical field, including through to the face image of acquisition time series sampling obtains multiple frame sequences, constructs space-time feature association graph and predicts correction face position, divides optimization image sequence into task package and distributes to edge computing device, constructs feature point local tensor field and extracts feature description information, finally selects optimal feature fusion matrix.The application improves the real-time performance and accuracy of face feature extraction, reduces the consumption of computing resources.

Description

Technical Field

[0001] This invention relates to the field of facial recognition technology, and in particular to a distributed real-time facial feature extraction method and system based on edge computing. Background Technology

[0002] With the rapid development of artificial intelligence technology, facial recognition, as an important biometric identification technology, is widely used in various fields such as security monitoring, identity verification, and smart device unlocking. Traditional facial feature extraction is usually processed on cloud servers, requiring the collected image data to be uploaded to the cloud and then processed by high-performance servers for feature extraction and recognition. However, with the popularization of IoT devices and the expansion of facial recognition application scenarios, this centralized processing mode can no longer meet the requirements of real-time performance, privacy protection, and network bandwidth.

[0003] Edge computing, as a computing paradigm, deploys computing and storage resources at the network edge, close to the data source. This effectively reduces data transmission latency, improves real-time processing, and alleviates the computing pressure on the central cloud. Migrating facial feature extraction tasks from the cloud to the edge, utilizing distributed computing, better meets the requirements for low latency and high reliability, making it particularly suitable for facial recognition applications in mobile scenarios.

[0004] However, existing facial feature extraction methods still have some shortcomings. Most existing facial feature extraction methods are based on single-frame image processing, lacking effective utilization of spatiotemporal information of faces in continuous video sequences. They are also unable to cope with complex scenarios such as changes in facial pose and illumination, affecting the accuracy and stability of feature extraction. Existing edge computing resource allocation strategies usually adopt a static approach, which cannot dynamically adjust task allocation according to facial motion characteristics and edge device load, resulting in low efficiency of computing resource utilization and unstable processing latency. In existing technologies, the feature point extraction process often adopts a fixed algorithm pattern, lacking an adaptive mechanism based on local information. This makes it impossible to differentiate the characteristics of different facial regions, affecting the accuracy and discriminability of feature description. Summary of the Invention

[0005] This invention provides a distributed real-time facial feature extraction method and system based on edge computing, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides a distributed real-time facial feature extraction method based on edge computing, comprising:

[0007] The face image data collected by the terminal device is sampled in time sequence to obtain a continuous multi-frame image sequence;

[0008] Establish an edge computing resource pool that includes multiple edge computing devices;

[0009] A spatiotemporal feature association map is constructed from a series of consecutive image frames to extract the facial motion trajectory. The face position is predicted by combining the recursive Bayesian method and corrected by the reprojection error compensation method to generate a spatiotemporally aligned optimized face image sequence.

[0010] The optimized face image sequence is divided into continuous frame task packages, and a coloring and marking method is used to allocate the continuous frame task packages. A feature point motion prediction mechanism is established, and the continuous frame task packages are allocated to edge computing devices to perform feature extraction tasks.

[0011] In edge computing devices, a local tensor field of feature points is constructed and the potential energy value is obtained. Multi-level seed points are established to determine the feature point extraction sequence. Feature description information is extracted based on the tensor field gradient, a tensor feature sequence is generated, and a feature fusion matrix is ​​constructed.

[0012] The feature fusion matrix generated by each edge computing device is verified in a time sequence. The optimal feature fusion matrix is ​​selected according to the preset mutual information threshold and then sent to the terminal device.

[0013] In one optional embodiment, constructing a spatiotemporal feature association map for a series of consecutive image frames and extracting the facial motion trajectory includes:

[0014] For each frame in a continuous multi-frame image sequence, a set of feature points is extracted, the spatial distance between each feature point and its surrounding feature points is calculated, the influence radius of the Gaussian kernel function is adaptively determined based on the local distribution density of the feature points, and the spatial distance is weighted using the Gaussian kernel function to obtain the local density value of each feature point.

[0015] For each feature point, the spatial gradient is calculated based on the local density value. The density gradient field pointing towards the direction of increasing local density value is obtained. The cosine of the angle between the density gradient direction and the direction of the line connecting the feature points is calculated. The spatial distance is then weighted using a Gaussian kernel function to obtain the spatial correlation strength between feature points.

[0016] The ratio of the local density values ​​of corresponding feature points in adjacent frames is calculated as the density transfer ratio of the feature points, and the temporal correlation between feature points in adjacent frames is established based on the density transfer ratio.

[0017] A spatiotemporal feature association graph is constructed using feature points as vertices, spatial association strength as intra-frame edge weights, and density transfer ratio as inter-frame edge weights.

[0018] In the spatiotemporal feature association graph, a density flow field is constructed based on the density gradient field and the density transfer ratio. The streamline trajectory is iteratively calculated along the intra-frame edge weights and inter-frame edge weights of the spatiotemporal feature association graph to obtain the face motion trajectory.

[0019] In one optional embodiment, in the spatiotemporal feature association graph, a density flow field is constructed based on the density gradient field and the density transfer ratio. The streamline trajectory is iteratively calculated along the intra-frame edge weights and inter-frame edge weights of the spatiotemporal feature association graph to obtain the face motion trajectory, including:

[0020] The first direction vector of the density gradient field is extracted from the spatiotemporal feature correlation map, the density transfer ratio is mapped to the second direction vector, and the first and second direction vectors are fused by adaptive weight to construct the density flow field.

[0021] In the density flow field, feature points are used as the initial points of streamlines. The motion direction of feature points in the spatial domain is determined by the intra-frame edge weights of the spatiotemporal feature correlation graph, and the motion direction of feature points in the temporal domain is determined by the inter-frame edge weights. The feature point positions are iteratively updated until convergence, and streamline trajectories are generated.

[0022] Starting from the region with the highest feature point density, streamline trajectories are traced in the forward and reverse directions in the density flow field to obtain the motion trajectory sequence of the feature point group. Curve fitting is then performed on the motion trajectory sequence to obtain the face motion trajectory.

[0023] In one optional embodiment, the optimized face image sequence is divided into consecutive frame task packages, a coloring and marking method is used to allocate the consecutive frame task packages, a feature point motion prediction mechanism is established, and the consecutive frame task packages are allocated to edge computing devices to perform feature extraction tasks, including:

[0024] Obtain the time interval of each frame in the optimized face image sequence, calculate the motion displacement and motion direction of the image frame based on the time interval, and divide the continuous frame images into continuous frame task packages of adaptive size based on the motion displacement and motion direction.

[0025] The number of feature points and the area of ​​the spatial distribution region of each frame image in the continuous frame task package are counted. The ratio of the number of feature points to the area of ​​the spatial distribution region is calculated, and the ratio is used as the correlation strength between the continuous frame task packages.

[0026] Adjacent consecutive frame task packets are colored based on their association strength. Adjacent consecutive frame task packets with an association strength higher than a preset strength threshold are marked with different colored tags. The number of colored tags is equal to the number of edge computing devices, thus obtaining the colored tag for each consecutive frame task packet.

[0027] A motion state model of feature points is established for consecutive frame task packets with the same coloring mark. Based on the motion state model, the migration direction of feature points between consecutive frame task packets with different coloring marks is predicted, and feature point detection is performed in the region corresponding to the migration direction of the feature points.

[0028] Consecutive frame task packets with the same coloring mark are assigned to corresponding edge computing devices. Each edge computing device performs parallel feature extraction tasks based on the feature point migration direction. The number of frames in the consecutive frame task packets is dynamically adjusted according to the processor utilization of the edge computing devices.

[0029] In one optional embodiment, establishing a motion state model of feature points for consecutive frame task packets with the same coloring label includes:

[0030] For feature points in a continuous frame task packet, a K-order neighborhood graph is constructed with the feature points as the center. The topological distance matrix and angle difference matrix between the feature points and their neighboring feature points are calculated to obtain the feature point distribution density per unit area in the K-order neighborhood graph.

[0031] Based on the topological distance matrix, the angle difference matrix, and the feature point distribution density, local motion constraints are constructed. The affine transformation relationship of the feature points in the continuous frame task packet is calculated, and the scale component, rotation component, and translation component of the affine transformation relationship are extracted.

[0032] The degree of violation of the local motion constraints and the residual of the affine transformation relationship are used as optimization objectives. The optimization objectives are iterated using the gradient descent method. Based on the optimization results, the evolution law of the motion state of the feature points and the correspondence with actual observations are determined, thus obtaining the motion state model of the feature points.

[0033] In one optional embodiment, a local tensor field of feature points is constructed in the edge computing device and the potential energy value is obtained. A multi-level seed point is established to determine the feature point extraction sequence. Feature description information is extracted based on the tensor field gradient to generate a tensor feature sequence. The construction of a feature fusion matrix includes:

[0034] In edge computing devices, local tensor descriptors between feature points are constructed, and the orientation parameters, scale parameters, and intensity parameters of the corresponding feature points are extracted. A tensor field is established, and the regional correlation and structural importance of feature points in the tensor field are calculated to obtain the potential energy value of the feature points.

[0035] The feature point with the largest potential energy value is selected as the first-level seed point. Multi-level seed points are determined based on the potential energy difference. The optimal traversal path is selected, and the position index information of the feature points on the optimal traversal path is recorded to generate the feature point extraction sequence.

[0036] Calculate the gradient distribution value of the tensor field, extract the first feature description information for regions where the gradient distribution value is greater than the preset gradient intensity threshold, and extract the second feature description information for regions where the gradient distribution value is less than the preset gradient intensity threshold, so as to obtain a multi-dimensional feature description of the feature points.

[0037] The feature extraction order is determined based on the feature point extraction sequence. The structural parameters and neighborhood data of the feature points are extracted, and tensor feature vectors are generated and organized according to the order of the feature point extraction sequence to form a tensor feature sequence.

[0038] Calculate the phase difference values ​​of adjacent feature vectors in the tensor feature sequence, construct the phase difference matrix and calculate the feature fusion weight coefficients to generate the feature fusion weight matrix. Use the feature fusion weight matrix to fuse the tensor feature sequence to obtain the feature fusion matrix.

[0039] In one optional embodiment, the feature point with the largest potential energy value is selected as the first-level seed point. Multi-level seed points are determined based on the potential energy difference. An optimal traversal path is selected, and the position index information of the feature points on the optimal traversal path is recorded. The generation of the feature point extraction sequence includes:

[0040] In the tensor field, the feature point with the largest potential energy value is determined as the first-level seed point. A traversal window is established with the first-level seed point as the center, and the potential energy difference of the feature points within the traversal window is calculated.

[0041] Select the feature point with the smallest potential energy difference as the second-level seed point, and record the traversal direction and traversal distance from the first-level seed point to the second-level seed point.

[0042] In the traversal direction, a search sector is determined, and according to a preset number of preferred features, a corresponding number of feature points in the search sector, sorted from high to low by potential energy value, are selected as third-level seed points.

[0043] Using the third-level seed point as a node, connect them to form a candidate traversal path, and calculate the potential energy gradient of adjacent feature points on the candidate traversal path.

[0044] The candidate traversal path with the smallest potential gradient is selected as the optimal traversal path. The position index information of the feature points is recorded along the optimal traversal path to generate a feature point extraction sequence.

[0045] A second aspect of this invention provides a distributed real-time facial feature extraction system based on edge computing, comprising:

[0046] The first unit is used to perform time-series sampling of the face image data collected by the terminal device to obtain a continuous multi-frame image sequence.

[0047] The second unit is used to establish an edge computing resource pool that includes multiple edge computing devices.

[0048] The third unit is used to construct a spatiotemporal feature association map for a series of consecutive multi-frame image sequences, extract the facial motion trajectory, predict the facial position by combining the recursive Bayesian method, correct it by using the reprojection error compensation method, and generate a spatiotemporally aligned optimized facial image sequence.

[0049] The fourth unit is used to divide the optimized face image sequence into continuous frame task packages, allocate the continuous frame task packages using a coloring and marking method, establish a feature point motion prediction mechanism, and allocate the continuous frame task packages to edge computing devices to perform feature extraction tasks.

[0050] The fifth unit is used to construct the local tensor field of feature points in the edge computing device and obtain the potential energy value, establish multi-level seed points to determine the feature point extraction sequence, extract feature description information based on the tensor field gradient, generate tensor feature sequence, and construct feature fusion matrix.

[0051] The sixth unit is used to perform time-series verification of the feature fusion matrix generated by each edge computing device, select the optimal feature fusion matrix according to the preset mutual information threshold, and send the optimal feature fusion matrix to the terminal device.

[0052] A third aspect of the present invention provides an electronic device, comprising:

[0053] processor;

[0054] Memory used to store processor-executable instructions;

[0055] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0057] In this embodiment of the invention, by constructing a spatiotemporal feature association graph and combining it with a recursive Bayesian method to predict face location, and using reprojection error compensation for correction, spatiotemporal alignment optimization of face images in motion is achieved, effectively improving the accuracy and stability of face feature extraction. A coloring and marking method is used to allocate continuous frame task packets, and a feature point motion prediction mechanism is established, achieving efficient scheduling and load balancing of edge computing resources, significantly reducing system response time and improving the real-time performance of feature extraction. A local tensor field of feature points is constructed in the edge computing device, and the feature extraction sequence is determined based on multi-level seed points. Feature description information is extracted based on the tensor field gradient, and the optimal feature fusion matrix is ​​selected through temporal verification and mutual information thresholding, greatly improving the robustness and discriminative ability of feature expression and adapting to the face recognition needs in complex environments. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the distributed real-time facial feature extraction method based on edge computing according to an embodiment of the present invention.

[0059] Figure 2 Flowchart for constructing a motion state model for feature points;

[0060] Figure 3 This is a network structure diagram of the feature point extraction algorithm. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0063] Figure 1 This is a flowchart illustrating the distributed real-time facial feature extraction method based on edge computing according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0064] The face image data collected by the terminal device is sampled in time sequence to obtain a continuous multi-frame image sequence;

[0065] Establish an edge computing resource pool that includes multiple edge computing devices;

[0066] A spatiotemporal feature association map is constructed from a series of consecutive image frames to extract the facial motion trajectory. The face position is predicted by combining the recursive Bayesian method and corrected by the reprojection error compensation method to generate a spatiotemporally aligned optimized face image sequence.

[0067] The optimized face image sequence is divided into continuous frame task packages, and a coloring and marking method is used to allocate the continuous frame task packages. A feature point motion prediction mechanism is established, and the continuous frame task packages are allocated to edge computing devices to perform feature extraction tasks.

[0068] In edge computing devices, a local tensor field of feature points is constructed and the potential energy value is obtained. Multi-level seed points are established to determine the feature point extraction sequence. Feature description information is extracted based on the tensor field gradient, a tensor feature sequence is generated, and a feature fusion matrix is ​​constructed.

[0069] The feature fusion matrix generated by each edge computing device is verified in a time sequence. The optimal feature fusion matrix is ​​selected according to the preset mutual information threshold and then sent to the terminal device.

[0070] In one optional implementation, constructing a spatiotemporal feature association map for a sequence of consecutive multi-frame images and extracting the facial motion trajectory includes:

[0071] For each frame in a continuous multi-frame image sequence, a set of feature points is extracted, the spatial distance between each feature point and its surrounding feature points is calculated, the influence radius of the Gaussian kernel function is adaptively determined based on the local distribution density of the feature points, and the spatial distance is weighted using the Gaussian kernel function to obtain the local density value of each feature point.

[0072] For each feature point, the spatial gradient is calculated based on the local density value. The density gradient field pointing towards the direction of increasing local density value is obtained. The cosine of the angle between the density gradient direction and the direction of the line connecting the feature points is calculated. The spatial distance is then weighted using a Gaussian kernel function to obtain the spatial correlation strength between feature points.

[0073] The ratio of the local density values ​​of corresponding feature points in adjacent frames is calculated as the density transfer ratio of the feature points, and the temporal correlation between feature points in adjacent frames is established based on the density transfer ratio.

[0074] A spatiotemporal feature association graph is constructed using feature points as vertices, spatial association strength as intra-frame edge weights, and density transfer ratio as inter-frame edge weights.

[0075] In the spatiotemporal feature association graph, a density flow field is constructed based on the density gradient field and the density transfer ratio. The streamline trajectory is iteratively calculated along the intra-frame edge weights and inter-frame edge weights of the spatiotemporal feature association graph to obtain the face motion trajectory.

[0076] In one specific implementation, for a sequence of consecutive multi-frame face images, a set of feature points is extracted from each frame. Feature point extraction algorithms such as SIFT or SURF can be used to extract key points from the image as the feature point set. Assume the feature point set extracted from the i-th frame is Pi = {p1, p2, ..., pn}, where each feature point contains location coordinates and descriptive information.

[0077] Calculate the spatial distance between each feature point and its surrounding feature points. For feature points pi and pj, calculate the Euclidean distance dij between them. For example, for feature point p1 located at coordinates (10, 15) and feature point p2 located at coordinates (13, 19), their spatial distance d12 is 5 pixels. Adaptively determine the influence radius dc of the Gaussian kernel function based on the local distribution density of the feature points. Determine the value of dc by calculating the average k-nearest neighbor distance of the feature points. For example, when k=7, calculate the average distance from each feature point to its 7 nearest neighbor feature points, and then average this average distance across all feature points to obtain the value of dc. For example, in a certain frame of the image, dc = 12 pixels.

[0078] The Gaussian kernel function is used to perform weighted calculations of spatial distances. The Gaussian kernel function is expressed as exp(-dij). 2 / dc 2 ), where dij is the distance between feature points and dc is the radius of influence. When the distance between two points is 6 pixels and dc is 12 pixels, the Gaussian kernel value is approximately 0.75. For each feature point pi, the sum of its Gaussian kernel value and that of all other feature points is calculated to obtain the local density value ρi of that feature point. For example, if the Gaussian kernel values ​​of a feature point and its 10 surrounding feature points are [0.95, 0.86, 0.75, 0.64, 0.52, 0.42, 0.31, 0.22, 0.15, 0.09], then the local density value of that feature point is the sum of these values, 4.91.

[0079] For each feature point, the spatial gradient of the local density value is calculated to obtain the density gradient field pointing in the direction of increasing local density value. On the image plane, the local density values ​​of the feature point distribution are interpolated to construct a density field, and then the gradient of this density field is calculated. For example, the density gradient direction at feature point p1 is (0.6, 0.8), which indicates the direction of the fastest density increase at that point.

[0080] Calculate the cosine of the angle between the density gradient direction and the direction of the line connecting the feature points. For feature points pi and pj, calculate the cosine of the angle between the gradient direction vector and the direction vector of the line connecting the feature points, cosθij. For example, if the direction of the line connecting feature points p1 and p2 is (0.8, 0.6), and the gradient direction at p1 is (0.6, 0.8), then the cosine of the angle is 0.96. Combining this with a Gaussian kernel function to weight the spatial distance, multiply cosθij by the Gaussian kernel function value to obtain the spatial association strength sij between feature points. When cosθij is 0.96 and the Gaussian kernel function value is 0.75, the spatial association strength sij is 0.72.

[0081] Calculate the ratio of the local density values ​​of corresponding feature points in adjacent frames. Use a feature matching algorithm such as FLANN to determine corresponding feature point pairs in adjacent frames. For feature point pi in frame t and its corresponding feature point pi' in frame t+1, calculate the density transfer ratio tij = ρi' / ρi. For example, if the local density value of pi is 4.91 and the local density value of pi' is 5.15, the density transfer ratio is 1.05, indicating a slight increase in the density of this feature point. Establish the temporal correlation between feature points in adjacent frames based on the density transfer ratio.

[0082] A spatiotemporal feature graph G is constructed using feature points as vertices, spatial association strength as intra-frame edge weights, and density transfer ratio as inter-frame edge weights. In this graph, feature points in each frame form a layer, with points within a layer connected by spatial association strength, and corresponding points in different layers connected by density transfer ratios. For example, for a sequence containing 5 frames of images, with 100 feature points extracted from each frame, the constructed spatiotemporal feature graph contains 500 vertices, approximately 4500 intra-frame edges, and approximately 400 inter-frame edges.

[0083] In the spatiotemporal feature correlation map, a density flow field is constructed based on the density gradient field and the density transfer ratio. The density flow field defines the flow direction and intensity of feature points in space and time. For intra-frame flow, the direction is determined by the density gradient, and the intensity is determined by the spatial correlation intensity; for inter-frame flow, the direction is determined by the temporal direction, and the intensity is determined by the density transfer ratio.

[0084] The streamline trajectory is iteratively calculated along the intra-frame and inter-frame edge weights of the spatiotemporal feature correlation graph. Starting from the initial feature point, iterative calculations are performed under the guidance of the density flow field. In each iteration, the next movement direction and step size are determined based on the intra-frame and inter-frame correlation strength of the current point. For example, if the maximum intra-frame correlation strength between the current point and its neighbors is 0.72, and the corresponding inter-frame correlation strength is 1.05, then movement is preferentially carried out along this direction, with the step size proportional to the correlation strength. Through multiple iterations, a complete streamline trajectory is formed, representing the motion trajectory of a specific part of the face.

[0085] The extracted streamline trajectories are filtered and smoothed to remove abnormal trajectories, resulting in the final set of facial motion trajectories. For example, in a sequence of 10 facial images, 5-10 stable trajectories are extracted from key areas such as the corners of the eyes and mouth to accurately reflect the movement of various parts of the face during changes in facial expression.

[0086] In one optional implementation, in the spatiotemporal feature correlation graph, a density flow field is constructed based on the density gradient field and the density transfer ratio. The streamline trajectory is iteratively calculated along the intra-frame edge weights and inter-frame edge weights of the spatiotemporal feature correlation graph to obtain the face motion trajectory, including:

[0087] The first direction vector of the density gradient field is extracted from the spatiotemporal feature correlation map, the density transfer ratio is mapped to the second direction vector, and the first and second direction vectors are fused by adaptive weight to construct the density flow field.

[0088] In the density flow field, feature points are used as the initial points of streamlines. The motion direction of feature points in the spatial domain is determined by the intra-frame edge weights of the spatiotemporal feature correlation graph, and the motion direction of feature points in the temporal domain is determined by the inter-frame edge weights. The feature point positions are iteratively updated until convergence, and streamline trajectories are generated.

[0089] Starting from the region with the highest feature point density, streamline trajectories are traced in the forward and reverse directions in the density flow field to obtain the motion trajectory sequence of the feature point group. Curve fitting is then performed on the motion trajectory sequence to obtain the face motion trajectory.

[0090] In one specific implementation, a set of facial feature points from multiple consecutive frames of images is acquired to construct a spatiotemporal feature association graph. This graph includes two types of edges: intra-frame association edges and inter-frame association edges. Intra-frame association edges represent the spatial relationship between feature points within the same frame, while inter-frame association edges represent the temporal relationship between feature points in adjacent frames. For each feature point, its local density value in the feature space is calculated. This density value can be obtained by counting the number of other feature points within a preset radius centered on that feature point. For example, a circular region with a radius of 10 pixels is selected, and the number of feature points falling within this region is counted as the density value of that point.

[0091] When extracting the first direction vector of the density gradient field from the spatiotemporal feature correlation map, for each feature point, the rate of change of its density value relative to the eight surrounding directions is calculated. Specifically, the neighboring points in the eight directions (0°, 45°, 90°, 135°, 180°, 225°, 270°, 315°) around the feature point are selected, the density differences in these directions are calculated, and normalized to a unit vector. For example, if the density value of a feature point is 15, and the density value of its neighboring point in the 45° direction is 20, then the density gradient in that direction is 5, pointing towards 45°. The density gradients in the eight directions are combined and calculated, and the direction with the largest density change is taken as the first direction vector.

[0092] The density transfer ratio is determined by analyzing the migration of feature points between adjacent frames. For a feature point in the current frame, the corresponding feature point in the next frame is found through the inter-frame association edges, and the density ratio between the two points is calculated. For example, if the density of a feature point in the current frame is 12, and the density of the corresponding point in the next frame is 18, then the density transfer ratio is 1.5. This ratio is mapped to a second direction vector, pointing in the direction of increasing density, and the magnitude of the vector is proportional to the ratio.

[0093] When adaptively weighting the first and second direction vectors, the weights are adjusted based on the local density distribution characteristics of the feature points. In regions with drastic density changes, the first direction vector has a larger weight (e.g., 0.7); in regions with gradual density changes, the second direction vector has a larger weight (e.g., 0.7). Through this adaptive weight allocation, the two direction vectors are fused to construct the density flow field.

[0094] When generating streamline trajectories in a density flow field, feature points are selected as the initial points of the streamlines. For the initial points, the intra-frame edge weights based on the spatiotemporal feature correlation graph determine their motion direction in the spatial domain. The intra-frame edge weights can be calculated based on the Euclidean distance between feature points, with a larger weight for closer points. For example, if the distance between two feature points is 5 pixels, the intra-frame edge weight can be set to 0.8; if the distance is 15 pixels, the weight can be set to 0.3. The inter-frame edge weights determine the motion direction of feature points in the temporal domain. The inter-frame edge weights can be calculated based on the matching confidence of feature points in adjacent frames, with a larger weight for higher matching confidence. For example, if the matching confidence of corresponding feature points in two frames is 0.9, the inter-frame edge weight can be set to 0.85.

[0095] The feature point positions are iteratively updated. Starting from the initial feature point, the feature point moves to a new position with a fixed step size (e.g., 2 pixels) according to the direction of the density flow field. The flow field direction is recalculated at the new position, and the movement continues. This process is repeated iteratively. When the feature point moves a distance less than a threshold (e.g., 0.1 pixels) or reaches the maximum number of iterations (e.g., 100 times), convergence is considered achieved, and a streamline trajectory is generated.

[0096] To obtain the complete facial motion trajectory, it is necessary to determine the region with the highest feature point density as the starting position. For example, in facial images, areas such as the tip of the nose and the corners of the eyes typically have high feature point density; therefore, the top 5% of feature points with the highest density values ​​can be selected as the starting position. Starting from this position, streamline trajectories are traced in both the forward and reverse directions within the density flow field. Forward tracing captures the motion trend in future frames, while reverse tracing reconstructs the motion history in past frames. Combining these two methods yields a complete sequence of feature point group motion trajectories.

[0097] The obtained motion trajectory sequence is subjected to curve fitting using cubic spline interpolation to ensure a smooth transition of the fitted curve at the trajectory points. The control point interval is set to 5 frames, and the curve shape is adjusted at each control point based on the actual trajectory point location. For example, for a trajectory sequence containing 50 frames, the positions of frames 1, 6, 11...46 are selected as control points, and a smooth, continuous curve is generated using cubic spline interpolation to obtain the face motion trajectory.

[0098] The above process effectively captures subtle facial movements in video sequences, achieving high-precision facial motion trajectory extraction. This method combines the advantages of density gradient fields and density transfer ratios, adapting to facial motion characteristics in different scenarios and providing crucial technical support for applications such as facial expression analysis and emotion recognition.

[0099] In one optional implementation, the optimized face image sequence is divided into consecutive frame task packages, a coloring and marking method is used to allocate the consecutive frame task packages, a feature point motion prediction mechanism is established, and the consecutive frame task packages are allocated to edge computing devices to perform feature extraction tasks, including:

[0100] Obtain the time interval of each frame in the optimized face image sequence, calculate the motion displacement and motion direction of the image frame based on the time interval, and divide the continuous frame images into continuous frame task packages of adaptive size based on the motion displacement and motion direction.

[0101] The number of feature points and the area of ​​the spatial distribution region of each frame image in the continuous frame task package are counted. The ratio of the number of feature points to the area of ​​the spatial distribution region is calculated, and the ratio is used as the correlation strength between the continuous frame task packages.

[0102] Adjacent consecutive frame task packets are colored based on their association strength. Adjacent consecutive frame task packets with an association strength higher than a preset strength threshold are marked with different colored tags. The number of colored tags is equal to the number of edge computing devices, thus obtaining the colored tag for each consecutive frame task packet.

[0103] A motion state model of feature points is established for consecutive frame task packets with the same coloring mark. Based on the motion state model, the migration direction of feature points between consecutive frame task packets with different coloring marks is predicted, and feature point detection is performed in the region corresponding to the migration direction of the feature points.

[0104] Consecutive frame task packets with the same coloring mark are assigned to corresponding edge computing devices. Each edge computing device performs parallel feature extraction tasks based on the feature point migration direction. The number of frames in the consecutive frame task packets is dynamically adjusted according to the processor utilization of the edge computing devices.

[0105] In one specific implementation, the time interval between each frame in the optimized face image sequence is obtained, for example, the time difference between adjacent frames is 30 milliseconds. The motion displacement and direction of the image frames are calculated based on the time interval, specifically by comparing the coordinate changes of facial feature points in adjacent frames. For example, the calculated average inter-frame displacement is 5 pixels, with the main motion direction being a 45-degree angle to the left. Based on these motion parameters, consecutive frame images are divided into adaptively sized consecutive frame task packages. When the motion displacement is small (e.g., less than 3 pixels) and the direction is stable, 8-12 frames can be divided into one task package; when the motion displacement is large (e.g., greater than 10 pixels) or the direction changes significantly, 3-5 frames are divided into one task package to adapt to dynamically changing scenarios.

[0106] For each pre-defined consecutive frame task packet, the number of feature points and the area of ​​their spatial distribution in each frame are counted. Feature points are obtained using the FAST corner detection algorithm. For example, if 128 feature points are detected in a frame, these feature points are distributed within an area of ​​12,800 square pixels. The ratio of the number of feature points to the area of ​​the spatial distribution is calculated as 128 ÷ 12,800 = 0.01. This ratio is used as the correlation strength between consecutive frame task packets. The difference in correlation strength between adjacent task packets reflects the degree of change in image content; the smaller the difference, the stronger the correlation.

[0107] Adjacent consecutive frame packets are color-coded based on their association strength. A preset strength threshold of 0.003 is set. When the association strength difference between adjacent packets exceeds this threshold, they are assigned different color tags. The number of color tags is equal to the number of edge computing devices. For example, if there are four edge computing devices, four color tags—red, green, blue, and yellow—are used. Thus, a packet with an association strength of 0.01 might be marked as red, while an adjacent packet with an association strength of 0.015 might be marked as green.

[0108] For consecutive frame task packets with the same coloring mark, the system establishes a motion state model for feature points. Taking the red-marked task packet as an example, by analyzing the positional changes of feature points in consecutive frames, a state model describing the motion trend of feature points is established. This model considers the velocity and acceleration changes of feature points; for example, if a trend of a face rotating to the right is detected, it predicts that the feature points in the next task packet will move to the right by approximately 7 pixels. Based on this motion state model, the system predicts the migration direction of feature points between consecutive frame task packets corresponding to different coloring marks and focuses on detecting feature points in areas where they are likely to appear, thereby improving detection efficiency.

[0109] Consecutive frame task packets with the same coloring are assigned to corresponding edge computing devices. For example, red-marked task packets are assigned to device 1, green-marked task packets to device 2, and so on. Each edge computing device performs parallel feature extraction tasks based on the predicted feature point migration direction. When the processor utilization of device 1 exceeds 85%, the system dynamically reduces the number of frames in the red task packets from 10 frames to 6 frames; when the processor utilization of device 2 is below 40%, the system increases the number of frames in the green task packets from 6 frames to 9 frames to achieve balanced utilization of computing resources.

[0110] For example, for a 30fps face video stream, this method can divide the 300-frame video sequence into approximately 30-40 consecutive frame task packets and distribute them to four edge computing devices for parallel processing. Through a feature point motion prediction mechanism, the accuracy of feature point detection is improved by approximately 15%, while reducing computational redundancy by approximately 30%. Dynamically adjusting the task packet size under different load conditions keeps the processor utilization of each device within the ideal range of 50%-80%, achieving efficient utilization of computing resources and reducing overall processing time by approximately 25% compared to traditional methods.

[0111] In this embodiment, adaptively sized continuous frame task packets are partitioned using motion displacement and direction information between image frames, making the task packets more closely match changes in image content and helping to reduce computational redundancy. Association strength is calculated based on feature point density and distribution area, and a coloring method is used for task allocation, ensuring better balance in the tasks processed by each edge device and avoiding single-point overload. The number of task packet frames is dynamically adjusted using processor utilization, allowing task allocation to be flexibly adjusted according to the real-time load of the edge devices, improving overall computational efficiency and response speed. By establishing a feature point motion state model to predict the migration direction of feature points, it helps to accurately detect changes in feature points between different task packets, maintaining the spatiotemporal continuity of feature extraction. Each task packet can be allocated to multiple edge devices for parallel processing, effectively reducing feature extraction latency and meeting real-time requirements.

[0112] In one optional implementation, establishing a motion state model of feature points for consecutive frame task packets with the same coloring label includes:

[0113] For feature points in a continuous frame task packet, a K-order neighborhood graph is constructed with the feature points as the center. The topological distance matrix and angle difference matrix between the feature points and their neighboring feature points are calculated to obtain the feature point distribution density per unit area in the K-order neighborhood graph.

[0114] Based on the topological distance matrix, the angle difference matrix, and the feature point distribution density, local motion constraints are constructed. The affine transformation relationship of the feature points in the continuous frame task packet is calculated, and the scale component, rotation component, and translation component of the affine transformation relationship are extracted.

[0115] The degree of violation of the local motion constraints and the residual of the affine transformation relationship are used as optimization objectives. The optimization objectives are iterated using the gradient descent method. Based on the optimization results, the evolution law of the motion state of the feature points and the correspondence with actual observations are determined, thus obtaining the motion state model of the feature points.

[0116] In one specific implementation, for feature points in a continuous frame task packet, a K-order neighborhood graph is constructed centered on the feature point. The K nearest neighbors around the feature point are selected as its neighbors. Typically, the value of K can be set between 8 and 12 to balance computational complexity and model stability. For example, for a video frame with a resolution of 1920×1080, when the feature point density is approximately 0.001 pixels per square pixel, a K value of 10 can achieve good results. After construction, the topological distance matrix between the feature point and its neighboring feature points is calculated. Each element in the distance matrix represents the Euclidean distance between the central feature point and its corresponding neighboring point, in pixels. Simultaneously, an angle difference matrix is ​​calculated, which records the angle formed between the central feature point and each pair of adjacent neighboring points. For example, for 10 neighboring points, the angle difference matrix contains 10 elements, each representing the angle formed between two adjacent neighboring points and the central point, in degrees. Furthermore, the feature point distribution density per unit area in the K-order neighborhood graph is obtained by dividing the number of feature points within the area covered by the neighborhood graph by the area of ​​that area. For example, if the neighborhood map covers an area of ​​200 square pixels and contains 15 feature points, then the distribution density is 0.075 points / square pixel.

[0117] Based on the obtained topological distance matrix, angle difference matrix, and feature point distribution density, local motion constraints are constructed. These constraints mainly consist of three parts: distance preservation constraints, angle preservation constraints, and density consistency constraints. The distance preservation constraint requires that the relative distance between a feature point and its neighbors remain essentially constant across consecutive frames, with an allowed variation range typically not exceeding 10%. The angle preservation constraint requires that the angle formed by a feature point and its neighbors change little across consecutive frames, typically allowing a variation range of ±5 degrees. The density consistency constraint requires that the feature point density variation in a local region conforms to the motion type; for example, for rigid body motion, the density variation should be less than 5%, while for non-rigid body motion, the upper limit of density variation can be relaxed to 15%. Subsequently, the affine transformation relationship of the feature points in the consecutive frame task packets is calculated. The affine transformation describes how a feature point transforms from one frame to the next, including scaling, rotation, and translation. Specifically, the least squares method is used to estimate the optimal affine transformation matrix based on the corresponding positions of the feature point and its neighbors in adjacent frames. The scale component, rotation component, and translation component are extracted from the affine transformation matrix. The scale component represents the scaling ratio of the feature point region, for example, from 1.0 to 1.05 represents a 5% magnification; the rotation component represents the rotation angle of the feature point region, such as a 3-degree clockwise rotation; the translation component represents the displacement vector of the feature point, such as a 5-pixel horizontal movement or a 2-pixel vertical movement.

[0118] The optimization objective is constructed from the degree of violation of local motion constraints and the residuals of the affine transformation relationship. The degree of violation refers to the difference between the actual observed feature point motion and the local motion constraints, which can be quantified using a weighted squared difference. For example, the distance constraint violation can be weighted at 0.6, the angle constraint violation at 0.3, and the density constraint violation at 0.1. The affine transformation residual is the Euclidean distance between the predicted feature point position using the affine transformation and the actual observed position; typically, the average residual should be less than 2 pixels. The optimization objective function is a weighted sum of the degree of violation and the residual, with a weight ratio of 7:3. The optimization objective is iterated using gradient descent. The initial learning rate of gradient descent is set to 0.01, gradually decreasing to 0.001 with each iteration. The number of iterations is typically set to 100 to 200, or terminated early when the objective function changes by less than 0.001. In practical cases, for a continuous frame task package containing 30 frames, the optimization process usually converges after 150 iterations, with the final objective function value reduced to less than 5% of the initial value. The optimization results determine the evolution of the motion state of feature points and their correspondence with actual observations. The evolution of motion state includes the feature point's trajectory, velocity changes, and acceleration characteristics. For example, feature points may exhibit linear motion (displacement proportional to time), accelerated motion (displacement proportional to the square of time), or periodic motion (displacement exhibiting sinusoidal characteristics). The correspondence with actual observations records the mapping between the actual and predicted positions of feature points in each frame, which can be used to evaluate tracking accuracy and correct subsequent predictions. The final feature point motion state model is a temporal prediction model capable of predicting the position and motion characteristics of feature points in future frames based on their historical motion states, achieving a prediction accuracy of over 95% in normal scenarios. This model can be used in applications such as feature point tracking, motion analysis, and anomaly detection.

[0119] Typically, feature point motion analysis techniques are mainly based on optical flow methods (such as the Lucas-Kanade algorithm), feature matching (such as SIFT and SURF matching), or Kalman filtering-based tracking methods. These methods have limitations when handling feature point motion analysis. Traditional optical flow methods mainly rely on the assumption of constant brightness, which easily fails in scenes with changing illumination or fast movement. Feature matching methods usually process each feature point independently, ignoring the topological relationships between feature points and the overall structural constraints. Filtering-based methods often use simplified motion models (such as uniform velocity or uniform acceleration models), which are difficult to adapt to complex nonlinear motion. Existing methods lack effective utilization of changes in the distribution density of feature points in local areas, resulting in poor performance in scenes with deformation and partial occlusion.

[0120] This embodiment constructs a feature point motion analysis framework based on local topology. By introducing a K-order neighborhood graph and multiple constraints, it more comprehensively describes and constrains the motion characteristics of feature points. The introduction of the K-order neighborhood graph structure considers not only the motion of individual feature points but also the overall motion pattern of their neighboring feature points, enhancing the ability to perceive changes in local structure. A multi-dimensional constraint system including a topological distance matrix, an angle difference matrix, and a distribution density is constructed, making feature point motion analysis more comprehensive and robust. Combining local motion constraints with affine transformations retains the simplicity of affine transformations while incorporating local structural information, improving the model's adaptability to complex motions. The use of a gradient descent-based optimization strategy enables the model to minimize prediction errors while satisfying local constraints, balancing local consistency and global accuracy.

[0121] like Figure 2 The diagram shown illustrates the flowchart for constructing the motion state model of feature points.

[0122] In one optional implementation, a local tensor field of feature points is constructed in the edge computing device and the potential energy value is obtained. A multi-level seed point is established to determine the feature point extraction sequence. Feature description information is extracted based on the tensor field gradient, a tensor feature sequence is generated, and a feature fusion matrix is ​​constructed, including:

[0123] In edge computing devices, local tensor descriptors between feature points are constructed, and the orientation parameters, scale parameters, and intensity parameters of the corresponding feature points are extracted. A tensor field is established, and the regional correlation and structural importance of feature points in the tensor field are calculated to obtain the potential energy value of the feature points.

[0124] The feature point with the largest potential energy value is selected as the first-level seed point. Multi-level seed points are determined based on the potential energy difference. The optimal traversal path is selected, and the position index information of the feature points on the optimal traversal path is recorded to generate the feature point extraction sequence.

[0125] Calculate the gradient distribution value of the tensor field, extract the first feature description information for regions where the gradient distribution value is greater than the preset gradient intensity threshold, and extract the second feature description information for regions where the gradient distribution value is less than the preset gradient intensity threshold, so as to obtain a multi-dimensional feature description of the feature points.

[0126] The feature extraction order is determined based on the feature point extraction sequence. The structural parameters and neighborhood data of the feature points are extracted, and tensor feature vectors are generated and organized according to the order of the feature point extraction sequence to form a tensor feature sequence.

[0127] Calculate the phase difference values ​​of adjacent feature vectors in the tensor feature sequence, construct the phase difference matrix and calculate the feature fusion weight coefficients to generate the feature fusion weight matrix. Use the feature fusion weight matrix to fuse the tensor feature sequence to obtain the feature fusion matrix.

[0128] In one specific implementation, the image data is preprocessed, including noise reduction, enhancement, and normalization. The preprocessed image data is input to a feature point detection module, which uses a differential filter to detect salient points in the image, such as corner points and edge points. For each detected feature point, its orientation parameter, scale parameter, and intensity parameter are extracted. The orientation parameter is obtained by calculating the principal direction of the pixel gradient in the neighborhood of the feature point, typically using an 8-directional interval division; the scale parameter is determined by the response value of the detected feature point in different scale spaces, with a typical value range of 1.0 to 4.0; the intensity parameter is obtained by calculating the magnitude of the response value at the feature point, representing the salience of the feature point.

[0129] Based on these parameters, local tensor descriptors are constructed around the feature points. These tensor descriptors are represented using second-order symmetric tensors and contain structural information about the feature points. The tensor field is constructed by calculating the pixel gradient covariance matrix within the neighborhood of each feature point, typically set to 15×15 pixels. For each feature point, its regional correlation with surrounding feature points is calculated. This correlation is based on a similarity measure between tensors, specifically using the tensor inner product method; a larger inner product value indicates a higher correlation. Furthermore, the structural importance of the feature points is calculated, determined by the eigenvalues ​​of the feature point's tensor. Greater differences in eigenvalues ​​indicate more significant structural information and higher importance. Combining regional correlation and structural importance, the potential energy value of the feature point is obtained. This potential energy value typically ranges from 0 to 1; a higher potential energy value indicates greater importance of the feature point within the image structure.

[0130] The feature point with the highest potential energy value is selected as the first-level seed point. For example, if a feature point has a potential energy value of 0.95, which is higher than other feature points, it is set as the first-level seed point. Multiple levels of seed points are determined based on the potential energy difference, with a threshold of 0.05. When the potential energy difference between a feature point and the previous-level seed point is greater than the threshold, that feature point is set as the next-level seed point. For example, when a feature point with a potential energy value of 0.89 is detected, its potential energy difference with the first-level seed point is 0.06, which is greater than the threshold of 0.05, so it is set as the second-level seed point. For the third-level seed point, its potential energy value is 0.83, and its potential energy difference with the second-level seed point is also 0.06, which is also greater than the threshold, so it is selected as the third-level seed point, and so on.

[0131] After determining the multi-level seed points, the optimal traversal path is selected. The optimal traversal path is determined based on the connection cost between seed points, which considers the spatial distance and potential energy difference between seed points. The system uses an improved minimum spanning tree algorithm to find the optimal path, with the distance weight coefficient in the connection cost calculation set to 0.7 and the potential energy difference weight coefficient set to 0.3. After the optimal path is determined, the position index information of the feature points on the path is recorded, generating a feature point extraction sequence, which contains the feature point indices arranged in the optimal order.

[0132] The gradient distribution values ​​of the tensor field are calculated by calculating the magnitude of the tensor gradient at each point in the tensor field. For regions where the gradient distribution value is greater than a preset gradient intensity threshold, the first descriptive information of the features is extracted. The preset gradient intensity threshold is set to 0.4. When the gradient distribution value of a region is greater than 0.4, the structural features of that region are extracted, including the principal direction, the secondary direction, and the direction intensity ratio. The principal direction represents the direction of the tensor's principal eigenvector, the secondary direction represents the direction of the tensor's secondary eigenvector, and the direction intensity ratio represents the ratio of tensor eigenvalues, reflecting the degree of anisotropy of the structure. For regions where the gradient distribution value is less than the preset gradient intensity threshold, the second descriptive information of the features is extracted, including the region's mean, variance, and entropy value. This descriptive information reflects the texture characteristics of the region. In this way, a multi-dimensional feature description of the feature points is obtained.

[0133] The feature extraction order is determined based on the feature point extraction sequence, and the structural parameters and neighborhood data of each feature point are extracted. The structural parameters include the tensor eigenvalues ​​and eigenvectors of the feature point, and the neighborhood data includes the pixel value distribution of the region surrounding the feature point. This information is organized into tensor feature vectors, each with a dimension of 256, containing a complete description of the feature point. These tensor feature vectors are then organized according to the order of the feature point extraction sequence to form a tensor feature sequence.

[0134] The phase difference between adjacent eigenvectors in the tensor feature sequence is calculated, obtained by calculating the angle between the eigenvectors. For example, the phase difference between the first and second eigenvectors is 15 degrees, and the phase difference between the second and third eigenvectors is 20 degrees. These phase difference values ​​are organized into a phase difference matrix, the size of which depends on the number of eigenvectors. Based on the phase difference matrix, feature fusion weight coefficients are calculated. The weight coefficients are inversely proportional to the phase difference; the smaller the phase difference, the larger the weight coefficient. A feature fusion weight matrix is ​​generated, and this matrix is ​​used to perform weighted fusion of the tensor feature sequence to obtain the feature fusion matrix. The feature fusion matrix contains a comprehensive description of the feature points and can be used for subsequent image matching and recognition tasks.

[0135] In one optional implementation, the feature point with the largest potential energy value is selected as the first-level seed point. Multi-level seed points are determined based on the potential energy difference. An optimal traversal path is selected, and the position index information of the feature points on the optimal traversal path is recorded. The generated feature point extraction sequence includes:

[0136] In the tensor field, the feature point with the largest potential energy value is determined as the first-level seed point. A traversal window is established with the first-level seed point as the center, and the potential energy difference of the feature points within the traversal window is calculated.

[0137] Select the feature point with the smallest potential energy difference as the second-level seed point, and record the traversal direction and traversal distance from the first-level seed point to the second-level seed point.

[0138] In the traversal direction, a search sector is determined, and according to a preset number of preferred features, a corresponding number of feature points in the search sector, sorted from high to low by potential energy value, are selected as third-level seed points.

[0139] Using the third-level seed point as a node, connect them to form a candidate traversal path, and calculate the potential energy gradient of adjacent feature points on the candidate traversal path.

[0140] The candidate traversal path with the smallest potential gradient is selected as the optimal traversal path. The position index information of the feature points is recorded along the optimal traversal path to generate a feature point extraction sequence.

[0141] In one specific implementation, tensor field data to be processed is acquired. This tensor field can be multi-dimensional data such as image data, video data, and audio data. For each feature point, its potential energy value is calculated. The calculation of the potential energy value can be based on characteristics such as the gradient, curvature, and entropy of the feature point. For example, for image data, the Harris corner detection algorithm can be used to calculate the potential energy value of each pixel; for audio data, the energy distribution after short-time Fourier transform can be used as the potential energy value.

[0142] After calculating the potential energy value, the feature point with the largest potential energy value is determined as the first-level seed point. Assuming that in a 100×100 two-dimensional tensor field, the feature point with coordinates (35, 42) is calculated to have the largest potential energy value of 0.95, then this point is set as the first-level seed point.

[0143] A traversal window is established centered on the first-level seed point. The size of the traversal window can be dynamically adjusted according to the complexity of the tensor field. For example, a 15×15 window can be set to cover 225 feature points around the first-level seed point. Within this window, the potential energy difference between each feature point and the first-level seed point is calculated. The potential energy difference can be calculated as the absolute difference between the potential energy values ​​of the feature points and the potential energy values ​​of the first-level seed point.

[0144] After calculating the potential energy difference between all feature points within the window and the first-level seed point, the feature point with the smallest potential energy difference is selected as the second-level seed point. For example, assuming the feature point with coordinates (30, 40) within the window has a potential energy value of 0.92 and a potential energy difference of 0.03 with the first-level seed point, which is the smallest among all feature points, it is set as the second-level seed point.

[0145] Record the traversal direction and distance from the first-level seed point to the second-level seed point. The traversal direction can be calculated from the coordinate difference between the two points. In the example above, the traversal direction from the first-level seed point (35, 42) to the second-level seed point (30, 40) is (-5, -2), and the traversal distance is sqrt(5). 2 +2 2 = sqrt(29) ≈ 5.4 units.

[0146] The search sector is determined based on the traversal direction from the first-level seed point to the second-level seed point. The search sector is a region formed by extending to both sides at a certain angle, with the second-level seed point as the vertex and the traversal direction as the axis of symmetry. For example, the angle of the search sector can be set to 60 degrees, meaning it extends 30 degrees to each side of the traversal direction as the center.

[0147] Within the search sector, a predetermined number of feature points are selected as third-level seed points, ranked from highest to lowest potential energy value. For example, if the predetermined number of selections is 3, then the three feature points with the highest potential energy values ​​within the search sector are selected as third-level seed points. Assume the coordinates of these three points are (25, 38), (28, 35), and (22, 37), with potential energy values ​​of 0.88, 0.85, and 0.82, respectively.

[0148] Using the third-level seed points as nodes, connect them to form candidate traversal paths. Since there are 3 third-level seed points, 3 candidate traversal paths can be formed: a path from the first-level seed point to the second-level seed point and then to the third-level seed point (25, 38); a path from the first-level seed point to the second-level seed point and then to the third-level seed point (28, 35); and a path from the first-level seed point to the second-level seed point and then to the third-level seed point (22, 37).

[0149] Calculate the potential energy gradient of adjacent feature points on each candidate traversal path. The potential energy gradient represents the rate of change of potential energy values ​​on the path, which can be calculated by dividing the difference in potential energy values ​​between adjacent feature points by the distance between the two points. For each candidate traversal path, calculate the potential energy gradient from the first-level seed point to the second-level seed point, and the potential energy gradient from the second-level seed point to the third-level seed point, and then take the average as the total potential energy gradient of the path.

[0150] For example, the potential gradient of the first path is: [(0.95-0.92) / 5.4+(0.92-0.88) / sqrt((30-25)] 2 +(40-38) 2 )] / 2=[0.0056+0.0080] / 2=0.0068. Similarly, calculate the potential gradient for other paths.

[0151] The candidate traversal path with the smallest potential gradient is selected as the optimal traversal path. Assuming the potential gradient of the second path is 0.0062 and the potential gradient of the third path is 0.0075, the second path has the smallest potential gradient and is selected as the optimal traversal path.

[0152] Record the position index information of feature points along the optimal traversal path to generate a feature point extraction sequence. In this example, the position indices of feature points on the optimal traversal path are: (35, 42), (30, 40), (28, 35).

[0153] Through the above steps, the process of determining multi-level seed points based on potential energy differences, selecting the optimal traversal path, and generating a feature point extraction sequence is completed. This method can effectively extract representative feature point sequences from tensor fields, providing support for subsequent data processing and analysis.

[0154] Feature point extraction techniques typically include algorithms such as SIFT, SURF, and ORB. These algorithms detect feature points based on local gradient information or the statistical properties of image patches. However, these methods have some limitations when dealing with complex tensor fields. They usually evaluate and extract feature points independently, lacking consideration of the relationships between feature points. Traditional methods often employ fixed thresholds or a fixed number of feature point selection strategies, making it difficult to adapt to data of varying complexity. In multidimensional data processing, these methods struggle to effectively capture the topological relationships and potential energy changes between feature points.

[0155] This embodiment establishes a multi-level seed point traversal mechanism based on potential energy difference. By considering the potential energy gradient relationship between feature points, it adaptively constructs a feature point extraction sequence. Introducing the concept of potential energy difference ensures that feature point selection considers not only absolute potential energy values ​​but also relative changes, enhancing sensitivity to local structural changes. A multi-level seed point strategy (first level, second level, and third level) forms a hierarchical feature point selection mechanism, effectively capturing structural relationships in the tensor field. The search sector is determined based on the traversal direction, and the optimal traversal path is selected using the potential energy gradient minimization criterion, enabling the extracted feature point sequence to better reflect the essential characteristics of the tensor field.

[0156] like Figure 3 The diagram shown illustrates the network structure of the feature point extraction algorithm.

[0157] The distributed real-time facial feature extraction system based on edge computing according to embodiments of the present invention includes:

[0158] The first unit is used to perform time-series sampling of the face image data collected by the terminal device to obtain a continuous multi-frame image sequence.

[0159] The second unit is used to establish an edge computing resource pool that includes multiple edge computing devices.

[0160] The third unit is used to construct a spatiotemporal feature association map for a series of consecutive multi-frame image sequences, extract the facial motion trajectory, predict the facial position by combining the recursive Bayesian method, correct it by using the reprojection error compensation method, and generate a spatiotemporally aligned optimized facial image sequence.

[0161] The fourth unit is used to divide the optimized face image sequence into continuous frame task packages, allocate the continuous frame task packages using a coloring and marking method, establish a feature point motion prediction mechanism, and allocate the continuous frame task packages to edge computing devices to perform feature extraction tasks.

[0162] The fifth unit is used to construct the local tensor field of feature points in the edge computing device and obtain the potential energy value, establish multi-level seed points to determine the feature point extraction sequence, extract feature description information based on the tensor field gradient, generate tensor feature sequence, and construct feature fusion matrix.

[0163] The sixth unit is used to perform time-series verification of the feature fusion matrix generated by each edge computing device, select the optimal feature fusion matrix according to the preset mutual information threshold, and send the optimal feature fusion matrix to the terminal device.

[0164] A third aspect of the present invention provides an electronic device, comprising:

[0165] processor;

[0166] Memory used to store processor-executable instructions;

[0167] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0168] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0169] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed real-time facial feature extraction method based on edge computing, characterized in that, include: The face image data collected by the terminal device is sampled in time sequence to obtain a continuous multi-frame image sequence; Establish an edge computing resource pool that includes multiple edge computing devices; A spatiotemporal feature association map is constructed from a series of consecutive image frames to extract the facial motion trajectory. The face position is predicted by combining the recursive Bayesian method and corrected by the reprojection error compensation method to generate a spatiotemporally aligned optimized face image sequence. The optimized face image sequence is divided into continuous frame task packages, and a coloring and marking method is used to allocate the continuous frame task packages. A feature point motion prediction mechanism is established, and the continuous frame task packages are allocated to edge computing devices to perform feature extraction tasks. In edge computing devices, a local tensor field of feature points is constructed and potential energy values ​​are obtained. Multi-level seed points are established to determine the feature point extraction sequence. Feature description information is extracted based on the tensor field gradient, generating a tensor feature sequence. A feature fusion matrix is ​​constructed, including: In edge computing devices, local tensor descriptors between feature points are constructed, and the orientation parameters, scale parameters, and intensity parameters of the corresponding feature points are extracted. A tensor field is established, and the regional correlation and structural importance of feature points in the tensor field are calculated to obtain the potential energy value of the feature points. The feature point with the largest potential energy value is selected as the first-level seed point. Multi-level seed points are determined based on the potential energy difference. The optimal traversal path is selected, and the position index information of the feature points on the optimal traversal path is recorded to generate the feature point extraction sequence. Calculate the gradient distribution value of the tensor field, extract the first feature description information for regions where the gradient distribution value is greater than the preset gradient intensity threshold, and extract the second feature description information for regions where the gradient distribution value is less than the preset gradient intensity threshold, so as to obtain a multi-dimensional feature description of the feature points. The feature extraction order is determined based on the feature point extraction sequence. The structural parameters and neighborhood data of the feature points are extracted, and tensor feature vectors are generated and organized according to the order of the feature point extraction sequence to form a tensor feature sequence. Calculate the phase difference values ​​of adjacent feature vectors in the tensor feature sequence, construct the phase difference matrix and calculate the feature fusion weight coefficients to generate the feature fusion weight matrix, and use the feature fusion weight matrix to fuse the tensor feature sequence to obtain the feature fusion matrix. The feature fusion matrix generated by each edge computing device is verified in a time sequence. The optimal feature fusion matrix is ​​selected according to the preset mutual information threshold and then sent to the terminal device.

2. The method according to claim 1, characterized in that, A spatiotemporal feature association map is constructed from a sequence of consecutive images to extract the facial motion trajectory, including: For each frame in a continuous multi-frame image sequence, a set of feature points is extracted, the spatial distance between each feature point and its surrounding feature points is calculated, the influence radius of the Gaussian kernel function is adaptively determined based on the local distribution density of the feature points, and the spatial distance is weighted using the Gaussian kernel function to obtain the local density value of each feature point. For each feature point, the spatial gradient is calculated based on the local density value. The density gradient field pointing towards the direction of increasing local density value is obtained. The cosine of the angle between the density gradient direction and the direction of the line connecting the feature points is calculated. The spatial distance is then weighted using a Gaussian kernel function to obtain the spatial correlation strength between feature points. The ratio of the local density values ​​of corresponding feature points in adjacent frames is calculated as the density transfer ratio of the feature points, and the temporal correlation between feature points in adjacent frames is established based on the density transfer ratio. A spatiotemporal feature association graph is constructed using feature points as vertices, spatial association strength as intra-frame edge weights, and density transfer ratio as inter-frame edge weights. In the spatiotemporal feature association graph, a density flow field is constructed based on the density gradient field and the density transfer ratio. The streamline trajectory is iteratively calculated along the intra-frame edge weights and inter-frame edge weights of the spatiotemporal feature association graph to obtain the face motion trajectory.

3. The method according to claim 2, characterized in that, In the spatiotemporal feature association map, a density flow field is constructed based on the density gradient field and the density transfer ratio. The streamline trajectory is iteratively calculated along the intra-frame edge weights and inter-frame edge weights of the spatiotemporal feature association map to obtain the face motion trajectory, including: The first direction vector of the density gradient field is extracted from the spatiotemporal feature correlation map, the density transfer ratio is mapped to the second direction vector, and the first and second direction vectors are fused by adaptive weight to construct the density flow field. In the density flow field, feature points are used as the initial points of streamlines. The motion direction of feature points in the spatial domain is determined by the intra-frame edge weights of the spatiotemporal feature correlation graph, and the motion direction of feature points in the temporal domain is determined by the inter-frame edge weights. The feature point positions are iteratively updated until convergence, and streamline trajectories are generated. Starting from the region with the highest feature point density, streamline trajectories are traced in the forward and reverse directions in the density flow field to obtain the motion trajectory sequence of the feature point group. Curve fitting is then performed on the motion trajectory sequence to obtain the face motion trajectory.

4. The method according to claim 1, characterized in that, The optimized face image sequence is divided into consecutive frame task packages. A color-coding method is used to allocate these consecutive frame task packages. A feature point motion prediction mechanism is established. The consecutive frame task packages are then allocated to edge computing devices to perform feature extraction tasks, including: Obtain the time interval of each frame in the optimized face image sequence, calculate the motion displacement and motion direction of the image frame based on the time interval, and divide the continuous frame images into continuous frame task packages of adaptive size based on the motion displacement and motion direction. The number of feature points and the area of ​​the spatial distribution region of each frame image in the continuous frame task package are counted. The ratio of the number of feature points to the area of ​​the spatial distribution region is calculated, and the ratio is used as the correlation strength between the continuous frame task packages. Adjacent consecutive frame task packets are colored based on their association strength. Adjacent consecutive frame task packets with an association strength higher than a preset strength threshold are marked with different colored tags. The number of colored tags is equal to the number of edge computing devices, thus obtaining the colored tag for each consecutive frame task packet. A motion state model of feature points is established for consecutive frame task packets with the same coloring mark. Based on the motion state model, the migration direction of feature points between consecutive frame task packets with different coloring marks is predicted, and feature point detection is performed in the region corresponding to the migration direction of the feature points. Consecutive frame task packets with the same coloring mark are assigned to corresponding edge computing devices. Each edge computing device performs parallel feature extraction tasks based on the feature point migration direction. The number of frames in the consecutive frame task packets is dynamically adjusted according to the processor utilization of the edge computing devices.

5. The method according to claim 4, characterized in that, Establishing a motion state model for feature points in consecutive frame task packets with the same coloring label includes: For feature points in a continuous frame task packet, a K-order neighborhood graph is constructed with the feature points as the center. The topological distance matrix and angle difference matrix between the feature points and their neighboring feature points are calculated to obtain the feature point distribution density per unit area in the K-order neighborhood graph. Based on the topological distance matrix, the angle difference matrix, and the feature point distribution density, local motion constraints are constructed. The affine transformation relationship of the feature points in the continuous frame task packet is calculated, and the scale component, rotation component, and translation component of the affine transformation relationship are extracted. The degree of violation of the local motion constraints and the residual of the affine transformation relationship are used as optimization objectives. The optimization objectives are iterated using the gradient descent method. Based on the optimization results, the evolution law of the motion state of the feature points and the correspondence with actual observations are determined, thus obtaining the motion state model of the feature points.

6. The method according to claim 1, characterized in that, The feature point with the largest potential energy value is selected as the first-level seed point. Multi-level seed points are determined based on the potential energy difference. An optimal traversal path is selected, and the position index information of the feature points on the optimal traversal path is recorded. The resulting feature point extraction sequence includes: In the tensor field, the feature point with the largest potential energy value is determined as the first-level seed point. A traversal window is established with the first-level seed point as the center, and the potential energy difference of the feature points within the traversal window is calculated. Select the feature point with the smallest potential energy difference as the second-level seed point, and record the traversal direction and traversal distance from the first-level seed point to the second-level seed point. In the traversal direction, a search sector is determined, and according to a preset number of preferred features, a corresponding number of feature points in the search sector, sorted from high to low by potential energy value, are selected as third-level seed points. Using the third-level seed point as a node, connect them to form a candidate traversal path, and calculate the potential energy gradient of adjacent feature points on the candidate traversal path. The candidate traversal path with the smallest potential gradient is selected as the optimal traversal path. The position index information of the feature points is recorded along the optimal traversal path to generate a feature point extraction sequence.

7. A distributed real-time facial feature extraction system based on edge computing, used to implement the method of any one of claims 1-6, characterized in that, include: The first unit is used to perform time-series sampling of the face image data collected by the terminal device to obtain a continuous multi-frame image sequence. The second unit is used to establish an edge computing resource pool that includes multiple edge computing devices. The third unit is used to construct a spatiotemporal feature association map for a series of consecutive multi-frame image sequences, extract the facial motion trajectory, predict the facial position by combining the recursive Bayesian method, correct it by using the reprojection error compensation method, and generate a spatiotemporally aligned optimized facial image sequence. The fourth unit is used to divide the optimized face image sequence into continuous frame task packages, allocate the continuous frame task packages using a coloring and marking method, establish a feature point motion prediction mechanism, and allocate the continuous frame task packages to edge computing devices to perform feature extraction tasks. The fifth unit is used to construct the local tensor field of feature points in the edge computing device and obtain the potential energy value, establish multi-level seed points to determine the feature point extraction sequence, extract feature description information based on the tensor field gradient, generate tensor feature sequence, and construct feature fusion matrix. The sixth unit is used to perform time-series verification of the feature fusion matrix generated by each edge computing device, select the optimal feature fusion matrix according to the preset mutual information threshold, and send the optimal feature fusion matrix to the terminal device.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images