A mobile terminal 3D human posture estimation method and system based on cooperative processing

Abstract

This invention relates to the field of computer vision, and more particularly to a mobile 3D human pose estimation method and system based on collaborative processing. The method involves acquiring three consecutive frames: the current frame and its adjacent preceding and following frames. Feature extraction is performed on the current frame at a first depth, and on the preceding and following frames at a second depth, where the second depth is less than the first depth. Based on the extraction results, the features extracted from the three frames are fused to obtain image features with fused temporal information. These fused temporal information image features are input into a pre-defined staged localization model. The model sequentially performs first-stage localization and second-stage localization, and outputs the execution results. Based on the output results, the initial position and fine-tuning are processed to determine the coordinates of key points in the 3D human pose. The input is into a two-stage localization model employing a coarse-to-fine strategy; this collaborative mechanism balances the conflict between accuracy and speed.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a mobile 3D human pose estimation method and system based on collaborative processing. Background Technology

[0002] Mobile 3D human pose estimation is a computer vision technology specifically designed for mobile devices with limited computing resources, such as smartphones and tablets. Its core objective is to rapidly and stably infer the three-dimensional spatial coordinates of human joints from continuous video frames, using a standard RGB camera built into the device, while meeting real-time and low-power requirements.

[0003] First, existing mobile 3D pose estimation solutions face a fundamental contradiction in utilizing temporal information to improve stability. One mainstream approach uses pure single-frame estimation, which, while computationally efficient, completely severs the inter-frame correlation, inevitably resulting in random jitter in the output and poor smoothness in videos, severely impacting the user experience. Another approach attempts to incorporate temporal information, such as using recurrent networks or 3D convolutions to process consecutive frames, but this typically constructs a model with significantly higher computational and memory overhead, making it difficult to meet the stringent real-time and power consumption requirements of mobile devices.

[0004] Secondly, at the implementation level, existing technologies have further limitations in terms of the efficiency of collaboration between different subtasks or modules within the model, as well as the system's adaptability to external dynamic environments. On the one hand, in order to balance multiple performance indicators (such as accuracy and speed) under mobile constraints, existing designs often struggle to achieve efficient and close collaboration between different parts of the model. This may lead to insufficiently intelligent resource allocation, or, when dealing with complex situations, the optimization goals of different modules may interfere with each other, limiting further improvements in overall performance. On the other hand, facing the ever-changing motion states and application requirements in real-world scenarios, the processing logic of existing methods is usually pre-defined and lacks flexibility. This makes it difficult for the system to dynamically adjust its behavior strategy according to the actual situation, resulting in suboptimal performance in some scenarios or inefficient utilization of computational resources. Summary of the Invention

[0005] To address one or more problems in the prior art, the main objective of this application is to provide a mobile 3D human pose estimation method and system based on collaborative processing.

[0006] To achieve the aforementioned objectives, this application proposes a mobile 3D human pose estimation method based on collaborative processing, the method comprising:

[0007] Acquire three consecutive frames of image consisting of the current frame and its adjacent frames before and after it;

[0008] The current frame is subjected to feature extraction at a first depth, and the preceding and following frames are subjected to feature extraction at a second depth, wherein the second depth is less than the first depth;

[0009] Based on the extracted results, the features extracted from the three frames of images are fused to obtain image features with fused temporal information;

[0010] The image features fused with temporal information are input into a preset stage localization model. The first stage localization and the second stage localization are executed sequentially through the stage localization model, and the execution results are output. The first stage localization is used to output the initial position of each key point based on the image features. The second stage localization is used to output a fine adjustment amount to correct the initial position based on the initial position.

[0011] Based on the output execution results, the initial position and fine adjustment amount are processed to determine the coordinates of key points of 3D human pose.

[0012] This application also provides a mobile 3D human pose estimation system based on collaborative processing, including:

[0013] The temporal image acquisition module is used to acquire three consecutive frames of images consisting of the current frame and its adjacent preceding and following frames;

[0014] An asymmetric feature extraction module is used to perform feature extraction at a first depth on the current frame and feature extraction at a second depth on the preceding and following frames, wherein the second depth is less than the first depth.

[0015] The temporal feature fusion module is used to fuse the features extracted from three frames of images based on the extraction results, so as to obtain image features with fused temporal information;

[0016] The stage localization module is used to input the image features fused with temporal information into a preset stage localization model, and to perform first-stage localization and second-stage localization sequentially through the stage localization model, and output the execution results; the first-stage localization is used to output the initial position of each key point based on the image features; the second-stage localization is used to output a fine adjustment amount for correcting the initial position based on the initial position.

[0017] The posture coordinate determination module is used to process the initial position and fine adjustment amount based on the output execution results to determine the coordinates of key points of 3D human posture.

[0018] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0019] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0020] This application's embodiment of the mobile 3D human pose estimation method and system based on collaborative processing solves the long-standing core problem of balancing accuracy, speed, and stability in mobile 3D human pose estimation through a systematic collaborative architecture. First, by using three frames—the current frame and its adjacent frames—as input, crucial short-term temporal context information is injected into the model from the source, laying a data foundation to overcome the inherent jitter problem in single-frame estimation. Second, asymmetric feature extraction at different depths is performed on the current frame and the preceding and following frames, minimizing additional computational overhead while introducing necessary temporal information, thus ensuring real-time processing capabilities under the constraints of mobile computing power. Furthermore, the extracted temporal features are fused and input into a two-stage localization model employing a "coarse-to-fine" strategy. This model first performs rapid low-resolution global coarse localization, and then performs high-precision fine-tuning in parallel within multiple extremely small local regions based on the coarse localization results. This collaborative mechanism eliminates the need for intensive high-precision calculations across the entire image, balancing the trade-off between accuracy and speed. Finally, by synthesizing the coarse localization coordinates and the fine offset, a high-precision and spatiotemporally smooth 3D pose is output. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a mobile 3D human pose estimation method based on collaborative processing according to an embodiment of this application.

[0022] Figure 2 This is a flowchart illustrating a mobile 3D human pose estimation method based on collaborative processing according to an embodiment of this application.

[0023] Figure 3 This is a schematic block diagram of a mobile 3D human pose estimation system based on collaborative processing according to an embodiment of this application;

[0024] Figure 4 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] Reference Figure 1 This application provides a mobile 3D human pose estimation method based on collaborative processing, the method comprising:

[0028] S1. Obtain three consecutive frames of image consisting of the current frame and its adjacent frames before and after it;

[0029] S2. Perform feature extraction at a first depth on the current frame and feature extraction at a second depth on the previous and next frames, wherein the second depth is less than the first depth;

[0030] S3. Based on the extraction results, the features extracted from the three frames of images are fused to obtain image features with fused temporal information;

[0031] S4. Input the image features fused with the time-series information into a preset stage localization model, and sequentially execute the first stage localization and the second stage localization through the stage localization model, and output the execution results; the first stage localization is used to output the initial position of each key point based on the image features; the second stage localization is used to output a fine adjustment amount for correcting the initial position based on the initial position.

[0032] S5. Based on the output execution results, process the initial position and fine adjustment amount to determine the coordinates of the key points of the 3D human body pose.

[0033] As described in steps S1-S3 above, step one introduces basic temporal context information into the model. By simultaneously acquiring the current frame and its immediate preceding and following frames, the system can obtain motion cues within a very short time window. The preceding and following frames provide the trajectory of minute changes in the current human posture over time, which helps the model understand the instantaneous motion trend and direction of the joints. Embedding temporal information from the data input lays the foundation for subsequent modules to transition from static image understanding to dynamic sequence understanding. This helps overcome the output jitter problem that inevitably occurs in pure single-frame models due to information loss, providing original information to ensure the stability of posture sequences. Step two is the core manifestation of "asymmetric temporal coding". Since the current frame is the target frame for the final posture output, a deeper network is used for fine feature extraction to retain rich spatial details and semantic information. For the preceding and following frames, their main value lies in providing temporal supplementation and reference information, so a shallower and faster network is used for lightweight feature extraction. This differentiated processing strategy of "one deep and many shallow" is an intelligent allocation of computing resources based on task importance. While effectively incorporating temporal context information, it minimizes the additional computational overhead caused by the increase in the number of frames (from a single frame to three frames). It solves the problem of a surge in computational load caused by simply stacking multiple frames or using uniform depth processing, making it possible to process multi-frame input in real time on mobile devices with limited computing power, achieving a clever balance between "introducing temporal information" and "controlling computational load." Step three, after completing the differential feature extraction, integrates the feature maps from the current frame, the previous frame, and the next frame into a unified composite feature representation rich in spatiotemporal information through feature fusion operations (such as channel concatenation or weighted summation). This fused feature not only includes the fine appearance of the current frame but also embeds short-term motion cues encoded by features from previous and next frames, enhancing the feature representation of spatiotemporal context. This feature serves as input to the subsequent localization model, enabling it to reason within a continuous microscopic time segment rather than relying solely on isolated static images, thus significantly improving robustness to motion blur, transient occlusion, and other conditions.

[0034] As described in steps S4-S5 above, step four implements a "coarse-to-fine" collaborative localization mechanism. The first-stage localization involves a fast, low-resolution global search. It processes the entire fused feature map, outputting a low-spatial-resolution probability heatmap. Each peak in the heatmap corresponds to a general area of ​​a keypoint. This localization is fast, but its accuracy is limited by the heatmap's resolution. The second-stage localization involves precise, local, high-resolution fine-tuning. Guided by the coarse location output from the first stage, it crops multiple small local feature regions centered on that location from the fused feature map. Each small region is specifically used to refine a keypoint, predicting the precise sub-pixel offset of that point relative to the coarse location within this extremely small area. The two stages are tightly coupled; the output of the previous stage directly determines the areas that the next stage needs to focus on. This decomposes the time-consuming problem of high-precision full-map search into two efficient steps: "fast global coarse localization" and "parallel local refinement." This avoids the huge overhead of intensive computation directly on the original high-resolution feature map, which is crucial for achieving high-precision estimation on mobile devices. The two stages have clear division of labor and work collaboratively. Step five is the synthesis stage of the two-stage localization results. The principle is to algebraically add the discretized, grid-aligned coarse position coordinates provided in the first stage to the continuous, fine offset vectors predicted in the second stage. This synthesis operation combines the discrete position information provided by the low-resolution heatmap with the continuous sub-pixel correction information predicted by the local network. This yields keypoint coordinates with both high spatial accuracy and sub-pixel level detail. The coarse position provides stability and global consistency, while the fine offset provides subtle localization capabilities that are difficult to achieve with a single heatmap. It should be noted that "keypoints" are not arbitrary points of interest in the image, but specifically refer to human joints or feature areas with clear anatomical and kinematic definitions. The set of these points constitutes a simplified human skeletal model, including core joints such as the head and neck, shoulders, elbows, wrists, hips, knees, and ankles. The fundamental principle of accurately estimating the positions of these points in three-dimensional space lies in the ability to uniquely and digitally define and reconstruct the human body's posture, orientation, and limb movements through their spatial connections. Each keypoint coordinate carries semantic information describing the human body's morphology and motion state. Outputting these specific keypoint coordinates gives the calculation results of this method direct and universal application value. They form the cornerstone of downstream high-level task understanding; for example, the lines between joints can generate skeletal animation, the angles between joints can be used for action recognition, and the 3D trajectories of joints can be used for motion analysis.

[0035] As mentioned above, this scheme does not choose between "single-frame jitter" and "multi-frame sluggishness," but instead constructs a new path through the core technology of "asymmetric temporal feature encoding." While inevitably introducing three frames of temporal information to eradicate jitter, differentiated deep extraction ensures that the computational increment is kept to an extremely low level, thus ensuring that speed and stability are achieved simultaneously at the architectural level. Regarding the limitations of "internal model efficiency and external adaptability": this scheme achieves efficient allocation of computational resources within the model through a "two-stage collaborative localization from coarse to fine" mechanism. It transforms the massive global precise search problem into a combination of global coarse search and multiple micro-local fine-tuning problems, representing a qualitative leap in efficiency. This not only guarantees the coexistence of high accuracy and high speed in principle, but its structured design also ensures that accuracy improvement no longer depends on simply increasing the network burden.

[0036] Reference Figure 2 In one embodiment, the first-stage localization and second-stage localization performed sequentially through the stage localization model include the following steps:

[0037] S41. The image features of the fused temporal information are processed by the first stage positioning to generate a heat map of the spatial probability distribution of key points, and the response position of each key point in the heat map constitutes the initial position of the key point.

[0038] S42. Based on the second stage positioning, for the initial position of each key point in the heat map, determine the local feature region corresponding to the key point in the image features, and predict the three-dimensional spatial offset of the key point from the initial position in the local feature region.

[0039] S43. By combining the initial position of each key point with the three-dimensional spatial offset, the coordinates of the 3D human pose key point of that key point are determined.

[0040] As described above, in Step 1, the core principle of the first-stage localization is to transform the continuous coordinate regression problem into probability distribution prediction in a discrete space. The network receives image features fused with temporal information and outputs a three-dimensional probability matrix for each predetermined human keypoint through operations such as convolution and upsampling. The value at each spatial location in this matrix represents the probability that the keypoint exists in this three-dimensional voxel. By finding the maximum response point in the probability matrix corresponding to each keypoint, the most likely coarse spatial grid location of the keypoint can be determined. This step is essentially a low-resolution global spatial sampling and pattern recognition. It transforms the complex direct coordinate prediction into a more stable and easier-to-learn heatmap classification problem. The heatmap provides a global probabilistic landscape of keypoint locations and has a certain robustness to noise and local occlusion. It provides crucial prior information for the second stage, narrowing the infinite search space to a clear, discrete candidate location, which is a prerequisite for subsequent efficient refinement. Step 2, the second-stage localization, involves high-resolution local feature matching and micro-regression. Using the initial grid positions of each keypoint provided by the first-stage heatmap as navigation coordinates, a small cubic feature region centered at these coordinates is extracted from the original high-dimensional fused image features through back-mapping. This local feature region contains the most relevant appearance and contextual information near the keypoint. Subsequently, a lightweight sub-network focuses on analyzing this local feature block and regresses a three-dimensional continuous vector. This vector accurately represents the subtle distance difference between the true location of the keypoint and its grid center point in the X, Y, and Z dimensions. This achieves sub-pixel or sub-voxel level localization accuracy, overcoming the quantization error ceiling inherent in the first-stage heatmap due to spatial resolution limitations. By focusing on extremely small local regions, this step can utilize limited network capacity to capture extremely subtle local image evidence, such as edges and texture boundaries, thereby making millimeter-level corrections to the precise center of the keypoint. This design decomposes the huge computational cost of full-image refining into multiple parallelizable micro-refining tasks, resulting in a significant improvement in efficiency. Step three performs a deterministic coordinate reconstruction. It algebraically adds the initial position coordinates, based on a discrete grid, obtained in the first stage, to the continuous offset vector predicted in the second stage. Specifically, the initial position provides the integer part or basic dimension of the coordinates, while the offset provides the fractional part or fine dimension. The synthesis of the two mathematically completes a precise reconstruction from discrete probabilistic representation to continuous geometric coordinates. The final output combines robustness and high accuracy. The initial position, derived from an understanding of the global scene, has good stability; the fine offset, derived from the discrimination of local features, has high accuracy. The synthesis operation seamlessly combines the advantages of both, ensuring that the final coordinates are neither coarse due to the resolution limitations of the heatmap nor easily affected by local interference or ambiguity due to purely local regression.This is the final result of the two-stage collaborative work, and its accuracy far exceeds the level that either stage could achieve independently.

[0041] In this embodiment, traditional thinking assumes that higher accuracy necessarily requires more complex and slower full-map computation. However, this solution decouples the two approaches, allowing a "fast network" to handle large-scale searches (efficient elimination) and a "small network" to handle small-scale decisions (efficient and accurate), thus achieving a non-linear efficiency improvement at the system level. This establishes a virtuous cycle of error suppression. The heatmap from the first stage provides strong priors for the second stage, greatly reducing the ambiguity and failure risk of local searches; the refinement results from the second stage, in turn, verify and correct the coarse estimation quality of the first stage. The two form a synergistic relationship of mutual correction and reinforcement, rather than a simple concatenation. This gives the model resilience against the "risk of losing key information." Even if asymmetric encoding weakens features in certain details between frames, the first stage can still provide a roughly correct region based on global context and temporal fusion features. As long as the approximate region is correct, the second stage can use its powerful local feature analysis capabilities to "find" the precise location within that region.

[0042] In one embodiment, after the step of determining the coordinates of key points in the 3D human pose, the method further includes:

[0043] The coordinates of the 3D human pose key points determined by multiple consecutive frames of images are low-pass filtered to obtain the first filtering result;

[0044] Kalman filtering is applied to the coordinates of key points of 3D human pose determined by multiple consecutive frames of images to obtain the second filtering result;

[0045] Analyze the motion change data of key point coordinates, determine adaptive weights based on the motion change data of key point coordinates, and perform weighted filtering on the key point coordinates based on the adaptive weights to obtain a third filtering result;

[0046] The first filtering result, the second filtering result, and the third filtering result are fused together to output smoothed attitude data.

[0047] As described above, the first step is frequency cutoff in signal processing. The time-varying coordinate sequence of human keypoints is considered a time-domain signal composed of a mixture of real motion signals and high-frequency noise. By setting a cutoff frequency, the filter allows signal components with frequencies below this threshold to pass through primarily; these typically correspond to intentional, smooth, slow human movements. Conversely, it strongly suppresses signal components with frequencies above this threshold; these typically correspond to high-frequency spikes caused by model prediction errors, image noise, or minor jitter. This effect can be achieved using moving averages or digital filters. This effectively filters out rapid, irrational high-frequency jitter in the pose sequence, making the motion trajectory visually smoother. This is particularly effective in eliminating random errors in single model predictions and is a direct means of improving basic visual stability. The second step is Bayesian optimal estimation based on a kinematic model. It treats human motion as a dynamic system and predicts the keypoint position at the next moment by establishing a simple motion model of keypoint velocity, acceleration, and other state variables. Simultaneously, it treats the keypoint coordinates output in the preceding steps as noisy "observations." Kalman filtering, through its two core steps of prediction and update, continuously and optimally fuses model predictions with actual observations. It not only utilizes historical information to smooth current observations but also makes short-term predictions of the system's future state, giving the attitude sequence physical plausibility and coherence. It can compensate for transient information loss caused by model processing delays or motion blur, making the output trajectory more consistent with real kinematics. For uniform or uniformly accelerated motion, it provides smoother estimates ahead of observations, significantly reducing lag. Step three involves dynamically assessing motion uncertainty and applying differentiated smoothing strength. First, it analyzes the historical sequence of keypoint coordinates in real time, calculating motion change data such as instantaneous velocity or inter-frame displacement. Based on the magnitude of this data, one or more weighting coefficients are dynamically calculated. Its core logic is: when motion changes drastically, it is assumed that the human body is performing a rapid, intentional action, and the filtering strength should be reduced to avoid motion lag and distortion caused by over-smoothing; when motion changes are gentle or nearly static, it is assumed that high-frequency components are more likely noise, and the filtering strength should be increased to achieve the ultimate stabilization effect. Weighted filtering modulates the original coordinates or intermediate results based on these dynamic weights. This resolves the fundamental contradiction of fixed-parameter filtering in dynamic scenarios. It achieves intelligent smoothing strategies, enabling the system to dynamically and adaptively switch between "maintaining motion sensitivity" and "pursuing ultimate smoothness," thus simultaneously meeting the processing requirements for both fine micro-motions and large, rapid movements. Step four involves the application of ensemble learning in signal processing. It acknowledges the inherent limitations of single filtering strategies: low-pass filtering may lose details of fast movements, Kalman filtering may introduce bias when the model mismatches, and the weight calculation itself in adaptive filtering fluctuates.By assigning fusion weights to the first, second, and third filtering results respectively, and combining them linearly or nonlinearly, the fusion weights can be fixed empirical values ​​or fine-tuned based on the evaluation of the confidence of the current filter outputs. The fused output sequence inherits the smooth basis of low-pass filtering, incorporates the physical prior of Kalman filtering, and maintains the agile response of adaptive filtering to motion states, ultimately outputting a highly stable and natural attitude data stream that is superior across multiple evaluation dimensions.

[0048] It is worth mentioning that low-pass filtering, as a fundamental smoothing unit with extremely low computational cost, is retained. Finally, through an intelligent fusion step, the system can strategically allocate the weight of the three results. For example, when resources are extremely limited, it can be configured to prioritize the low-pass filtering result; when pursuing high dynamic performance, the roles of adaptive and Kalman filtering can be highlighted. This structure itself possesses good scalability and trade-off capabilities.

[0049] In one embodiment, the heatmap of the spatial probability distribution of key points has a 28×28×24 three-dimensional tensor form; wherein, the spatial dimension of the three-dimensional tensor is 28×28×28, the channel dimension is 24, and each channel in the channel dimension corresponds to a human key point.

[0050] As mentioned above, the heatmap has a spatial dimension of 28×28×28, which defines a fine-grained spatial discretization and probability quantization strategy. The spatial information of the input image is first encoded into image features that incorporate temporal information. The first-stage localization network performs a series of downsampling and convolution operations on these features, ultimately mapping each spatial location (corresponding to a 16×16 pixel local region in the original image) to a 28×28 two-dimensional grid point, achieving the first spatial dimensionality reduction. The introduction of the third dimension "28" is based on the principle of discretizing and modeling depth or scale information. It is not a direct physical depth, but an abstract depth level representation learned by the network, used to encode the distance of keypoints relative to the camera or scene, the possible hierarchical relationships between different limbs, and the probability distribution of 3D positions at low resolution. In this way, the spatial position probability of each keypoint is modeled within a 28×28×28 three-dimensional discrete volume. The network is explicitly designed to have 24 independent output channels, each channel being a complete 28×28×28 three-dimensional probability volume. The principle is to assign each channel a specific task to learn and predict a predefined keypoint of the human body with specific semantics. This one-to-one forced mapping allows the network to be specifically optimized during training for the unique appearance features, movement patterns, and contextual relationships of each keypoint. The network parameters of different channels can focus on capturing the characteristics of different joints. For example, the channel responsible for predicting the wrist will focus more on the fine texture of the hand area and its connection with the elbow, while the channel responsible for predicting the hip will focus on the core area of ​​the torso.

[0051] In one embodiment, determining the local feature region corresponding to the key point in the image features includes:

[0052] Based on the initial position, a local feature block is determined in the image features of the fused temporal information; the local feature block corresponds to a 16×16 pixel region centered at the initial position in the space of the input image.

[0053] As described above, determining local feature blocks based on the initial location is a spatial attention-guided feature selection process. The first-stage localization process outputs the response location with the highest probability in the heatmap, which is a grid coordinate in a 28×28×28 discrete space. This step back-projects these coordinates onto the previously generated fused temporal image feature map with higher spatial semantic density through a fixed spatial mapping relationship. Based on the projected center coordinates, a fixed-size 3D feature sub-block is cropped or indexed from this feature map. The essence of this operation is to use the low-resolution semantic information provided in the first stage about the approximate location of key points to activate and extract the high-resolution image features in the second stage about the precise location of key points, achieving precise navigation from coarse-grained semantics to fine-grained evidence. The local feature block corresponds to a 16×16 pixel region in the input image, based on a fixed conversion relationship between the spatial resolution of the first-stage heatmap and the resolution of the original input image. Given an input image of 448×448 pixels and a heatmap spatial size of 28×28, it can be calculated that one grid cell of the heatmap corresponds to a 16×16 pixel square region in the input image. Therefore, when the initial position of the heatmap is taken as the center, the physical area covered by the input image is naturally a 16×16 pixel region centered on this position. This design explicitly limits the search range of the second-stage refinement task to a clearly defined physical neighborhood that matches the upper limit of the coarse localization error in the first stage.

[0054] In one embodiment, the three-dimensional spatial offset is a three-dimensional vector, and the three components of the three-dimensional vector represent the coordinate correction of the key point in the three spatial dimensions of X, Y and Z, respectively.

[0055] As mentioned above, the 3D spatial offset is a 3D vector representing the minute spatial displacement of a keypoint from its initial position to its true position. It is modeled as a vector with direction and length in 3D Euclidean space. This vector fully represents all the spatial information required for correction. In the neural network implementation, the second-stage local refinement network is designed with a fixed number of 3 output neurons in the regression layer. Each neuron directly outputs a scalar value, and these three scalar values ​​sequentially or correspondingly constitute the three components of the 3D vector. These three components represent the coordinate correction of the keypoint in the X, Y, and Z spatial dimensions, respectively, using Cartesian coordinate decomposition fully aligned with the final 3D pose coordinate system. The total displacement of the keypoint in 3D space is orthogonally decomposed along three mutually perpendicular reference axes. The X component represents the left-right offset correction in the horizontal direction of the image, the Y component represents the up-down offset correction in the vertical direction, and the Z component represents the near-far offset correction in the depth direction perpendicular to the imaging plane. This decomposition method gives the correction amount in each dimension independent physical meaning. This design forms an extremely concise, efficient, and mathematically rigorous coordinate generation pipeline. It avoids complex coordinate transformations or nonlinear mappings, reducing the steps involved in introducing computational errors.

[0056] In one embodiment, the method further includes, in the step of performing a second depth feature extraction on the preceding and following frames:

[0057] For each frame in the preceding and following frames, shallow feature extraction and mid-level feature extraction are performed in parallel to obtain shallow feature maps and mid-level feature maps, respectively.

[0058] Calculate the correlation coefficient between the shallow feature map and a certain intermediate layer feature map during the deep feature extraction process of the current frame;

[0059] Based on the correlation coefficient graph, different weights are applied to features at different locations in the middle layer feature graph;

[0060] The weighted mid-layer feature map is fused with the shallow-layer feature map to obtain the final output feature of the frame.

[0061] As mentioned above, the first step involves multi-layered information preservation and efficiency stratification. The strategy of performing only a single shallow extraction on consecutive frames is abandoned in favor of a dual-path parallel encoding architecture. The shallow feature extraction network remains lightweight, responsible for capturing basic low-level visual features such as edges and textures, ensuring computational efficiency. Simultaneously, a new parallel network branch with moderate depth and capacity is added to perform mid-level feature extraction. This branch can learn and retain richer semantic information and more discriminative local patterns, which are crucial for understanding motion deformation and occlusion boundaries. Through parallel execution, the system generates two feature representations with different information densities for each frame without significantly increasing the time consumed by serial computation. This fundamentally expands the information capacity of consecutive frames. The mid-level features, as an effective supplement to the shallow features, retain key details that might be filtered out by rapid dimensionality reduction operations, providing more sufficient feature basis for subsequent judgments on whether the frame contains important temporal events. The second step is to establish spatiotemporal attention associations across frames. A reference feature map is selected from an intermediate layer feature map in the deep feature extraction process of the current frame. This feature map contains both the semantic content of the current frame and its implicit spatial structure. By calculating the spatial correlation between the shallow feature maps of the preceding and following frames and this reference feature map, a correlation coefficient map can be generated. Each value in the coefficient map quantifies the degree of matching or correlation strength between the features at a certain position in the preceding and following frames and the corresponding features at the current frame. Essentially, it uses the fast shallow features of the preceding and following frames as proxies to evaluate the similarity or continuity of each local region with the current frame, thereby inferring which regions may have changed significantly or remained stable. An adaptive, data-driven attention map is generated. This correlation coefficient map is not a fixed template, but dynamically reflects the real spatiotemporal dependency between the preceding and following frames and the current frame, accurately indicating potential regions where motion, occlusion, or appearance changes occur, providing intelligent guidance signals for subsequent feature selection. Step three is guided feature importance selection. It uses the correlation coefficient map obtained in the previous step as a set of spatial weights and applies them element-wise to the corresponding intermediate layer feature maps. The core logic is that regions with high relevance indicate consistency between the current and previous frames, potentially belonging to a static background or a uniformly moving section, where the reliability and importance of the mid-level features are high. Conversely, regions with low relevance suggest potential critical changes, where the difference information encoded by the mid-level features becomes crucial. The system can dynamically adjust the intensity of features at various locations in the mid-level feature map based on a preset mapping strategy, such as directly using high relevance as enhancement weights or reversing low relevance as attention weights. This achieves intelligent purification and enhancement of mid-level feature information. Through relevance-guided weighting, the system adaptively strengthens the truly valuable parts of the mid-level features—whether stable context or critical change information—while suppressing potentially redundant or interfering parts. This ensures that the final feature representation prepared for the previous and previous frames focuses on the visual evidence most contributing to pose inference in the current frame.Step four involves layered feature complementarity and integration. The weighted, modulated mid-level features now carry attention-selected key information. These are fused with the shallow features, which serve as the basis and maintain complete spatial sampling. The fusion operation can be channel concatenation or element-wise addition, aiming to combine broad, fundamental shallow spatial information with focused, enhanced mid-level semantic information to form a unified and more expressive composite feature map. This produces a significantly improved frame feature representation. This final feature retains shallow details of the entire image to ensure spatial integrity while highlighting the mid-level semantics of key regions dynamically relevant to the current frame. As the output representing the preceding and following frames in asymmetric coding, it greatly improves the temporal feature quality input to subsequent fusion modules.

[0062] In this embodiment, the existing inter-frame asymmetric processing strategy, widely adopted in mobile 3D pose estimation schemes to balance temporal inputs and computational constraints, presents a technical contradiction: the active and uniform feature simplification of preceding and following reference frames fundamentally conflicts with the instantaneous, sparsity, and locality of key temporal events in dynamic videos, such as motion initiation, occlusion occurrence / end, and rapid deformation. This uniform simplification strategy distributes computational resources evenly across all regions of the entire frame. However, the temporal information crucial for pose estimation—for example, motion vectors generated within a few frames of a joint just beginning to move, pixel-level changes in the edges of a limb from occlusion to reappearance, or localized blurring caused by high-speed motion—is highly concentrated in both time and space. During the significant dimensionality reduction and compression of feature maps, these high-value sparse signals are easily weakened or even lost along with a large amount of low-value static or redundant background information.

[0063] This solution, by introducing dual-path parallel extraction, first retains more information at the source; then, through relevance-guided weighting, it intelligently identifies and strengthens valuable parts in each frame. This is equivalent to equipping each frame with a "feature quality inspector," ensuring that redundancy is lost and essential information is transmitted, thus completely reversing the risk of information loss. By calculating the relevance with the current frame, this embodiment explicitly models short-term spatiotemporal relationships during the feature extraction stage. This enables the system to automatically "detect" which regions in previous and subsequent frames are motion origins, occlusion boundaries, or deformation key points. This capability is directly encoded into the output features, making subsequent temporal feature fusion and pose estimation models more capable of capturing and understanding dynamic events.

[0064] In one feasible embodiment, before the step of acquiring three consecutive frames consisting of the current frame and its adjacent preceding and following frames, the method further includes:

[0065] The first current frame and its neighboring first reference frame are obtained from the video stream at an initial sampling interval;

[0066] Calculate the image difference features between the first current frame and the first reference frame;

[0067] Based on the image difference features, the posture jitter coefficient in the current motion scene is analyzed; wherein, the posture jitter coefficient is positively correlated with the magnitude of the image difference features;

[0068] Based on the analysis results of the attitude jitter coefficient, the frame sampling interval used for the final attitude estimation is dynamically adjusted and determined; wherein, the larger the attitude jitter coefficient, the smaller the determined frame sampling interval.

[0069] Three frames are captured from the video stream according to the adjusted frame sampling interval.

[0070] As described above, step one establishes a lightweight pre-analysis mechanism. Instead of directly using the final three frames, the system quickly acquires two representative frames from a short video segment at a lower initial sampling frequency (e.g., intervals of one or two frames). The first current frame can be considered an approximation of the target time, while the first reference frame provides a short-term baseline. This approach performs a low-cost "scene reconnaissance" before the main algorithm runs, providing raw data for subsequent dynamic decision-making. Step two involves motion perception and change quantification. By comparing the two frames at the pixel level or feature level, their differences are calculated. This can be achieved by calculating the optical flow field amplitude, the statistical characteristics of the inter-frame difference map, or by comparing the differences between the feature maps extracted by the lightweight network. Essentially, this quantifies the intensity and spatial distribution of visual content changes between the two frames, thus objectively measuring the scene's motion intensity. It transforms the subjective perception of "motion speed" into a calculable, objective numerical feature. This difference feature forms the basis for subsequent judgments, directly reflecting the dynamic level of the video stream over a short period. Step three involves establishing a mapping model from underlying image changes to high-level task risks. The attitude jitter coefficient is a predictive metric used to assess the risk level of jitter or instability in the final attitude output sequence under the current motion intensity, provided a fixed sampling strategy is used. The analysis logic is: the larger the image difference features, the more intense the scene motion, which leads to greater inter-frame attitude displacement and poorer motion continuity captured by the model under a fixed sampling interval, thus resulting in a higher predicted attitude jitter risk. A predefined or lightweight learned mapping function is used to convert the difference feature values ​​into continuous or discrete jitter coefficients. This enables predictive judgment of potential output quality. It elevates image-level changes to a performance prediction level directly related to the core task objective, giving the decision-making process a clear goal orientation. Step four is the core decision-making step of adaptive control. It implements a negative feedback adjustment strategy: when the predicted attitude jitter coefficient is large, it indicates intense motion, requiring higher temporal resolution to capture details, thus reducing the sampling interval; conversely, when the predicted jitter coefficient is small, it indicates smooth motion, allowing for an increase in the sampling interval to reduce redundancy. The decision-making logic is based on a trade-off between "information density" and "computational efficiency," aiming to allocate an appropriate temporal sampling rate for the current scene. This achieves intelligent and optimized sampling strategies. It breaks the rigid pattern of fixed intervals, enabling the system to dynamically allocate resources according to the real-time scene, fundamentally solving the inadequacy of fixed windows when facing different motion speeds. Step five, based on the sampling interval optimized for the current scene determined in the previous step, re-acquires three frames of images from the video stream for the main processing flow. This ensures that the temporal density of the data input to the core pose estimation model matches the dynamic characteristics of the current scene, providing tailored input data for the core algorithm.Whether it's high-speed action or a static scene, the temporal distribution of the input frames has been optimized, thus laying the best data foundation for the entire system to achieve the best balance between accuracy, speed and stability from the source.

[0071] It's worth noting that the "fixed three-frame temporal window" strategy may be fundamentally incompatible with dynamic and ever-changing real-world scenarios. Specifically, for high-speed, large-displacement movements, the three-frame time span is too short, resulting in insufficient capture of intermediate motion states or discontinuous information. This makes it difficult for the model to reconstruct a smooth, physically plausible motion trajectory, easily leading to prediction lag or trajectory breaks. In slow-moving or nearly static scenes, three consecutive frames contain a large amount of repetitive information. The fixed window forces redundant calculations and fusion, causing unnecessary computational overhead and energy consumption, failing to achieve optimal energy efficiency.

[0072] In this embodiment, to address the issue of the three-frame window being static, this solution transforms the sampling interval from a fixed parameter into a variable dynamically calculated from the real-time scene. This completely resolves the contradiction of "insufficient sampling" leading to blurry modeling under high-speed motion with a fixed window, and "over-sampling" leading to computational redundancy under static scenes, achieving universal adaptability to complex and variable motion patterns. It realizes a paradigm shift from "blind processing" to "foresight": traditional processes directly process input data, remaining "blindly ignorant" of the data's temporal characteristics. This solution adds a lightweight pre-analysis stage, calculating image differences and mapping them to "pose jitter coefficients," enabling the system to anticipate potential output quality risks from the current input before formal processing and adjust the input strategy accordingly. This proactive adjustment based on forward assessment is more fundamental and efficient than a posteriori filtering based on output results.

[0073] Reference Figure 3 This application also provides a mobile 3D human pose estimation system based on collaborative processing, comprising:

[0074] The temporal image acquisition module 1 is used to acquire three consecutive frames of images consisting of the current frame and its adjacent preceding and following frames;

[0075] Asymmetric feature extraction module 2 is used to perform feature extraction at a first depth on the current frame and feature extraction at a second depth on the previous and next frames, wherein the second depth is less than the first depth;

[0076] The temporal feature fusion module 3 is used to fuse the features extracted from the three frames of images based on the extraction results to obtain image features with fused temporal information;

[0077] The stage localization module 4 is used to input the image features fused with temporal information into a preset stage localization model, and to perform first-stage localization and second-stage localization sequentially through the stage localization model, and output the execution results; the first-stage localization is used to output the initial position of each key point based on the image features; the second-stage localization is used to output a fine adjustment amount for correcting the initial position based on the initial position.

[0078] The posture coordinate determination module 5 is used to process the initial position and fine adjustment amount based on the output execution results to determine the coordinates of key points of 3D human posture.

[0079] As described above, it is understood that each component of the mobile 3D human pose estimation system based on collaborative processing proposed in this application can achieve the function of any of the mobile 3D human pose estimation methods based on collaborative processing described above, and the specific structure will not be repeated.

[0080] Reference Figure 4 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores monitoring data and other data. The network interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a mobile 3D human pose estimation method based on co-processing.

[0081] The processor described above executes the mobile 3D human pose estimation method based on collaborative processing, including: acquiring three consecutive frames of images consisting of the current frame and its adjacent preceding and following frames; performing feature extraction at a first depth on the current frame and at a second depth on the preceding and following frames, wherein the second depth is less than the first depth; fusing the features extracted from the three frames based on the extraction results to obtain image features with fused temporal information; inputting the image features with fused temporal information into a preset stage localization model, performing first-stage localization and second-stage localization sequentially through the stage localization model, and outputting the execution results; the first-stage localization is used to output the initial position of each key point based on the image features; the second-stage localization is used to output a fine adjustment amount for correcting the initial position based on the initial position; and processing the initial position and fine adjustment amount based on the output execution results to determine the coordinates of the 3D human pose key points.

[0082] One embodiment of this application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements a mobile 3D human pose estimation method based on co-processing, including the following steps: acquiring three consecutive frames of images consisting of the current frame and its adjacent preceding and following frames; performing feature extraction at a first depth on the current frame and at a second depth on the preceding and following frames, wherein the second depth is less than the first depth; fusing the features extracted from the three frames based on the extraction results to obtain image features with fused temporal information; inputting the image features with fused temporal information into a preset stage localization model, performing first-stage localization and second-stage localization sequentially through the stage localization model, and outputting the execution results; the first-stage localization is used to output the initial position of each key point based on the image features; the second-stage localization is used to output a fine adjustment amount for correcting the initial position based on the initial position; and processing the initial position and fine adjustment amount based on the output execution results to determine the coordinates of the 3D human pose key points.

[0083] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0084] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0085] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A mobile 3D human pose estimation method based on collaborative processing, characterized in that, The method includes: Acquire three consecutive frames of image consisting of the current frame and its adjacent frames before and after it; The current frame is subjected to feature extraction at a first depth, and the preceding and following frames are subjected to feature extraction at a second depth, wherein the second depth is less than the first depth; Based on the extracted results, the features extracted from the three frames of images are fused to obtain image features with fused temporal information; The image features fused with temporal information are input into a preset stage localization model. The stage localization model sequentially performs first-stage localization and second-stage localization, and outputs the execution results. The first-stage localization is used to output the initial position of each key point based on the image features. The second-stage localization is used to output a fine adjustment amount to correct the initial position based on the initial position. Based on the output execution results, the initial position and fine adjustment amount are processed to determine the coordinates of the key points of the 3D human pose. The step of sequentially performing the first-stage localization and the second-stage localization using the staged localization model includes: The image features of the fused temporal information are processed by the first-stage localization to generate a heat map of the spatial probability distribution of key points. The response positions of each key point in the heat map constitute the initial positions of the key points. Based on the second-stage positioning, for the initial position of each key point in the heat map, the local feature region corresponding to the key point is determined in the image features, and the three-dimensional spatial offset of the key point from the initial position is predicted within the local feature region. The coordinates of the 3D human pose key point are determined by combining the initial position of each key point with the three-dimensional spatial offset. The heatmap of the spatial probability distribution of key points includes 24 channels, each channel has a spatial dimension of 28×28×28, and each channel corresponds to a human key point. Determining the local feature region corresponding to the key point in the image features includes: Based on the initial position, a local feature block is determined in the image features of the fused temporal information; the local feature block corresponds to a 16×16 pixel region centered on the initial position in the space of the input image.

2. The mobile 3D human pose estimation method based on collaborative processing according to claim 1, characterized in that, After the step of determining the coordinates of key points in the 3D human pose, the method further includes: The coordinates of the 3D human pose key points determined by multiple consecutive frames of images are low-pass filtered to obtain the first filtering result; Kalman filtering is applied to the coordinates of key points of 3D human pose determined by multiple consecutive frames of images to obtain the second filtering result; Analyze the motion change data of key point coordinates, determine adaptive weights based on the motion change data of key point coordinates, and perform weighted filtering on the key point coordinates based on the adaptive weights to obtain a third filtering result; The first filtering result, the second filtering result, and the third filtering result are fused together to output smoothed attitude data.

3. The mobile 3D human pose estimation method based on collaborative processing according to claim 1, characterized in that, The three-dimensional spatial offset is a three-dimensional vector, and the three components of the three-dimensional vector represent the coordinate correction of the key point in the three spatial dimensions of X, Y and Z, respectively.

4. The mobile 3D human pose estimation method based on collaborative processing according to claim 1, characterized in that, In the step of performing second-depth feature extraction on the preceding and following frames, the method further includes: For each frame in the preceding and following frames, shallow feature extraction and mid-level feature extraction are performed in parallel to obtain shallow feature maps and mid-level feature maps, respectively. Calculate the correlation coefficient between the shallow feature map and a certain intermediate layer feature map during the deep feature extraction process of the current frame; Based on the correlation coefficient graph, different weights are applied to features at different locations in the middle layer feature graph; The weighted mid-layer feature map is fused with the shallow-layer feature map to obtain the final output feature of the frame.

5. A mobile 3D human pose estimation system based on collaborative processing, used in the method described in any one of claims 1-4, characterized in that, include: The temporal image acquisition module is used to acquire three consecutive frames of images consisting of the current frame and its adjacent preceding and following frames; An asymmetric feature extraction module is used to perform feature extraction at a first depth on the current frame and feature extraction at a second depth on the preceding and following frames, wherein the second depth is less than the first depth. The temporal feature fusion module is used to fuse the features extracted from three frames of images based on the extraction results, so as to obtain image features with fused temporal information; The stage localization module is used to input the image features fused with temporal information into a preset stage localization model, and to perform first-stage localization and second-stage localization sequentially through the stage localization model, and output the execution results; the first-stage localization is used to output the initial position of each key point based on the image features; the second-stage localization is used to output a fine adjustment amount for correcting the initial position based on the initial position. The posture coordinate determination module is used to process the initial position and fine adjustment amount based on the output execution results to determine the coordinates of key points of 3D human posture.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images