A human motion posture detection method and system

Abstract

The application relates to the technical field of posture detection, and discloses a human motion posture detection method and system. The application performs spherical projection and multi-view block splitting on a panoramic RGB video stream, and outputs space-time features. A first detection network is combined with multi-view block to output spherical coordinates and confidence of a joint. A second detection network outputs spherical coordinates and a feature vector of a root node, and combines screened block features to complete depth regression and three-dimensional position calculation. The three-dimensional position of the root node is taken as a reference, the spherical coordinates of the joint, a preset bone length and bone topological constraints are combined to perform inverse kinematics calculation, and an initial three-dimensional position of the joint is obtained. Low-confidence joints are introduced into a time sequence prediction network, the position sequence of historical high-confidence joints is combined to complete position correction, and then the position is verified through bone topological constraint checking and time sequence smoothing processing. Finally, the output motion posture is improved in terms of spatial rationality and time sequence continuity.

Description

Technical Field

[0001] This application relates to the field of posture detection technology, and specifically to a method and system for detecting human motion posture. Background Technology

[0002] Human motion posture detection is a core technology in fields such as computer vision, human-computer interaction and motion analysis. To meet the needs of human motion capture from a panoramic perspective, the industry has gradually introduced spherical projection technology to spatially map panoramic images in order to eliminate the blind spots of planar imaging.

[0003] In existing pose detection solutions for panoramic RGB videos, regional distortion caused by spherical projection is not effectively adapted; joint detection does not combine multi-view detection information for fusion and filtering; when solving joint positions using inverse kinematics, fixed preset bone lengths are used, which do not adapt to spherical projection distortion and changes in relative joint positions, making it easy for the three-dimensional position of the joints to deviate from the human physiological structure; position correction is achieved by simply merging poses, which easily leads to joint position conflicts and temporal pose jumps, and the output motion poses cannot meet the detection requirements of complex panoramic motion scenes in terms of spatial rationality and temporal continuity.

[0004] Based on the shortcomings of the existing technology, the technical problem to be solved by this application is how to achieve accurate and stable detection of human motion posture in panoramic complex motion scenes. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to provide a human motion posture detection method and system that effectively solves the problems of distortion interference, inaccurate feature representation, large positioning deviation and poor posture continuity, and has higher detection accuracy and scene adaptability in panoramic human motion posture detection.

[0006] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides a method for detecting human motion posture, including: The panoramic RGB video stream is acquired, and each frame image is spherically projected and split into multi-view tiles. Spherical convolution is performed on the multi-view tiles, and temporal convolution is performed after sparse sampling of multiple consecutive frames of images. The spatiotemporal features are then output. The spatiotemporal features are input into the first detection network, and the spherical coordinates and confidence scores of the joints in the projected coordinate system are output. The spherical coordinates with the highest confidence scores are selected from the multi-view tiles at the same time as the spherical coordinates of the corresponding joints. The spatiotemporal features are input into the second detection network, which outputs the spherical coordinates of the root node and the corresponding feature vector. The depth of the root node is estimated by the depth regression network, and the three-dimensional position of the root node is calculated by combining the corresponding spherical coordinates. Based on the three-dimensional position of the root node, combined with the spherical coordinates of the joint and the topological constraints of the bone, as well as the dynamic bone length obtained by distortion adaptation correction of the preset bone length, the initial three-dimensional position of the joint is solved by inverse kinematics. The initial 3D positions of low-confidence joints in the current frame and the 3D position sequences of high-confidence joints in historical frames are input into the temporal prediction network to obtain the corrected 3D positions. After skeletal topological constraints, these positions are merged with the initial 3D positions of the high-confidence joints and then temporally smoothed to output the motion pose of the current frame.

[0007] Furthermore, the integrated output spatiotemporal features include: Based on the position of multi-view tiles in the projected coordinate system, the spatial features obtained by spherical convolution are weighted to obtain weighted spatial features. The image after sparse sampling of multiple consecutive frames is segmented according to the inter-frame motion amplitude, and the corresponding temporal convolution is used to extract features for different motion segments to obtain segmented temporal features. Based on the timestamp and projection coordinate system, the weighted spatial features and segmented temporal features are aligned. Weight allocation and feature fusion are performed through a gated fusion unit. The fused features are then corrected by topological normalization according to the spherical projection curvature distribution, and spatiotemporal features are output.

[0008] Furthermore, the multi-view image patch is obtained by splitting the image after spherical projection of each frame in the panoramic RGB video stream according to a preset polar angle interval and azimuth angle interval; the weighting processing is to construct a spherical distortion weight matrix based on the polar angle and azimuth angle corresponding to the multi-view image patch, and to weight the spatial features; the motion segment includes a fast motion segment and a slow motion segment, and the fast motion segment is used for feature extraction by dilated temporal convolution, while the slow motion segment is used for feature extraction by ordinary temporal convolution.

[0009] Furthermore, the spherical coordinates and confidence level of the output joint in the projected coordinate system include: The spatiotemporal features are decomposed into tile spatiotemporal features, which are input into the first detection network and the network is parsed and output as the initial spherical coordinates, initial confidence and feature response values ​​of all candidate joints in the corresponding tile. Candidate joints with feature response values ​​less than the response threshold are removed, and the initial spherical coordinates of the remaining candidate joints are calibrated for distortion compensation. The calibrated spherical coordinates and their corresponding initial confidence scores are summarized in all multi-view tiles of the same joint at the same time. The initial confidence scores are corrected based on the spatial overlap of the calibrated spherical coordinates to obtain the confidence score of the joint. The calibrated spherical coordinates with the highest confidence scores are selected as the spherical coordinates of the joint.

[0010] Furthermore, the first detection network includes branch detection units corresponding to multi-view tiles, the branch detection units being adapted to the projection distortion degree of the corresponding tiles; the initial spherical coordinates are calibration coordinates in the projection coordinate system.

[0011] Furthermore, the calculation of the three-dimensional position of the root node includes: The spatiotemporal features of the map tiles are input into the second detection network, which outputs the spherical coordinates of the root node and the corresponding feature vector, and filters out the spatiotemporal features of the target map tiles that match the feature vector. The spatiotemporal features of the target map are input into a depth regression network for depth estimation to obtain the root node depth and depth confidence. The root node depth is then weighted and calibrated, and the spherical coordinates of the root node are calibrated for distortion compensation. Based on the mapping relationship between the projected coordinate system and the Cartesian coordinate system, and combined with the calibrated spherical coordinates and the calibrated root node depth, the three-dimensional position of the root node is calculated.

[0012] Furthermore, the deep regression network includes regression branches corresponding to multi-view tiles, and the regression branches are adapted to the projection distortion degree of the corresponding target tiles; tiles whose spatiotemporal features and feature vectors have a similarity greater than a similarity threshold are selected as spatiotemporal features of the target tiles; Root node depths with depth confidence less than a preset threshold are removed. The spatial fit between the remaining root node depths and the spherical coordinates of the root nodes is calculated. The spatial fit and depth confidence are weighted and fused together to obtain the weight, and then the weighted sum is obtained to obtain the calibrated root node depth.

[0013] Furthermore, the initial three-dimensional position of the joint is determined by: A hierarchical topology is constructed according to the human skeleton hierarchy. Hierarchical weights are assigned based on the confidence of joints. Inverse kinematics calculations are performed hierarchically based on dynamic bone length and spherical coordinates of joints to obtain joint positions. The initial three-dimensional position of the joint is obtained by performing skeletal topology constraint verification and abnormal position correction on the joint positions at different levels.

[0014] Furthermore, the output motion pose of the current frame includes: Based on the confidence level of the joints, high-confidence joints and low-confidence joints are divided. The initial 3D position of the low-confidence joints in the current frame and the 3D position sequence of the high-confidence joints in the historical frames are extracted. The results are input into the temporal prediction network and the corrected 3D position of the low-confidence joints is predicted according to the hierarchical topology. Skeletal topological constraints are applied to the corrected 3D position and distortion compensation calibration is performed. Dynamic weights are constructed based on the joint confidence to weight and merge the initial 3D position of high-confidence joints with the calibrated corrected 3D position. The merged joint 3D positions are subjected to global topology consistency verification and anomaly correction. Segmented temporal smoothing is performed in combination with segmented temporal features to output the motion pose of the current frame.

[0015] Secondly, this application provides a human motion posture detection system, comprising: The processing module is used to acquire panoramic RGB video streams, spherically project each frame of image and split it into multi-view tiles, perform spherical convolution on the multi-view tiles, perform temporal convolution after sparse sampling of multiple consecutive frames of images, and output spatiotemporal features in combination. The detection module is used to input spatiotemporal features into the first detection network and output the spherical coordinates and confidence of the joint in the projected coordinate system. The spherical coordinates with the highest confidence in the multi-view tiles at the same time are selected as the spherical coordinates of the corresponding joint. The localization module is used to input spatiotemporal features into the second detection network, output the spherical coordinates of the root node and the corresponding feature vector, estimate the depth of the root node through the depth regression network, and calculate the three-dimensional position of the root node by combining the corresponding spherical coordinates. The solver module is used to solve the initial three-dimensional position of the joint by using the three-dimensional position of the root node as a reference, combining the spherical coordinates of the joint and the topological constraints of the bone, as well as the dynamic bone length obtained by distortion adaptation correction of the preset bone length. The fusion module is used to input the initial 3D position of the low-confidence joints in the current frame and the 3D position sequence of the high-confidence joints in the historical frames into the temporal prediction network to obtain the corrected 3D position. After skeletal topological constraints, it is merged with the initial 3D position of the high-confidence joints and then output the motion pose of the current frame after temporal smoothing.

[0016] Compared with the prior art, the beneficial effects achieved by this application are as follows: This application performs spherical projection and multi-view tile segmentation on a panoramic RGB video stream, extracts spatial features by combining spherical convolution, performs differential temporal convolution according to motion characteristics, and outputs spatiotemporal features to eliminate detection interference caused by distortion and motion speed differences from the feature source. The first detection network combines multi-view tile information to output the spherical coordinates and confidence of the joints, and selects high-confidence coordinates as the final result, thereby reducing the probability of single-view false detection and false negative detection, and improving the accuracy of joint positioning and the rationality of confidence judgment.

[0017] The second detection network outputs the spherical coordinates and feature vectors of the root node, and completes depth regression and 3D position calculation. Based on the calibrated spherical coordinates, a 3D mapping is completed to improve the accuracy of the 3D positioning of the root node and provide a stable and reliable spatial reference for subsequent joint position calculation. Using the 3D position of the root node as the reference, inverse kinematics calculation is performed by combining the spherical coordinates of the joint, the bone parameters adapted to the distortion, and the bone topological constraints, so that the initial 3D position of the joint conforms to the physiological structure of the human body and the actual projected spatial relationship.

[0018] By introducing low-confidence joints into a temporal prediction network and combining them with the 3D position sequence of historical high-confidence joints to complete position correction, and then performing skeletal topology constraint verification and temporal smoothing processing, the position deviation and posture temporal jump problems of low-confidence joints are eliminated. The output motion posture is significantly improved in terms of spatial rationality and temporal coherence. The overall technical solution is adapted to the imaging characteristics of panoramic RGB video and complex human motion scenes, effectively solving the problems of distortion interference, inaccurate feature representation, large positioning deviation and poor posture coherence. It has higher detection accuracy and scene adaptability in panoramic human motion posture detection. Attached Figure Description

[0019] Figure 1 A flowchart of a method for detecting human motion posture; Figure 2 The logic flowchart for outputting the motion pose of the current frame; Figure 3 This is a schematic diagram of a human motion posture detection system. Detailed Implementation

[0020] The technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of this application, rather than limitations thereof.

[0021] Example 1: like Figure 1 As shown, this embodiment provides a method for detecting human motion posture, including: S1. Obtain the panoramic RGB video stream, spherically project each frame image and split it into multi-view tiles, perform spherical convolution on the multi-view tiles, perform temporal convolution after sparse sampling of multiple consecutive frames, and output the spatiotemporal features.

[0022] Specifically, the comprehensive output spatiotemporal features include: Based on the position of multi-view tiles in the projected coordinate system, the spatial features obtained by spherical convolution are weighted to obtain weighted spatial features. The image after sparse sampling of multiple consecutive frames is segmented according to the inter-frame motion amplitude, and the corresponding temporal convolution is used to extract features for different motion segments to obtain segmented temporal features. Based on the timestamp and projection coordinate system, the weighted spatial features and segmented temporal features are aligned. Weight allocation and feature fusion are performed through a gated fusion unit. The fused features are then corrected by topological normalization according to the spherical projection curvature distribution, and spatiotemporal features are output.

[0023] Among them, the multi-view image patch is obtained by splitting the image after spherical projection of each frame in the panoramic RGB video stream according to the preset polar angle interval and azimuth angle interval; the weighted processing is to construct a spherical distortion weight matrix based on the polar angle and azimuth angle corresponding to the multi-view image patch, and weight the spatial features; the motion segment includes fast motion segment and slow motion segment, and feature extraction is performed on the fast motion segment by dilated temporal convolution, and feature extraction is performed on the slow motion segment by ordinary temporal convolution.

[0024] The inherent regional distortion characteristics of panoramic images after spherical projection result in significant differences in the degree of distortion of multi-view tiles corresponding to different polar angles and azimuth angles. The degree of distortion in the polar region is much higher than that in the equatorial region. Directly using the original spherical convolution features will lead to spatial information distortion and will not provide stable feature support for subsequent joint detection and 3D localization of the root node. The multi-view tiles are obtained by splitting them according to preset polar angle intervals and azimuth angle intervals, and their positions are determined by the polar angle and azimuth angle.

[0025] For each multi-view image patch after splitting, a spherical convolution kernel that adapts to the characteristics of the spherical projection surface and is different from the planar convolution kernel is used for feature extraction. The resulting spatial features for each image patch include spatial information such as human body contours and joint candidate regions within the image patch. A spherical distortion weight matrix is ​​constructed based on the polar angle and azimuth angle corresponding to the multi-view image patch. The spatial features are weighted by this matrix to achieve distortion correction.

[0026] Specifically, the image after spherical projection of each frame in the panoramic RGB video stream is set with a projection radius of 100 pixels. It is then divided into 6 intervals with a polar angle range of 0°-180° and an azimuth angle range of 0°-360°, resulting in 48 multi-view tiles. For each multi-view tile, a 3×3 spherical convolution kernel is used. The kernel parameters are adjusted according to the curvature of the sphere. Those skilled in the art can adjust them accordingly in the actual process. The kernel stride is 1 in the equatorial region and 0.5 in the polar region. Two convolution operations are performed on the tile to extract 256-dimensional spatial features for each tile.

[0027] Based on the polar angle θ and azimuth angle φ of each map patch, the distortion coefficient k = cosθ is calculated, where θ is the polar angle of the patch center. The smaller the polar angle, the closer to the equator, the closer the distortion coefficient is to 1, the smaller the distortion degree, and the greater the weight. Conversely, the larger the polar angle, the closer to the pole, the smaller the distortion coefficient, the greater the distortion degree, and the smaller the weight. For example, the distortion coefficient of a map patch with a polar angle of 0° (equatorial region) is 1, and the weight is set to 1. The distortion coefficient of a map patch with a polar angle of 90° (polar region) is 0, and the weight is set to 0.6 to avoid feature loss due to a weight of 0. For a map patch with a polar angle of 30°, the distortion coefficient... The weight is set to 0.866; when the polar angle is greater than 90°, the distortion coefficient is negative and has no physical weighting meaning, so the weight of this type of tile is uniformly set to 0.6; based on the weight of all tiles, a spherical distortion weight matrix is ​​constructed, with each row of the matrix corresponding to the weight of the tile and each column corresponding to the dimension of the spatial feature.

[0028] The spatial features of each patch are multiplied dimension-by-dimensionally with the weights of the corresponding rows in the spherical distortion weight matrix to obtain the weighted spatial features of each patch. For example, the spatial feature vector of a patch in an equatorial region is [0.1, 0.2, ..., 0.3], with a weight of 1. After weighting, the spatial feature vector is still [0.1, 0.2, ..., 0.3]. For a patch with a polar angle of 60°, the weight is 0.5, and the spatial feature vector is [0.1, 0.2, ..., 0.3]. After weighting, the spatial feature vector is [0.05, 0.1, ..., 0.15], thus achieving distortion correction.

[0029] This corrects the distortion caused by spherical projection, ensuring that the spatial feature weights of multi-view blocks at different locations are matched with the degree of distortion, and that the weighted spatial features accurately reflect the spatial position information of the human body.

[0030] Human movement speed varies, and a uniform form of temporal convolution cannot meet the feature extraction needs of different motion states. Fast motion segments are prone to temporal feature loss, while slow motion segments are prone to feature redundancy. For the acquired panoramic RGB video stream, sparse sampling is performed at a rate of 1 frame every 3 frames to obtain a continuous sampled frame sequence, such as frames 1, 4, 7, 10... The human motion amplitude between two adjacent sampled frames is calculated, and fast motion segments and slow motion segments are divided according to the motion amplitude threshold. The motion segments include fast motion segments and slow motion segments. Dilated temporal convolution is used for fast motion segments, while ordinary temporal convolution is used for slow motion segments to extract the temporal features of each motion segment, including the inter-frame variation information of human motion.

[0031] For a panoramic RGB video stream with a frame rate of 30fps, the sampling interval is set to 3 frames, that is, 1 frame is extracted as a sampling frame every 3 frames to obtain a continuous sampling frame sequence, such as frame 1, frame 4, frame 7, frame 10, frame 13, frame 16... After sampling, the frame rate is 10fps, which reduces the amount of computation and retains key timing information. The optical flow method is used to calculate the human motion amplitude between two adjacent sampling frames, that is, the center of gravity offset of the human contour, in pixels. For example, the center of gravity offset between frame 1 and frame 4 is 5 pixels, the center of gravity offset between frame 4 and frame 7 is 20 pixels, and the center of gravity offset between frame 7 and frame 10 is 3 pixels.

[0032] Based on a sampled frame rate of 10fps and a projection radius of 100 pixels, and through repeated experimental calibration combined with the offset range of normal human movement, a motion amplitude threshold of 10 pixels was set. Those skilled in the art can adjust the motion amplitude threshold accordingly in practice. When the center-of-gravity offset between two adjacent frames is greater than 10 pixels, it is determined to be a fast motion segment; when the center-of-gravity offset between two adjacent frames is less than or equal to 10 pixels, it is determined to be a slow motion segment. Frames 4-7 are fast motion segments, and frames 1-4 and 7-10 are slow motion segments. For slow motion segments, ordinary temporal convolution is used for feature extraction, with a kernel size of 3×3, a stride of 1, an expansion rate of 1, and 2 convolutions to capture subtle inter-frame changes during slow motion. For fast motion segments, dilated temporal convolution is used for feature extraction, with a kernel size of 3×3, a stride of 1, an expansion rate of 2 to expand the receptive field and capture more inter-frame motion information, and 2 convolutions to capture drastic inter-frame changes during fast motion. The features are then integrated according to the temporal order of the sampled frames to obtain segmented temporal features.

[0033] This allows it to adapt to the temporal requirements of different human movement speeds, ensuring the capture of complete inter-frame motion information; accurately reflecting the temporal changes in human movement, and making the spatiotemporal features both spatial accuracy and temporal continuity.

[0034] Weighted spatial features focus on reflecting the spatial location information of the human body, while segmented temporal features focus on reflecting the temporal changes in human movement. The dimensions and benchmarks of the two are inconsistent. Direct fusion will lead to feature disorder and failure to form effective spatiotemporal features. At the same time, the fused features may still have topological distortion caused by the curvature of the spherical projection, affecting the accuracy of subsequent feature parsing. The timestamp of each sampled frame is matched with the sparsely sampled frame sequence, and the spherical projection coordinate system is consistent with the polar angle and azimuth angle of the multi-view tiles, so that the weighted spatial features and segmented temporal features correspond in time and space.

[0035] Through a learnable gated fusion unit, the fusion weights of weighted spatial features and segmented temporal features are dynamically allocated based on their importance. Fusion is achieved through feature concatenation and element-wise addition. Those skilled in the art can adjust the gated fusion unit in practice. Based on the curvature distribution of spherical projection, the fused features are topologically corrected to eliminate feature topological distortion caused by spherical curvature. The curvature varies at different polar angles, with smaller curvature at the equator and larger curvature at the poles. Using the timestamp of each sampling frame as the time reference and the spherical projection coordinate system as the spatial reference, the timestamp corresponding to each sampling frame is obtained for time reference alignment. For example, the timestamp of the first frame is 0.1s, the fourth frame is 0.4s, the seventh frame is 0.7s, and so on. The weighted spatial features and segmented temporal features are matched according to their timestamps to ensure that the spatial features and temporal features at the same timestamp correspond to the human body information of the same sampling frame.

[0036] In terms of spatial reference alignment, the spherical projection coordinate system is used as the reference. The weighted spatial features and segmented temporal features of each sampling frame are spatially correlated based on polar angle and azimuth angle to ensure that the position information of the spatial features corresponds to the motion information of the temporal features. For example, the weighted spatial features of the equatorial region patch of a sampling frame are aligned with the information on human motion in the equatorial region in the segmented temporal features of the sampling frame.

[0037] The gated fusion unit includes two learnable gating parameters ω1 and ω2, corresponding to the fusion weights of the weighted spatial feature F_s and the piecewise temporal feature F_t, respectively, satisfying ω1 + ω2 = 1. Through model training, the number of training iterations is 10,000, and the learning rate is 0.001. The gating parameters are dynamically adjusted; for example, when human movement is more intense (fast movement segment), temporal features are more important, so ω1 = 0.4 and ω2 = 0.6 are set; when human movement is more gentle (slow movement segment), spatial features are more important, so ω1 = 0.6 and ω2 = 0.4 are set. The weighted spatial features and segmented temporal features are weighted and summed according to the gating parameters to obtain the fused feature F_fuse=ω1×F_s+ω2×F_t. For example, for a certain sampling frame in a fast motion segment, F_s=[0.1,0.2,...,0.3], F_t=[0.5,0.6,...,0.7], ω1=0.4, ω2=0.6, the fused feature F_fuse=[0.4×0.1+0.6×0.5,0.4×0.2+0.6×0.6,...,0.4×0.3+0.6×0.7].

[0038] The curvature distribution of the spherical projection is obtained. The curvature value R = 1 / cos²θ is calculated based on the polar angle. When the polar angle is 0°, the curvature R = 1; when the polar angle is 30°, the curvature R ≈ 1.333; and when the polar angle is 90°, the curvature R approaches infinity. In practice, it is set to 10. Based on the curvature distribution, topological normalization correction is performed on the fused features. The corrected fused features... This weakens the high curvature in extreme regions, which leads to feature topological distortion. For example, the fused feature F_fuse=[0.333,0.666,...,1] for a 30° polar region has a curvature R≈1.333. After correction, F_norm = [0.333 / 1.1547, 0.666 / 1.1547, ..., 1 / 1.1547] ≈ [0.288, 0.577, ..., 0.866]. The fused features after correction of all sampled frames are integrated in the order of timestamps to obtain continuous spatiotemporal features, with each sampled frame corresponding to a 256-dimensional spatiotemporal feature.

[0039] This ensures the consistency of weighted spatial features and segmented temporal features in both time and space. The fusion ratio of spatial features and temporal features is dynamically adjusted according to the human motion state, making the fused features more in line with actual motion requirements. It also eliminates feature topology distortion caused by spherical curvature, ensuring that the topological structure of spatiotemporal features is consistent with the actual human posture, thereby improving the reliability of overall human motion posture detection.

[0040] S2. Input the spatiotemporal features into the first detection network, output the spherical coordinates and confidence of the joint in the projected coordinate system, and select the spherical coordinates with the highest confidence in the multi-view tiles at the same time as the spherical coordinates of the corresponding joint.

[0041] Specifically, the output joint's spherical coordinates in the projected coordinate system and its confidence level include: The spatiotemporal features are decomposed into tile spatiotemporal features, which are input into the first detection network and the network is parsed and output as the initial spherical coordinates, initial confidence and feature response values ​​of all candidate joints in the corresponding tile. Candidate joints with feature response values ​​less than the response threshold are removed, and the initial spherical coordinates of the remaining candidate joints are calibrated for distortion compensation. The calibrated spherical coordinates and their corresponding initial confidence scores are summarized in all multi-view tiles of the same joint at the same time. The initial confidence scores are corrected based on the spatial overlap of the calibrated spherical coordinates to obtain the confidence score of the joint. The calibrated spherical coordinates with the highest confidence scores are selected as the spherical coordinates of the joint.

[0042] The first detection network includes branch detection units corresponding to multi-view tiles, and the branch detection units are adapted to the projection distortion degree of the corresponding tiles; the initial spherical coordinates are the calibration coordinates in the projection coordinate system.

[0043] The spatiotemporal features output above are an overall feature sequence, covering the spatial and temporal information of all multi-view tiles. Joint detection, however, needs to be performed separately for each tile, and different tiles have different projection distortion. The spatiotemporal features are split according to the division rules of multi-view tiles, and the spatiotemporal features corresponding to each multi-view tile are split. Each tile corresponds to a set of tile spatiotemporal features, which correspond to the number and position of the aforementioned multi-view tiles.

[0044] The first detection network refers to a detection network that includes branch detection units corresponding to multi-view tiles. Each branch detection unit is adapted to the projection distortion degree of the corresponding tile to offset the distortion effects of different tiles. Each branch detection unit adopts a cascaded structure of two convolutional layers, one pooling layer, and one fully connected layer. The activation function is the ReLU activation function. The training loss function integrates joint coordinate smoothing loss and confidence cross-entropy loss. The training dataset uses a self-built panoramic human joint annotation dataset. Those skilled in the art can adjust the unit parameters accordingly in the actual process. The initial spherical coordinates are the calibration coordinates in the projection coordinate system, that is, the initial position coordinates of the joint in the projection coordinate system determined by the preset calibration rules. The initial confidence is the output of the first detection network, which represents the preliminary confidence that the candidate joint is a real joint. The feature response value is the degree of matching between the candidate joint features and the real joint features, which is used to judge the effectiveness of the candidate joint.

[0045] The aforementioned single-frame spherical projection image has been divided into 48 multi-view tiles, corresponding to the division of the spatiotemporal features of each frame into 48 sets of tile spatiotemporal features. Each set of tile spatiotemporal features corresponds to one multi-view tile, with the dimension remaining 256. The first detection network is set with 48 branch detection units, corresponding to the 48 multi-view tiles. The network parameters of each branch detection unit are adjusted according to the polar angle and azimuth angle of the corresponding tile to adapt to the projection distortion degree of the tile. For example, the branch detection unit in the equatorial region uses standard parameters, while the branch detection unit in the polar region adjusts the convolution kernel weights to compensate for distortion.

[0046] The spatiotemporal features of each group of map tiles are input into the corresponding branch detection unit. Each branch detection unit parses and outputs the relevant parameters of all candidate joints within the map tile. The initial spherical coordinates are calibrated with the pole of the projected coordinate system as the origin, with the polar angle ranging from 0° to 180° and the azimuth angle ranging from 0° to 360°. For example, the initial spherical coordinates of a candidate hip joint within a certain map tile are calibrated as (polar angle 45°, azimuth angle 90°). The initial confidence is represented by a range of 0-1, with higher confidence being closer to 1. For example, the initial confidence of this candidate hip joint is 0.7. The feature response value is also represented by a range of 0-1, used to measure the clarity and matching degree of the candidate joint features. For example, the feature response value of this candidate hip joint is 0.65.

[0047] Within each image patch, candidate joint parameters of the corresponding type are output according to the type of human joint, such as hip joint, knee joint and shoulder joint. For example, eight types of candidate joints can be detected within each image patch, and each candidate joint corresponds to a set of initial spherical coordinates, initial confidence and feature response values.

[0048] This allows for targeted handling of distortion effects on different map blocks, ensuring the accuracy of candidate joint parameter extraction within each map block and providing a reliable basis for subsequent candidate joint selection and coordinate calibration.

[0049] Among the candidate joints output by the first detection network, some candidate joints have low feature response values, indicating that their matching degree with the real joint features is low. They are invalid or interfering candidates. If they are retained, it will affect the accuracy of subsequent joint coordinates. The initial spherical coordinates are the calibration coordinates under the projected coordinate system, which do not consider the projection distortion of the blocks and have positional deviations. A preset response threshold is set, and candidate joints with feature response values ​​less than the response threshold are judged as invalid candidates and are removed. Only candidate joints with feature response values ​​not less than the response threshold are retained.

[0050] Based on the distortion coefficient of the map patch, the initial spherical coordinates of the remaining candidate joints are corrected to offset the positional deviation of the projection distortion, which is consistent with the distortion processing logic of the aforementioned weighted spatial features and root node coordinate calibration. Based on the distortion distribution properties of 48 sets of multi-view map patches, and combined with repeated calibration of massive panoramic joint samples, a preset response threshold of 0.3 is obtained to distinguish between effective joint features and invalid noise features. Those skilled in the art can adjust the threshold accordingly in the actual process. For the candidate joints output by each map patch, candidate joints with feature response values ​​less than 0.3 are removed. For example, if the feature response value of a candidate knee joint in a certain map patch is 0.25, which is lower than the response threshold, it is removed, while the feature response value of another candidate knee joint is 0.5, which is retained.

[0051] The distortion compensation is calculated based on the polar angle and azimuth angle of the corresponding patch. The distortion coefficient is consistent with the spherical distortion weight matrix. After calibration, the polar angle = initial polar angle × k, and the azimuth angle after calibration = initial azimuth angle × k. For example, if a candidate hip joint is located in a patch with a polar angle of 60°, its initial spherical coordinates are (polar angle 45°, azimuth angle 90°). The distortion coefficient k of this patch is 0.5. After calibration, the polar angle is 45° × 0.5 = 22.5°, and the azimuth angle is 90° × 0.5 = 45°. The calibrated spherical coordinates are (22.5°, 45°). The distortion coefficient of the patch in the equatorial region is 1, and the calibrated coordinates are consistent with the initial coordinates. Similarly, the distortion compensation calibration is completed for all remaining candidate joints to accurately reflect the true position of the joint in the projected coordinate system.

[0052] This improves the efficiency and accuracy of subsequent joint coordinate selection; eliminates the influence of projection distortion on the initial spherical coordinates, and makes the calibrated joint coordinates more closely match the actual spatial position.

[0053] The same joint can be captured by multiple multi-view tiles within the same time frame. Each tile outputs candidate coordinates and an initial confidence score for that joint. These coordinates may have slight differences. Within the same sampling frame, all multi-view tiles capturing the joint may output slightly different joint coordinates due to viewpoint differences. The positional similarity between the calibrated spherical coordinates of the same joint output by multiple tiles is called spatial overlap, i.e., spatial overlap = 1 - (current group comprehensive deviation - minimum comprehensive deviation) / (maximum comprehensive deviation - minimum comprehensive deviation), where comprehensive deviation is the most significant factor. Deviation = Absolute polar deviation + Absolute azimuth deviation. Absolute deviation refers to the absolute difference between the polar angle and azimuth angle of each set of spherical coordinates and their average values. The smaller the overall deviation, the higher the spatial overlap, indicating that the coordinates are closer to the true position and the higher the reliability. The spatial overlap is used as the basis for correction to adjust the initial confidence level, so that the corrected confidence level more accurately represents the reliability of the joint coordinates. After correcting the confidence level, the calibrated spherical coordinates with the highest confidence level are selected as the final spherical coordinates of the joint in the current time frame to ensure the uniqueness and accuracy of the coordinates.

[0054] Taking the hip joint as an example, within the same time frame (timestamp 0.4s), six multi-view tiles can capture the hip joint. Six sets of calibrated spherical coordinates and initial confidence scores are output, such as (22.5°, 45°), (23°, 44.5°), (22°, 45.5°), (30°, 50°), (22.3°, 44.8°), and (22.7°, 45.2°), corresponding to the initial confidence scores. The reliability scores were 0.7, 0.68, 0.72, 0.4, 0.69, and 0.71, respectively. Based on the above calculation logic, the mean polar angle was 23.75° and the mean azimuth angle was 45.83°. The overall deviations were 2.08°, 2.08°, 2.08°, 10.42°, 2.48°, and 1.68°, respectively. The final spatial overlap rates were 0.954, 0.954, 0.954, 0, 0.908, and 1, respectively.

[0055] The spatial overlap and the initial confidence are weighted and fused with weights of 0.4 and 0.6 respectively. That is, the corrected confidence = initial confidence × 0.6 + spatial overlap × 0.4. Then the corrected confidences are 0.7×0.6+0.954×0.4≈0.802, 0.68×0.6+0.954×0.4≈0.790, 0.72×0.6+0.954×0.4≈0.814, 0.4×0.6+0×0.4=0.24, 0.69×0.6+0.908×0.4≈0.777, and 0.71×0.6+1×0.4=0.826.

[0056] After correction, the coordinate with the highest confidence after correction is selected as the spherical coordinate of the hip joint. For example, among the above 6 sets of coordinates, the highest confidence after correction is 0.826, corresponding to coordinates (22.7°, 45.2°). This coordinate is then selected as the spherical coordinate of the hip joint in the projected coordinate system in the current frame. Similarly, the summary, confidence correction and coordinate filtering of other joints are completed to obtain the spherical coordinates and confidence of all joints in the current frame.

[0057] This effectively filters out the joint coordinates that are closest to the actual position. The corrected confidence level more accurately represents the reliability of the joint coordinates, providing accurate coordinates and confidence level for subsequent joint 3D position solving and posture correction. This determines the accuracy of human motion posture detection and ensures that subsequent inverse kinematics solving and temporal prediction correction can be carried out based on reliable basic parameters.

[0058] S3. Input the spatiotemporal features into the second detection network, output the spherical coordinates of the root node and the corresponding feature vector, estimate the depth of the root node through the depth regression network, and calculate the three-dimensional position of the root node by combining the corresponding spherical coordinates.

[0059] Specifically, calculating the three-dimensional position of the root node includes: The spatiotemporal features of the map tiles are input into the second detection network, which outputs the spherical coordinates of the root node and the corresponding feature vector, and filters out the spatiotemporal features of the target map tiles that match the feature vector. The spatiotemporal features of the target map are input into a depth regression network for depth estimation to obtain the root node depth and depth confidence. The root node depth is then weighted and calibrated, and the spherical coordinates of the root node are calibrated for distortion compensation. Based on the mapping relationship between the projected coordinate system and the Cartesian coordinate system, and combined with the calibrated spherical coordinates and the calibrated root node depth, the three-dimensional position of the root node is calculated.

[0060] The deep regression network includes regression branches corresponding to multi-view tiles, and the regression branches are adapted to the projection distortion degree of the corresponding target tiles; the spatiotemporal features of the tiles are selected as the spatiotemporal features of the target tiles if the similarity between the spatiotemporal features and the feature vectors is greater than the similarity threshold. Root node depths with depth confidence less than a preset threshold are removed. The spatial fit between the remaining root node depths and the spherical coordinates of the root nodes is calculated. The spatial fit and depth confidence are weighted and fused together to obtain the weight, and then the weighted sum is obtained to obtain the calibrated root node depth.

[0061] The root node serves as the baseline node for human motion posture detection, and its position detection accuracy determines the accuracy of subsequent joint 3D position solving. The spatiotemporal features of the tiles obtained from the aforementioned splitting cover information from all multi-view tiles, some of which did not capture the root node or whose captured root node features are blurred. The spatiotemporal features of the tiles here are consistent with the spatiotemporal features of the tiles obtained from the splitting in the previous steps. The second detection network is used to extract the spherical coordinates and feature vectors of the root node. Unlike the first detection network, its network structure is adapted to the feature extraction requirements of the root node. The second detection network adopts a stacked structure of three convolutional layers and two fully connected layers. The activation function is the ReLU activation function, and the training loss function is a fusion of feature vector matching loss and spherical coordinate regression loss. The training dataset uses a self-built panoramic human root node annotation dataset. Those skilled in the art can adjust the network parameters accordingly in the actual process.

[0062] The spherical coordinates of the root node are the initial coordinates in the projected coordinate system, and the feature vector is the feature representation of the root node, which is used to measure the matching degree between the spatiotemporal features of the tile and the features of the root node. The similarity between the spatiotemporal features of the tile and the feature vector of the root node is calculated, and the spatiotemporal features of the tile with a similarity greater than a preset similarity threshold are determined as the target spatiotemporal features of the tile, effectively representing the spatiotemporal features of the root node features.

[0063] The spatiotemporal features of the aforementioned 48 sets of map tiles are input into the second detection network. The second detection network parses and outputs the spherical coordinates and feature vector of the root node corresponding to each map tile. The feature vector of the root node (selecting the human hip joint as the root node) is 256-dimensional. The initial spherical coordinates are based on the pole of the projected coordinate system as the origin, and are calibrated according to the polar angle 0°-180° and the azimuth angle 0°-360°. For example, the initial spherical coordinates of the root node output by a certain map tile are (polar angle 30°, azimuth angle 180°), and the corresponding feature vector is [0.2,0.3,...,0.4].

[0064] Based on the distribution properties of the 48 sets of spatiotemporal features, and through repeated calibration using panoramic root node feature matching experiments, a preset similarity threshold of 0.6 was obtained to distinguish between effective matching features of the root node and irrelevant interference features. Those skilled in the art can adjust the threshold accordingly in the actual process. The cosine similarity method is used to calculate the similarity between the spatiotemporal features of each set of map tiles and the feature vector of the root node. For example, if the similarity between a certain spatiotemporal feature of a map tile and the feature vector of the root node is 0.75, which is greater than the similarity threshold, it is determined to be the spatiotemporal feature of the target map tile. If the similarity between a certain spatiotemporal feature of a map tile and the feature vector of the root node is 0.5, which is less than the similarity threshold, it is discarded. Eight sets of target spatiotemporal features are selected from the 48 sets of spatiotemporal features of the map tiles that effectively capture the spatiotemporal features of the root node features.

[0065] This ensures the targeted extraction of root node spherical coordinates and feature vectors, eliminates interference from invalid patches, and ensures the input quality for subsequent depth estimation.

[0066] Although the spatiotemporal features of the target tiles have been effectively filtered, different target tiles exhibit varying projection distortions, leading to errors in depth estimation. Furthermore, the initial spherical coordinates of the root node do not account for projection distortion, resulting in positional bias. The depth regression network comprises regression branches corresponding to tiles from multiple views. Each regression branch is adapted to the projection distortion degree of its corresponding target tile, offsetting the distortion effects of different tiles and improving depth estimation accuracy. Each regression branch of the depth regression network employs a simplified structure of two convolutional layers and one fully connected layer, using the LeakyReLU activation function and the depth value regression mean squared error loss as the training loss function. Model training is completed using multi-view depth-annotated samples in a panoramic scene, allowing those skilled in the art to adjust network parameters during actual operation. The root node depth is the vertical distance from the root node to the projection plane, and the depth confidence score, ranging from 0 to 1, is used to determine the validity of the depth estimation result.

[0067] Based on the distribution law of panoramic depth estimation error, a preset threshold is obtained by combining multi-scene test calibration. This threshold is used to filter out abnormal depth values ​​with low confidence. Those skilled in the art can adjust the preset threshold accordingly in actual process. First, depth values ​​with depth confidence less than the preset threshold are eliminated. Then, the spatial fit between the remaining root node depth and the root node spherical coordinates is calculated, i.e., spatial fit = 1 - (absolute depth deviation of this group - minimum depth deviation) / (maximum depth deviation - minimum depth deviation), with a value of 0-1. The standard depth for matching the spherical coordinates is set to 1.2m. The spatial fit and depth confidence are weighted and fused to obtain the weight. The remaining root node depths are weighted and summed to obtain the calibrated root node depth. The distortion compensation calibration of the root node's spherical coordinates is consistent with the distortion compensation calibration logic of the joint spherical coordinates mentioned above. Based on the distortion coefficient of the target block, the initial spherical coordinates of the root node are corrected to offset the position deviation caused by projection distortion.

[0068] The specific deep regression network has 48 regression branches. The network parameters of each regression branch are adjusted according to the polar angle and azimuth angle of the corresponding map patch to adapt to the projection distortion degree of the map patch. For example, the regression branch corresponding to the target map patch with a polar angle of 60° adjusts the convolution kernel weights to offset the distortion effect. The spatiotemporal features of the 8 selected target map patches are input into the regression branches of the corresponding map patches. Each regression branch outputs the root node depth and depth confidence. For example, the root node depths of the 8 target map patches are 1.2m, 1.18m, 1.22m, 1.5m, 1.19m, 1.21m, 0.9m, and 1.2m, respectively, and the corresponding depth confidences are 0.8, 0.78, 0.82, 0.4, 0.79, 0.81, 0.25, and 0.77, respectively. The preset threshold is 0.3. Depth values ​​with a depth confidence of less than 0.3 are removed, that is, the confidence of 0.9m is 0.25, leaving 7 valid depth values.

[0069] Using the initial spherical coordinates of the root node as a reference, the spatial fit degree between each effective depth and its coordinates is calculated, with a value ranging from 0 to 1. The higher the matching degree between the depth and the spherical coordinates and the smaller the deviation, the closer the spatial fit degree is to 1. The absolute deviations corresponding to the aforementioned 7 sets of effective depth values ​​are 0, 0.02, 0.02, 0.3, 0.01, 0.01, and 0, respectively, and the corresponding spatial fit degrees are 1, 0.933, 0.933, 0, 0.967, 0.967, and 1. The spatial fit degree and depth confidence are weighted and fused to obtain the weight, with a weight ratio of 0.4 and 0.6, respectively. Weight = Depth Confidence × 0.6 + Spatial Fit × 0.4, resulting in weights of 0.8 × 0.6 + 1 × 0.4 = 0.88, 0.78 × 0.6 + 0.933 × 0.4 = 0.8412, 0.82 × 0.6 + 0.933 × 0.4 = 0.8652, 0.4 × 0.6 + 0 × 0.4 = 0.24, 0.79 × 0.6 + 0.967 × 0.4 = 0.8608, 0.81 × 0.6 + 0.967 × 0.4 = 0.8728, and 0.77 × 0.6 + 1 × 0.4 = 0.862.

[0070] The remaining 7 groups of effective depths are weighted and summed according to their corresponding weights to obtain the calibrated depth, i.e., calibrated depth = ∑(effective depth of each group × corresponding weight) ÷ ∑(weights of all groups). Substituting the above 7 groups of effective depths and their corresponding weights into the calculation, the sum of the effective depths and corresponding weights of each group is approximately 6.579, and the sum of the weights of all groups is 5.422, resulting in a calibrated depth of approximately 1.21m. The same distortion coefficients as before are used, i.e., calibrated polar angle = initial polar angle × k, calibrated azimuth angle = initial azimuth angle × k. For example, if the initial spherical coordinates of the root node are (polar angle 30°, azimuth angle 180°), the polar angle of the corresponding target patch is 30°, and the distortion coefficients are... After calibration, the polar angle is 30°×0.866≈25.98°, and the azimuth angle is 180°×0.866≈155.88°. The spherical coordinates of the calibrated root node are (25.98°, 155.88°). Similarly, the distortion compensation calibration of the spherical coordinates of the root nodes corresponding to all target blocks is completed, and the average value of all calibrated coordinates is taken as the spherical coordinates of the calibrated root node.

[0071] This improves the accuracy of depth estimation, eliminates depth values ​​with large errors, and eliminates positional deviations caused by projection distortion, ensuring that the coordinates are consistent with the calibrated depth, thus providing accurate parameter support for subsequent operations.

[0072] The calibrated root node spherical coordinates are two-dimensional angular coordinates in the projected coordinate system and cannot be directly used as the reference for solving the three-dimensional position of the joints. The mapping relationship between the projected coordinate system and the Cartesian coordinate system refers to the transformation logic of converting the polar angle, azimuth angle (spherical coordinates) and depth in the projected coordinate system into three-dimensional coordinates (X, Y, Z) in the Cartesian coordinate system. The Cartesian coordinate system takes the center of the projected sphere as the origin, the Z-axis as the polar axis of the projected sphere, which corresponds to the polar direction of the projected coordinate system, and the X-axis and Y-axis lie in the projected equatorial plane, corresponding to the azimuth angle and polar angle.

[0073] By combining the calibrated spherical coordinates and the calibrated root node depth, the calibrated polar angle and azimuth angle are substituted into the mapping formula. Combined with the calibrated root node depth, the X, Y, and Z coordinates in the Cartesian coordinate system are calculated, which are the three-dimensional positions of the root node. The mapping relationship between the projected coordinate system and the Cartesian coordinate system is established. Based on the conversion logic between spherical coordinates and Cartesian coordinates, the mapping formula is set as follows: X = depth × sin(polar angle) × cos(azimuth angle), Y = depth × sin(polar angle) × sin(azimuth angle), Z = depth × cos(polar angle).

[0074] The polar angle and azimuth angle are converted to radians for calculation. After calibration, the spherical coordinates of the root node are (25.98°, 155.88°), which are 0.453 radians and 2.721 radians respectively. The calibrated depth is 1.21m. Substituting the above parameters, we get X = 1.21 × sin(0.453) × cos(2.721), sin(0.453) ≈ 0.438, cos(2.721) ≈ -0.92, therefore X ≈ 1.21 × 0.438 × (-0.92) ≈ -0.487m; Y = 1.21 × sin(0.453) × sin(2.721), sin(2.721) ≈ 0.438, cos(2.721) ≈ -0.92, so X ≈ 1.21 × 0.438 × (-0.92) ≈ -0.487m; 721)≈0.404, therefore Y≈1.21×0.438×0.404≈0.214m; Z=1.21×cos(0.453), cos(0.453)≈0.899, therefore Z≈1.21×0.899≈1.088m; the three-dimensional position of the root node in the Cartesian coordinate system is obtained as (-0.487m, 0.214m, 1.088m); the parameter settings of this mapping relationship are consistent with the parameters of the aforementioned spherical projection, ensuring that the transformed three-dimensional coordinates accurately reflect the actual spatial position of the root node, and at the same time maintain consistency with the coordinate system of the subsequent joint three-dimensional position solution, avoiding position deviation caused by coordinate system inconsistency.

[0075] This enables the transformation of the root node from projected coordinates to three-dimensional spatial coordinates, giving the root node position a clear spatial reference meaning. As the core reference for subsequent initial three-dimensional position solving of joints, it provides a reliable spatial reference for motion posture output and ensures the spatial consistency of overall human motion posture detection.

[0076] S4. Based on the three-dimensional position of the root node, combined with the spherical coordinates of the joint and the topological constraints of the bone, as well as the dynamic bone length obtained by distortion adaptation correction of the preset bone length, the initial three-dimensional position of the joint is solved by inverse kinematics.

[0077] Specifically, solving for the initial three-dimensional position of the joint includes: Based on the three-dimensional position of the root node, the dynamic bone length is obtained by performing distortion adaptation correction on the preset bone length according to the relative polar angle and azimuth angle between the joint and the root node in the projection coordinate system. A hierarchical topology is constructed according to the human skeleton hierarchy. Hierarchical weights are assigned based on the confidence of joints. Inverse kinematics calculations are performed hierarchically based on dynamic bone length and spherical coordinates of joints to obtain joint positions. The initial three-dimensional position of the joint is obtained by performing skeletal topology constraint verification and abnormal position correction on the joint positions at different levels.

[0078] The preset bone length is a fixed value, without considering the distortion caused by spherical projection. Furthermore, the relative positions of different joints and the root node differ, resulting in varying degrees of projection distortion. Directly using the preset bone length for joint position calculations would lead to deviations in the joint's 3D position. The relative polar angle and azimuth angle between the joint and the root node in the projection coordinate system refer to the difference in polar angle and azimuth angle between the spherical coordinates of each joint and the root node in the projection coordinate system within the same time frame. These values ​​characterize the relative spatial relationship between the joint and the root node. The preset bone lengths for each joint are based on standard human bone dimensions, such as the preset femoral length from the hip to the knee joint and the preset tibia length from the knee to the ankle joint. The preset bone lengths are corrected using the projection distortion coefficients corresponding to the relative polar angle and azimuth angle to eliminate the influence of projection distortion on bone length characterization. Dynamic bone length refers to the corrected bone length adapted to the current projection distortion and the relative position of the joint and the root node, dynamically adjusted as the joint posture changes.

[0079] Specifically, the root node is selected as the human hip joint, whose three-dimensional position, calculated as described above, is (-0.487m, 0.214m, 1.088m), corresponding to calibrated spherical coordinates of (25.98°, 155.88°). Taking the knee joint as an example, its calibrated spherical coordinates are (45°, 160°). The relative polar angle between the joint and the root node is calculated as 45° - 25.98° = 19.02°, and the relative azimuth angle is 160° - 155.88° = 4.12°. Here, the bone length... The distortion correction uses the relative polar angle of the joint to adapt to the local skeletal projection distortion of the human body. The calculation formula for the distortion coefficient is the same as that mentioned above. The preset length of the femur (from the hip joint to the knee joint) is 0.4m. The distortion coefficient is calculated in combination with the relative polar angle. Here, the relative polar angle is 19.02°, and cos19.02°≈0.945. Therefore, the distortion adaptation correction formula is: dynamic bone length = preset bone length × k. Substituting the values, the dynamic bone length corresponding to the knee joint is calculated to be 0.4m × 0.945 = 0.378m.

[0080] Similarly, taking the ankle joint as an example, its relative polar angle with the root node (hip joint) is 40°, the distortion coefficient k=cos40°≈0.766, the preset length of the tibia (from the knee joint to the ankle joint) is 0.38m, and the corrected dynamic bone length is 0.38m×0.766=0.291m; all joints are corrected according to the above logic, combined with their relative polar angle with the root node, to complete the distortion adaptation correction of the preset bone length, and obtain the dynamic bone length corresponding to each joint, ensuring that the bone length is adapted to the current projection distortion and relative position.

[0081] This effectively eliminates the influence of projection distortion on bone length, enabling dynamic bone length to accurately represent the actual bone length between joints in the current posture, and ensuring the spatial reference consistency of bone length for all joints.

[0082] The human skeleton has a clear hierarchical structure. If the overall inverse kinematics calculation is used, problems such as joint position conflicts and low calculation accuracy are likely to occur. The hierarchy is divided according to the physiological connection relationship of the human skeleton. The root node (hip joint) is the first level, the trunk joints and knee joints connected to the hip joint are the second level, the ankle joint connected to the knee joint and the shoulder joint connected to the trunk joint are the third level, and so on downwards. There are clear parent-child joint relationships between the levels. The hierarchical topology relationship between joints is constructed based on the skeleton hierarchy, and the parent joint (upper-level joint) and child joint (lower-level joint) of each joint are clearly defined.

[0083] Based on the output joint confidence score, the higher the confidence score, the greater the weight of the joint at the corresponding level, and the more significant its impact on the joint position calculation results. This ensures the dominant role of highly reliable joint coordinates. Inverse kinematics calculations are performed sequentially from the top to the bottom of the skeletal hierarchy. The joint position calculation at each level is based on the position of its parent joint, the dynamic bone length, and its own spherical coordinates. The three-dimensional position of the joint is solved by the inverse kinematics algorithm. Specifically, the topological relationship of the human skeleton is constructed. The first level is the root node (hip joint), whose three-dimensional position is known (-0.487m, 0.214m, 1.088m). The second level includes the knee joint and the lumbar spine joint, both of which are parent joints of the hip joint. The third level includes the ankle joint and the shoulder joint, with the knee joint and the lumbar spine joint as parent joints, respectively. The fourth level includes the toe joint and the elbow joint, with the ankle joint and the shoulder joint as parent joints, respectively. This completes the topological construction of the complete skeletal hierarchy.

[0084] The weights for each level are assigned based on the confidence level of the joint. The confidence level of the joint is a modified confidence level, and the weights for each level are consistent with the confidence level. For example, the confidence level of the knee joint is 0.71, and its weight in the second level calculation is 0.71; the confidence level of the ankle joint is 0.69, and its weight in the third level calculation is 0.69; the confidence level of the lumbar spine joint is 0.72, and its weight in the second level calculation is 0.72.

[0085] When performing inverse kinematics calculations in layers, starting from layer 1, the joint positions of each layer are calculated sequentially. For layer 2, the knee joint is calculated using the 3D position of the hip joint (parent joint) as a reference, combined with the dynamic bone length of the knee and hip joints (0.378m) and the calibrated spherical coordinates of the knee joint (45°, 160°). An inverse kinematics algorithm, such as the Jacobi iteration method, is used, with 20 iterations and a convergence threshold set to 1×10⁻⁶. -5The iteration step size is 0.1, and those skilled in the art can adjust the corresponding parameters in the actual process; based on the three-dimensional position of the parent joint, the initial coordinates of the three-dimensional position of the joint are given according to the current spherical coordinates of the joint and the dynamic bone length between the parent and child joints; hierarchical weights are introduced for error constraints. The higher the weight, the higher the priority of the constraint of matching the standard bone length and angle of the joint position; the three-dimensional coordinates of the joint are finely adjusted step by step according to a fixed iteration step size. After each fine adjustment, the actual spatial distance and joint angle are compared to see if they fit the constraint conditions. The adjustment is continuously iterated until the convergence accuracy is met or the maximum number of iterations is reached to obtain the preliminary three-dimensional position.

[0086] For example, using a weight of 0.71, the initial three-dimensional position of the knee joint is calculated as (-0.521m, 0.312m, 0.700m); the second-level lumbar spine joint calculation uses the hip joint as a reference, combining the dynamic bone length (0.25m), the spherical coordinates of the lumbar spine joint, and a weight of 0.72 to calculate the initial three-dimensional position of the lumbar spine joint; the third-level ankle joint calculation uses the initial three-dimensional position of the knee joint (parent joint) as a reference, combining the dynamic bone length of the ankle and knee joints (0.291m), the spherical coordinates of the ankle joint, and a weight of 0.69, and continues to calculate the initial three-dimensional position of the ankle joint through inverse kinematics; similarly, the initial three-dimensional positions of the joints at subsequent levels are completed, i.e., joint positions.

[0087] This ensures that joint position calculations conform to human physiological structure, highlights the leading role of highly reliable joint coordinates, and improves the accuracy of joint position calculations; it also ensures the correlation of joint positions at different levels, making the calculation results more consistent with the actual human movement posture.

[0088] The joint positions obtained by performing inverse kinematics calculations layer by layer may contain anomalies that do not conform to the topological constraints of the human skeleton. If these anomalies are directly used as the initial three-dimensional positions of the joints, it will lead to deviations in subsequent motion posture detection. Based on the physiological constraints of the human skeleton and the layered topological relationship, the joint positions at each level are checked for skeletal topological constraints, including whether the length of the bones between joints is consistent with the length of the dynamic bones, whether the joint range of motion is within the range of human physiological activity, and whether there are spatial conflicts between the joint positions between levels.

[0089] For abnormal joint positions detected, adjustments and corrections are made based on their parent joint positions, dynamic bone lengths, joint spherical coordinates, and confidence levels to ensure that the corrected joint positions conform to skeletal topological constraints. The specific rules for skeletal topological constraint verification are aligned with human physiological structures, with an allowable deviation of ±5% in inter-joint bone length, meaning the deviation between the actual calculated inter-joint distance and the dynamic bone length does not exceed 5%. The physiological range of motion for the knee joint is 0°-180°, where knee extension is 0° and flexion is 180°. The physiological range of motion for the ankle joint is -30°-90°, where plantar flexion is -30° and dorsiflexion is 90°. There is no spatial overlap between joint positions at different levels, and no joints exceed the range of motion of their parent joints.

[0090] The joint positions obtained from the aforementioned layered calculations were verified. Taking the ankle joint as an example, the distance between its initial three-dimensional position and that of the knee joint was 0.32m, while the dynamic bone length of the ankle and knee joints was 0.291m. The deviation was (0.32-0.291) / 0.291≈9.97%, exceeding the allowable range of ±5%, and was therefore determined to be an abnormal position. At the same time, the joint angle corresponding to the initial position of the ankle joint was -40°, exceeding the physiological range of -30°-90°, further confirming it as an abnormal position. When correcting abnormal positions, the three-dimensional coordinates of the parent joint used as the reference were maintained. With the original spherical coordinates of the ankle joint remaining constant, the relative orientation trend is maintained. Taking the initial three-dimensional position of the knee joint (parent joint) as the reference, the three-dimensional position of the ankle joint is adjusted by combining the dynamic bone length of the ankle and knee joints (0.291m), the calibrated spherical coordinates of the ankle joint, and the confidence level of the ankle joint. First, the distance between the ankle and knee joints is corrected to 0.291m, and then the joint angle is adjusted to -25° (which conforms to the physiological range of motion). Combining the spatial position of the spherical coordinates of the ankle joint, the three-dimensional position of the ankle joint is corrected to (-0.535m, 0.330m, 0.410m).

[0091] Taking the shoulder joint as an example, its initial three-dimensional position is more than 5% longer than the dynamic bone length of the lumbar spine joint (parent joint), and the joint angle exceeds the range of human physiological activity. Using the same logic as above, combined with the parent joint position, dynamic bone length, spherical coordinates, and confidence level, the abnormal position is corrected. All joint positions at all levels are checked for bone topology constraints and corrected for abnormalities according to the above rules. After correction, they are checked again to ensure that all joint positions conform to bone topology constraints and human physiological structure, thus obtaining the initial three-dimensional positions of all joints. For example, the initial three-dimensional position of the knee joint is (-0.518m, 0.309m, 0.702m), the initial three-dimensional position of the ankle joint is (-0.535m, 0.330m, 0.410m), and the initial three-dimensional position of the lumbar spine joint is (-0.480m, 0.180m, 1.320m). All joint positions meet the constraint requirements.

[0092] This effectively identifies abnormal joint positions in the hierarchical calculation, ensuring that the initial three-dimensional position of the joint conforms to the human physiological structure and skeletal topology, thus guaranteeing the accuracy and rationality of the joint position and preserving the positional advantage of high-confidence joints. It provides reliable basic joint position parameters for the final human motion posture detection, ensuring the accuracy and rationality of the overall posture detection.

[0093] S5. Input the initial 3D position of the low-confidence joint in the current frame and the 3D position sequence of the high-confidence joint in the historical frame into the temporal prediction network to obtain the corrected 3D position. After skeletal topological constraints, merge it with the initial 3D position of the high-confidence joint. After temporal smoothing, output the motion pose of the current frame.

[0094] Specifically, such as Figure 2 As shown, the output motion pose of the current frame includes: Based on the confidence level of the joints, high-confidence joints and low-confidence joints are divided. The initial 3D position of the low-confidence joints in the current frame and the 3D position sequence of the high-confidence joints in the historical frames are extracted. The results are input into the temporal prediction network and the corrected 3D position of the low-confidence joints is predicted according to the hierarchical topology. Skeletal topological constraints are applied to the corrected 3D position and distortion compensation calibration is performed. Dynamic weights are constructed based on the joint confidence to weight and merge the initial 3D position of high-confidence joints with the calibrated corrected 3D position. The merged joint 3D positions are subjected to global topology consistency verification and anomaly correction. Segmented temporal smoothing is performed in combination with segmented temporal features to output the motion pose of the current frame.

[0095] The initial 3D positions of low-confidence joints in the current frame are not accurate enough and are easily affected by projection distortion. Based on the confidence level of the joints, joints with a confidence level not less than a preset confidence threshold are classified as high-confidence joints, and joints with a confidence level less than the confidence threshold are classified as low-confidence joints. The confidence threshold is set in combination with the detection accuracy requirements to ensure the reliability of the positions of high-confidence joints. In the previous frame (a preset number of frames), the initial 3D positions of high-confidence joints are arranged in chronological order to reflect the temporal change pattern of joint movement.

[0096] Based on the temporal correlation characteristics of continuous human joint movement, the preset frame count of 5 frames was determined through multi-scenario testing and calibration. Those skilled in the art can adjust it according to the actual scenario. The temporal prediction network is used to predict the corrected position of low-confidence joints based on temporal features. It adopts a stacked structure of three temporal convolutional layers and two fully connected layers, with a built-in hierarchical topology adaptation layer. The activation function is the ReLU activation function. The training loss function integrates the mean square error loss of joint position prediction and the topological constraint loss of human skeleton. The network is trained based on a self-built panoramic human annotation dataset. Those skilled in the art can adjust the network parameters in actual operation. Its structure adapts to the temporal characteristics of human movement and captures the continuity of joint movement.

[0097] By using the corrected 3D positions of low-confidence joints from the time-series prediction network output, positional bias is initially eliminated. With a pre-set confidence threshold of 0.7, the knee joint (confidence 0.71), hip joint (root node, confidence 0.72), and lumbar spine joint (confidence 0.72) are classified as high-confidence joints; while the ankle joint (confidence 0.69), shoulder joint (confidence 0.68), and elbow joint (confidence 0.67) are classified as low-confidence joints.

[0098] Extract the initial 3D positions of low-confidence joints in the current frame, such as the initial 3D position of the ankle joint (-0.535m, 0.330m, 0.410m) and the initial 3D position of the shoulder joint (-0.280m, 0.150m, 1.520m). Extract the 3D position sequence of high-confidence joints from the previous 5 frames. Taking the knee joint as an example, its historical frame 3D position sequence is [(-0.518m, 0.309m, 0.702m), (-0.520m, 0.311m, 0.700m), (-0.522m, 0.313m, 0.698m), (-0.525m, 0.315m, 0.695m), (-0.528m, 0.317m, 0.692m)], reflecting the temporal movement trend of the knee joint, that is, slow bending and gradual decrease of the Z-axis coordinate.

[0099] The temporal prediction network is configured with a hierarchical topology adaptation layer. The initial 3D position of low-confidence joints and the historical position sequence of high-confidence joints are input into the temporal prediction network. The network associates the motion patterns of parent and child joints according to the hierarchical topology. For example, when predicting the corrected position of the ankle joint (child joint), it combines the historical position sequence of its parent joint (knee joint) to capture the motion correlation between the two. That is, when the knee joint bends, the ankle joint moves down accordingly. The predicted corrected 3D position of the ankle joint is (-0.532m, 0.332m, 0.405m), and the corrected 3D position of the shoulder joint is (-0.285m, 0.155m, 1.515m).

[0100] This provides a reliable temporal reference for the position correction of low-confidence joints, ensuring that the predicted correction position conforms to the human skeletal structure and improving the accuracy of low-confidence joint positions.

[0101] While the corrected 3D positions of low-confidence joints output by the temporal prediction network conform to the temporal motion pattern, they may not conform to the topological constraints of the human skeleton and do not consider the residual effects of projection distortion. Based on the physiological constraints and hierarchical topological relationships of the human skeleton, the corrected 3D positions are verified to meet the constraints such as joint motion angles and bone lengths. A dynamic weight allocation rule is constructed based on joint confidence, with higher weights for high-confidence joints and higher weights for low-confidence joints. The weights are positively correlated with the confidence, ensuring the dominant role of high-confidence joint positions. The initial 3D positions of high-confidence joints and the corrected 3D positions of low-confidence joints are fused according to the dynamic weights to obtain the intermediate 3D position of each joint, forming the position set of all joints in the current frame.

[0102] Specifically, the corrected 3D position of the low-confidence joints is first checked by applying skeletal topological constraints. Taking the ankle joint as an example, the distance between its corrected 3D position (-0.532m, 0.332m, 0.405m) and the initial 3D position (-0.518m, 0.309m, 0.702m) of the parent joint, the knee joint, is 0.298m. The deviation from the dynamic skeletal length (0.291m) of the two is 2.4%, which is within the allowable range of ±5%. The joint angle is -23°, which is within the physiological range of ankle joint activity (-30°-90°), and no correction is required. If the corrected position of a low-confidence joint (such as the elbow joint) is found to be abnormal, it is adjusted to meet the constraint requirements.

[0103] The distortion compensation calibration follows the same logic as described above, correcting only the spherical polar angle and azimuth angle corresponding to joints with low confidence, and then recalculating the three-dimensional coordinates. Taking the ankle joint as an example, its corresponding patch polar angle is 40°, k=cos40°≈0.766, that is, the corrected three-dimensional position coordinates after calibration = corrected three-dimensional position coordinates × k. Substituting into the calculation, the corrected three-dimensional position of the ankle joint after calibration is (-0.532×0.766≈-0.766). 0.407m, 0.332×0.766≈0.254m, 0.405×0.766≈0.310m); the polar angle of the block corresponding to the shoulder joint is 25°, k=cos25°≈0.906, and the corrected three-dimensional position of the shoulder joint after calibration is (-0.285×0.906≈-0.258m, 0.155×0.906≈0.140m, 1.515×0.906≈1.373m).

[0104] Dynamic weights are constructed based on joint confidence levels. The dynamic weight of a joint is calculated as: Joint Confidence Level / Average Confidence Level of All Joints. This ensures that joints with high confidence levels have higher weights. For example, the confidence level of a high-confidence knee joint is 0.71, while the confidence level of a low-confidence ankle joint, after calibration and adjustment, is 0.69. If the average confidence level of all joints is 0.695, then the weight of the knee joint is approximately 0.71 / 0.695 ≈ 1.02, and the weight of the ankle joint is approximately 0.69 / 0.695 ≈ 0.99. The weight of high-confidence joints is... The initial 3D positions are directly used, while the calibrated corrected 3D positions are used for low-confidence joints. During weighted merging, the weight of high-confidence joint positions is higher, while the weight of low-confidence joint positions is relatively lower. For example, the middle 3D position of the ankle joint = calibrated corrected 3D position × 0.99, and the middle 3D position of the knee joint = initial 3D position × 1.02. Since the weight is slightly higher than 1, its dominant role is slightly strengthened. All joints are weighted and merged according to this logic to obtain the final set of middle 3D positions of all joints in the current frame.

[0105] This eliminates the deviation of low-confidence joint correction positions, ensuring that they conform to human physiological structure and projection characteristics, making the merged three-dimensional joint position set more accurate and more in line with the actual human posture.

[0106] The weighted and merged 3D joint positions may have local topological constraints satisfied but global topological inconsistencies, and the merged joint positions may have abrupt changes in temporal sequence. Starting from the overall human skeletal structure, we verify whether the skeletal system composed of all joint 3D positions conforms to the global topological relationship of the human body, including the absence of distortion in the overall skeletal structure, the relative positions between joints conforming to the physiological structure of the human body, and the coherence of joint connections at each level. For anomalies detected by the global topology verification, such as overall skeletal offset and spatial conflicts between joints, we combine segmented temporal features and high-confidence joint positions for global adjustments to ensure the coherence and rationality of the overall skeletal structure.

[0107] By reusing segmented temporal features, different smoothing strategies are adopted for different motion segments. Weak smoothing is used for fast motion segments to preserve motion amplitude, while strong smoothing is used for slow motion segments to eliminate minor fluctuations, ensuring temporal continuity without losing motion features. After global verification, anomaly correction, and temporal smoothing, the human motion posture composed of the three-dimensional positions of all joints accurately reflects the actual motion state of the human body in the current frame, providing historical reference for posture detection in subsequent frames.

[0108] First, verify the 3D position of all joints merged in the current frame, i.e., whether the overall skeletal structure is symmetrical (e.g., the positions of the left and right hip joints and knee joints are symmetrical) and whether the spatial relationship between the trunk joints and limb joints is reasonable (e.g., the relative position of the lumbar spine joint and shoulder joint conforms to the human trunk structure). The verification found that the relative position of the merged shoulder joint and lumbar spine joint deviated significantly, causing a slight distortion of the entire trunk, which was judged as a global topological anomaly. When correcting the anomaly, based on the high-confidence lumbar spine joint and hip joint positions, and combined with the segmented temporal characteristics, the current frame belongs to the slow motion segment, and the trunk movement is gentle. Adjust the middle 3D position of the shoulder joint so that the relative position of the shoulder joint and lumbar spine joint conforms to the topological relationship of the human trunk. The corrected 3D position of the shoulder joint is (-0.260m, 0.142m, 1.370m). Verify again to ensure global topological consistency.

[0109] Based on the inter-frame motion amplitude, if the current frame is determined to be a slow-moving segment, a strong smoothing strategy is adopted, i.e., a moving average algorithm is used, with a window size of 3 frames. Taking the knee joint as an example, its 3D position after merging in the current frame is (-0.518m, 0.309m, 0.702m), and the 3D positions of the previous two frames are (-0.520m, 0.311m, 0.700m) and (-0.522m, 0.313m, 0.698m) respectively. Through moving average calculation, the smoothed 3D position of the knee joint is [(-0.518-0.520-0.522) / 3, (0.309+0.311+0.313) / 3, (0.702+0.700+0.698) / 3] = (-0.520m, 0.311m, 0.700m), eliminating small position fluctuations. If the current frame belongs to a fast-moving segment, a weak smoothing strategy is adopted. The strategy involves using a sliding window of 2 frames to preserve the rapid changes in joint motion and avoid excessive smoothing that could distort the pose. Similarly, the 3D positions of all joints in the current frame are obtained, including the hip joint (-0.486m, 0.208m, 1.078m), knee joint (-0.520m, 0.311m, 0.700m), ankle joint (-0.407m, 0.254m, 0.310m), lumbar joint (-0.480m, 0.180m, 1.320m), and shoulder joint (-0.260m, 0.142m, 1.370m). All joint positions conform to the global topological relationship and are temporally coherent and smooth. The 3D positions of all joints are integrated according to the topological relationship of the human skeleton to form the human motion pose of the current frame. This pose is then output for pose detection in subsequent frames, serving as a source of high-confidence joint position sequences for historical frames.

[0110] This ensures the overall rationality of the current frame's joint position set, eliminates temporal abrupt changes in joint positions, ensures the continuity of motion posture, and adapts to the needs of different motion speeds. The motion posture of the current frame is the final result of human motion posture detection. As a core component of the high-confidence joint 3D position sequence of historical frames in subsequent frame detection, it provides a reliable temporal reference for the temporal prediction of low-confidence joints in subsequent frames.

[0111] Example 2: like Figure 3 As shown, this embodiment provides a human motion posture detection system, including: a processing module for acquiring a panoramic RGB video stream, splitting each frame image into multi-view tiles after spherical projection, performing spherical convolution on the multi-view tiles, performing temporal convolution on sparse sampling of consecutive frames of images, and comprehensively outputting spatiotemporal features; The detection module is used to input spatiotemporal features into the first detection network and output the spherical coordinates and confidence of the joint in the projected coordinate system. The spherical coordinates with the highest confidence in the multi-view tiles at the same time are selected as the spherical coordinates of the corresponding joint. The localization module is used to input spatiotemporal features into the second detection network, output the spherical coordinates of the root node and the corresponding feature vector, estimate the depth of the root node through the depth regression network, and calculate the three-dimensional position of the root node by combining the corresponding spherical coordinates. The solver module is used to solve the initial three-dimensional position of the joint by using the three-dimensional position of the root node as a reference, combining the spherical coordinates of the joint and the topological constraints of the bone, as well as the dynamic bone length obtained by distortion adaptation correction of the preset bone length. The fusion module is used to input the initial 3D position of the low-confidence joints in the current frame and the 3D position sequence of the high-confidence joints in the historical frames into the temporal prediction network to obtain the corrected 3D position. After skeletal topological constraints, it is merged with the initial 3D position of the high-confidence joints and then output the motion pose of the current frame after temporal smoothing.

[0112] The above description is merely a preferred embodiment of this application. The scope of protection of this application is not limited to the above embodiments. All technical solutions falling within the scope of this application's concept are within the scope of protection of this application. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of this application should also be considered within the scope of protection of this application.

Claims

1. A method for detecting human motion posture, characterized in that, include: The panoramic RGB video stream is acquired, and each frame image is spherically projected and split into multi-view tiles. Spherical convolution is performed on the multi-view tiles, and temporal convolution is performed after sparse sampling of multiple consecutive frames of images. The spatiotemporal features are then output. The spatiotemporal features are input into the first detection network, and the spherical coordinates and confidence scores of the joints in the projected coordinate system are output. The spherical coordinates with the highest confidence scores are selected from the multi-view tiles at the same time as the spherical coordinates of the corresponding joints. The spatiotemporal features are input into the second detection network, which outputs the spherical coordinates of the root node and the corresponding feature vector. The depth of the root node is estimated by the depth regression network, and the three-dimensional position of the root node is calculated by combining the corresponding spherical coordinates. Based on the three-dimensional position of the root node, combined with the spherical coordinates of the joint and the topological constraints of the bone, as well as the dynamic bone length obtained by distortion adaptation correction of the preset bone length, the initial three-dimensional position of the joint is solved by inverse kinematics. The initial 3D positions of low-confidence joints in the current frame and the 3D position sequences of high-confidence joints in historical frames are input into the temporal prediction network to obtain the corrected 3D positions. After skeletal topological constraints, these positions are merged with the initial 3D positions of the high-confidence joints and then temporally smoothed to output the motion pose of the current frame.

2. The method for detecting human motion posture according to claim 1, characterized in that, The integrated output spatiotemporal features include: Based on the position of multi-view tiles in the projected coordinate system, the spatial features obtained by spherical convolution are weighted to obtain weighted spatial features. The image after sparse sampling of multiple consecutive frames is segmented according to the inter-frame motion amplitude, and the corresponding temporal convolution is used to extract features for different motion segments to obtain segmented temporal features. Based on the timestamp and projection coordinate system, the weighted spatial features and segmented temporal features are aligned. Weight allocation and feature fusion are performed through a gated fusion unit. The fused features are then corrected by topological normalization according to the spherical projection curvature distribution, and spatiotemporal features are output.

3. The method for detecting human motion posture according to claim 2, characterized in that, The multi-view image patch is obtained by splitting the image after spherical projection of each frame in the panoramic RGB video stream according to a preset polar angle interval and azimuth angle interval; the weighting processing is to construct a spherical distortion weight matrix based on the polar angle and azimuth angle corresponding to the multi-view image patch, and to weight the spatial features; the motion segment includes a fast motion segment and a slow motion segment, and the fast motion segment is used to extract features through dilated temporal convolution, while the slow motion segment is used to extract features through ordinary temporal convolution.

4. The method for detecting human motion posture according to claim 2, characterized in that, The output joint's spherical coordinates and confidence level in the projected coordinate system include: The spatiotemporal features are decomposed into tile spatiotemporal features, which are input into the first detection network and the network is parsed and output as the initial spherical coordinates, initial confidence and feature response values ​​of all candidate joints in the corresponding tile. Candidate joints with feature response values ​​less than the response threshold are removed, and the initial spherical coordinates of the remaining candidate joints are calibrated for distortion compensation. The calibrated spherical coordinates and their corresponding initial confidence scores are summarized in all multi-view tiles of the same joint at the same time. The initial confidence scores are corrected based on the spatial overlap of the calibrated spherical coordinates to obtain the confidence score of the joint. The calibrated spherical coordinates with the highest confidence scores are selected as the spherical coordinates of the joint.

5. The method for detecting human motion posture according to claim 4, characterized in that, The first detection network includes branch detection units corresponding to multi-view tiles, and the branch detection units are adapted to the projection distortion degree of the corresponding tiles; the initial spherical coordinates are the calibration coordinates in the projection coordinate system.

6. The method for detecting human motion posture according to claim 3, characterized in that, The calculation of the three-dimensional position of the root node includes: The spatiotemporal features of the map tiles are input into the second detection network, which outputs the spherical coordinates of the root node and the corresponding feature vector, and filters out the spatiotemporal features of the target map tiles that match the feature vector. The spatiotemporal features of the target map are input into a depth regression network for depth estimation to obtain the root node depth and depth confidence. The root node depth is then weighted and calibrated, and the spherical coordinates of the root node are calibrated for distortion compensation. Based on the mapping relationship between the projected coordinate system and the Cartesian coordinate system, and combined with the calibrated spherical coordinates and the calibrated root node depth, the three-dimensional position of the root node is calculated.

7. The method for detecting human motion posture according to claim 6, characterized in that, The deep regression network includes regression branches corresponding to multi-view tiles, and the regression branches are adapted to the projection distortion degree of the corresponding target tiles; tiles whose spatiotemporal features and feature vectors have a similarity greater than a similarity threshold are selected as spatiotemporal features of the target tiles; Root node depths with depth confidence less than a preset threshold are removed. The spatial fit between the remaining root node depths and the spherical coordinates of the root nodes is calculated. The spatial fit and depth confidence are weighted and fused together to obtain the weight, and then the weighted sum is obtained to obtain the calibrated root node depth.

8. The method for detecting human motion posture according to claim 6, characterized in that, The initial three-dimensional position of the joint to be solved includes: A hierarchical topology is constructed according to the human skeleton hierarchy. Hierarchical weights are assigned based on the confidence of joints. Inverse kinematics calculations are performed hierarchically based on dynamic bone length and spherical coordinates of joints to obtain joint positions. The initial three-dimensional position of the joint is obtained by performing skeletal topology constraint verification and abnormal position correction on the joint positions at different levels.

9. A method for detecting human motion posture according to claim 8, characterized in that, The output of the motion pose of the current frame includes: Based on the confidence level of the joints, high-confidence joints and low-confidence joints are divided. The initial 3D position of the low-confidence joints in the current frame and the 3D position sequence of the high-confidence joints in the historical frames are extracted. The results are input into the temporal prediction network and the corrected 3D position of the low-confidence joints is predicted according to the hierarchical topology. Skeletal topological constraints are applied to the corrected 3D position and distortion compensation calibration is performed. Dynamic weights are constructed based on the joint confidence to weight and merge the initial 3D position of high-confidence joints with the calibrated corrected 3D position. The merged joint 3D positions are subjected to global topology consistency verification and anomaly correction. Segmented temporal smoothing is performed in combination with segmented temporal features to output the motion pose of the current frame.

10. A human motion posture detection system, used to implement the human motion posture detection method according to any one of claims 1-9, characterized in that, include: The processing module is used to acquire panoramic RGB video streams, spherically project each frame of image and split it into multi-view tiles, perform spherical convolution on the multi-view tiles, perform temporal convolution after sparse sampling of multiple consecutive frames of images, and output spatiotemporal features in combination. The detection module is used to input spatiotemporal features into the first detection network and output the spherical coordinates and confidence of the joint in the projected coordinate system. The spherical coordinates with the highest confidence in the multi-view tiles at the same time are selected as the spherical coordinates of the corresponding joint. The localization module is used to input spatiotemporal features into the second detection network, output the spherical coordinates of the root node and the corresponding feature vector, estimate the depth of the root node through the depth regression network, and calculate the three-dimensional position of the root node by combining the corresponding spherical coordinates. The solver module is used to solve the initial three-dimensional position of the joint by using the three-dimensional position of the root node as a reference, combining the spherical coordinates of the joint and the topological constraints of the bone, as well as the dynamic bone length obtained by distortion adaptation correction of the preset bone length. The fusion module is used to input the initial 3D position of the low-confidence joints in the current frame and the 3D position sequence of the high-confidence joints in the historical frames into the temporal prediction network to obtain the corrected 3D position. After skeletal topological constraints, it is merged with the initial 3D position of the high-confidence joints and then output the motion pose of the current frame after temporal smoothing.

Images