Human body recognition method and device based on interactive space-time Transform, medium and program product

Abstract

The invention belongs to the technical field of image recognition, and discloses a human body recognition method and device based on interactive space-time Transform, a medium and a program product, and the method comprises the steps: obtaining an ordered image frame sequence; extracting position information of human body key points, and constructing a posture feature vector corresponding to the frame of image; generating a time sequence coding sequence fused with time information; processing and outputting an advanced feature sequence containing a global time sequence association relationship; each discriminant quantity in the discriminant quantity sequence represents the possibility of whether the corresponding time position is an action stage boundary or not; and action stage division of the operation process recorded by the continuous video data is realized. According to the method, the time sequence association and stage conversion characteristics of the human body posture can be adaptively captured without manually pre-defining stage rules or depending on a fixed time window, and the problems of inaccurate stage division and inconsistent evaluation results caused by individual differences of operating personnel in the prior art are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a human body recognition method, device, medium, and program product based on an interactive spatiotemporal Transformer. Background Technology

[0002] In training, assessment, and monitoring of pole climbing operations in power systems, it is necessary to divide the operation process of personnel into stages and conduct standardized evaluations to ensure operational safety and consistent operating standards. Pole climbing operations include continuous and purposeful operational stages such as pole lifting, ascent, positioning, stabilization, and operation. However, significant individual differences exist in actual execution. Specifically, different personnel exhibit varying rhythms of movement and durations of stages, and there are often blurred transitional intervals between stages.

[0003] Existing behavior recognition methods based on video or skeleton sequences mainly rely on manually pre-defined and labeled action stages, or segmented analysis using fixed-length time windows. However, manual stage segmentation depends on experience and is highly subjective; fixed time windows cannot adapt to individual differences and environmental changes, easily leading to stage misjudgments, directly affecting the objectivity and consistency of evaluation results. Furthermore, these methods separate stage segmentation from behavior recognition, resulting in poor model generalization ability, making it difficult to adapt to the real-world needs of power pole climbing operations, and failing to effectively solve the misjudgment problem caused by rhythm differences. Summary of the Invention

[0004] In response, the present invention provides a human body recognition method, device, medium, and program product based on interactive spatiotemporal Transformer, so as to at least partially solve the above-mentioned technical problems.

[0005] This invention provides a human body recognition method based on an interactive spatiotemporal Transformer, comprising the following steps:

[0006] Continuous video data of the workers is acquired, and the video data is subjected to frame sequence extraction and preprocessing to obtain an ordered image frame sequence;

[0007] For each frame of image, extract the position information of human key points, and construct the pose feature vector corresponding to the frame of image based on the position information of human key points.

[0008] The attitude feature vectors arranged in chronological order are used to form an attitude temporal feature sequence, and the attitude temporal feature sequence is encoded with time position to generate a temporal encoded sequence that integrates time information.

[0009] The temporal encoded sequence is input into the temporal feature association modeling unit, and after processing, a high-level feature sequence containing global temporal associations is output. The temporal feature association modeling unit is based on a multi-layer Transformer structure with a temporal attention mechanism.

[0010] The high-level feature sequence is input into the stage boundary discrimination output unit, which generates a discrimination quantity sequence based on the high-level feature sequence. Each discrimination quantity in the discrimination quantity sequence represents the probability of whether the corresponding time position is the boundary of the action stage.

[0011] Based on the changing characteristics of the discriminant sequence, the boundary points of multiple action stages are adaptively determined, thereby realizing the division of action stages in the operation process recorded by continuous video data.

[0012] In one possible embodiment, constructing the pose feature vector corresponding to the frame image includes:

[0013] Based on the location information of the key points of the human body, multi-dimensional features representing the human posture are calculated. The multi-dimensional features include at least the coordinates of the key points, the relative angles between key joints, or the distances between key joints.

[0014] The multidimensional features are normalized based on human scale to obtain the pose feature vector.

[0015] In one possible embodiment, the temporal feature association modeling unit includes L layers of Transformer encoder layers, each encoder layer sequentially including a multi-head self-attention mechanism module, a feedforward neural network module, and a layer normalization and residual connection module; the multi-head self-attention mechanism module is used to calculate the association weights between features at different time positions in the temporal encoded sequence, where L is an integer greater than 1.

[0016] In one possible embodiment, the stage boundary discrimination output unit is a structure containing at least one fully connected neural network, and the discrimination sequence is obtained by performing nonlinear mapping and probability transformation on the high-level feature sequence.

[0017] In one possible embodiment, adaptively determining the boundary points of multiple action stages based on the changing characteristics of the discriminant sequence includes:

[0018] The discriminant sequence is smoothed by filtering to obtain a smoothed discriminant curve;

[0019] Extract the local extreme points that satisfy the peak condition in the smoothed discriminant curve and use them as candidate points for the stage boundary;

[0020] Based on a preset minimum stage time interval threshold, the candidate points for the stage boundary are filtered, and candidate points with too close intervals are removed to obtain the final stage boundary points.

[0021] In one possible embodiment, after the action phase division is completed, the method further includes:

[0022] Based on each action stage after division, extract the statistical features or change features of the posture feature vector within that stage;

[0023] The statistical characteristics or change characteristics are compared with the preset normative indicators for the corresponding action stage to determine whether the operator's operation in that action stage is standardized.

[0024] In another aspect, this application also provides a human body recognition system based on an interactive spatiotemporal Transformer, comprising:

[0025] The preprocessing module is used to acquire continuous video data of the workers, extract and preprocess the video data to obtain an ordered image frame sequence;

[0026] The pose feature vector construction module is used to extract the position information of human key points in each frame of image, and construct the pose feature vector corresponding to the frame of image based on the position information of human key points.

[0027] The temporal coding sequence generation module is used to construct an attitude temporal feature sequence from the attitude feature vectors arranged in chronological order, and to perform time position encoding on the attitude temporal feature sequence to generate a temporal coding sequence that integrates time information.

[0028] The advanced feature sequence generation module is used to input the temporal encoded sequence into the temporal feature association modeling unit, and output an advanced feature sequence containing global temporal association relationships after processing. The temporal feature association modeling unit is based on a multi-layer Transformer structure with a temporal attention mechanism.

[0029] The discriminant sequence generation module is used to input the high-level feature sequence to the stage boundary discrimination output unit. The stage boundary discrimination output unit generates a discriminant sequence based on the high-level feature sequence. Each discriminant in the discriminant sequence represents the probability of whether the corresponding time position is the boundary of the action stage.

[0030] The stage division module is used to adaptively determine the boundary points of multiple action stages based on the change characteristics of the discriminant sequence, thereby realizing the division of action stages of the operation process recorded by continuous video data.

[0031] This application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the human body recognition method based on the interactive spatiotemporal Transformer as described above.

[0032] In another aspect, this application provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor to implement the human body recognition method based on interactive spatiotemporal Transformer as described above.

[0033] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the human body recognition method based on the interactive spatiotemporal Transformer as described above.

[0034] This application constructs a temporal feature association modeling unit using a multi-layer Transformer structure based on a temporal attention mechanism. Combined with a stage boundary discrimination output unit, it achieves integrated processing of posture feature modeling and stage boundary discrimination. This eliminates the need for manually predefined stage rules or reliance on fixed time windows, and adaptively captures the temporal association and stage transition features of human posture. It effectively solves the problems of inaccurate stage division and inconsistent evaluation results caused by individual differences among operators in existing technologies. It significantly improves the model's adaptability and generalization ability to complex working environments and different action rhythms, reduces system configuration and maintenance costs, and provides objective and reliable technical support for training, evaluation, and monitoring of power pole climbing operations, ensuring the uniformity and accuracy of operational standardization assessments. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0037] Figure 1 This is a schematic diagram of a human body recognition method based on an interactive spatiotemporal Transformer, provided as an embodiment of the present invention.

[0038] Figure 2 A schematic diagram of the temporal feature association modeling unit structure provided in the embodiments of this disclosure.

[0039] Figure 3 This is a schematic diagram of the action phase division process provided in the embodiments of this disclosure.

[0040] Figure 4 This is a schematic diagram of the structure of a human body recognition system based on an interactive spatiotemporal Transformer provided in an embodiment of the present invention.

[0041] Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] It should be noted that all user information (including but not limited to user device information, user personal information, object information corresponding to device usage data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, device usage data, etc.) involved in all embodiments of this disclosure are information and data authorized by the user or fully authorized by all parties.

[0044] The implementation process of the human body recognition method based on interactive spatiotemporal Transformer described in this invention will be described in detail below with reference to specific embodiments. It should be noted that this embodiment is only used to explain this invention and is not intended to limit the scope of protection of this invention. Conventional adjustments or substitutions made by those skilled in the art to each step without departing from the concept of this invention should be included within the scope of protection of this invention.

[0045] like Figure 1 As shown in the figure, an embodiment of the present invention discloses a human body recognition method based on an interactive spatiotemporal Transformer, schematic diagram 100, which includes the following method steps:

[0047] This method is applicable to training, assessment, and monitoring scenarios for pole climbing operations in power systems. Video can be continuously captured by fixed cameras on the ground or at a certain height, and must completely record the overall human movement of the operator from the ground to the top of the pole. Individual differences in height, body type, movement rhythm, and skill level are acceptable among operators, and short pauses, posture adjustments, or rhythm changes are permitted during the pole climbing process. This method uses extractable human posture information from the video sequence as input and can adapt to complex backgrounds, non-ideal lighting, and inconsistent movement rhythms in actual working environments. It is used for the automatic division of each stage of the pole climbing operation and the judgment of the standardization of operations at each stage.

[0048] In some embodiments, for step S1, for example, a fixed camera is used to capture video of the pole climbing operation. The installation position of the camera can be adjusted according to the needs of the operation scenario. For example, it can be installed on the ground at a horizontal distance of 5-10 meters from the pole climbing point, or on the middle crossarm of an adjacent tower, with its height higher than half the height of the tower being climbed, to ensure that the full-body movements of the operator and the relative position of the towers can be clearly captured. The acquisition parameters of the camera can be configured as follows: frame rate set to 25fps-30fps, resolution set to 1920×1080 pixels, and encoding format using H.264 to ensure the clarity and continuity of the video data while controlling the amount of data stored. During the acquisition process, the camera must remain fixed to avoid video frame blurring or posture extraction errors caused by equipment shaking. The acquisition time must cover the entire process from the operator's preparation to the completion of the pole climbing operation, for example, the acquisition time is 3-10 minutes.

[0049] The captured continuous pole-climbing video was processed by extracting frame sequences. Based on the original frame rate, the video was broken down into a sequence of image frames arranged chronologically. Each image frame corresponds to a unique time position identifier. For example, with the start time of the video as the zero point, each frame corresponds to an interval of 1 / 30th of a second, with time position identifiers t0, t1, t2, ..., t... n-1 Where n is the total number of video frames. During frame sequence extraction, video stream data needs to be read using a video decoding tool and stored frame by frame in an image file format, such as JPEG or PNG, to ensure that the frame order is not disordered and that there are no lost or duplicate frames, so as to guarantee the accuracy of subsequent time series modeling.

[0050] Preferably, the preprocessing includes noise reduction and foreground / background separation, with the aim of reducing background interference, optimizing image quality, and providing a good foundation for human body key point extraction.

[0051] Specifically, firstly, a Gaussian filtering algorithm is used to denoise each frame of the image. The kernel size of the Gaussian filter can be selected as 5×5, and the standard deviation σ can be configured to 1.0. Convolution operations are then used to smooth the image and remove Gaussian noise. The formula for the convolution operation is: In this context, (x,y) represents the relative coordinates within the filter kernel, and G(x,y) represents the filter weight at the corresponding position. Noise reduction is achieved by weighting and summing the weights with the image pixel values.

[0052] Secondly, image grayscale processing is performed to convert the color image into a grayscale image, reducing the interference of color information on pose extraction. Grayscale conversion uses a weighted average method, i.e., grayscale value Gray = 0.299R + 0.587G + 0.114B, where R, G, and B are the pixel values ​​of the red, green, and blue channels of the color image, respectively. This method can better preserve the brightness information of the image, meeting the requirements of human pose extraction for image detail.

[0053] Finally, an adaptive thresholding algorithm is used to separate the foreground and background of the grayscale image, highlighting the human body region of the workers. The adaptive threshold is calculated based on the pixel mean and variance of the local image region. For example, using a 15×15 local window as a unit, the segmentation threshold T(x,y) for each pixel is dynamically determined: Where mean(x,y) is the average pixel value within a local window centered at (x,y), and C is a constant that can be selected from 2 to 5. This dynamic threshold divides the image pixels into the foreground (human body region) and the background, achieving effective separation between the two and eliminating background interference for subsequent human body key point extraction.

[0054] In some embodiments, for step S2, for each preprocessed image frame, existing mature human pose estimation algorithms are used to extract the human key point location information of the worker. For example, OpenPose algorithm, HRNet algorithm or AlphaPose algorithm can be used. These algorithms can all achieve high-precision two-dimensional human key point detection.

[0055] For example, the extracted human key points include the major joint positions of the head, torso, upper limbs, and lower limbs, specifically including but not limited to: the vertex of the head, the center point of both eyes, the tip of the nose, the neck point, the two points on the left and right shoulders, the two points on the left and right elbows, the two points on the left and right wrists, the two points on the left and right hips, the two points on the left and right knees, the two points on the left and right ankles, and the two points on the left and right feet, totaling 21 key points. The position of each key point is represented by image pixel coordinates (x, y), where x is the horizontal coordinate and y is the vertical coordinate, with the origin at the top left corner of the image. To ensure the stability of the temporal analysis, the numbering and topological relationships of each key point remain consistent throughout the entire frame sequence. For example, the neck point is always numbered 6, the left shoulder point is numbered 7, and the right shoulder point is numbered 8. Furthermore, the connection relationships between key points, such as the neck point being connected to the left and right shoulder points, and the left shoulder point being connected to the left elbow point, remain fixed to avoid errors in the construction of the pose feature vector due to incorrect key point numbering.

[0056] Preferably, the multidimensional features representing human posture are calculated based on the position information of key points on the human body, including at least the coordinates of key points, the relative angles between key joints, or the distances between key joints.

[0057] Specifically, the key point coordinate features are directly derived from the extracted human key point pixel coordinates (x, y). For 21 key points, a total of 42-dimensional coordinate features are obtained. The relative angle calculation between key joints selects key joints such as the neck, shoulder, elbow, hip, knee, and ankle. For example, the angle between the neck and shoulder, the angle between the shoulder and elbow, the angle between the elbow and wrist, the angle between the hip and knee, and the angle between the knee and ankle. The angle of each joint is calculated using the vector dot product formula.

[0058] For joints A, B, and C, the joint angle θ satisfies: in Let A be a vector from B. Let B be the vector from B to C. and These are the magnitudes of two vectors. The angle value of each key joint can be calculated using this formula. Ten key joints are selected, forming a 10-dimensional angular feature. The distances between key joints are calculated using the following formulas: the distance between two points on the shoulder, the distance between two points on the hip, the distance between the hands, the distance between the feet, and the distance from the top of the head to the midpoint of the hip. The Euclidean distance formula is used for the two-dimensional coordinates (x1, y1) and (x2, y2). Eight key joint distances were selected to form an 8-dimensional distance feature. Through the above calculations, a multi-dimensional feature with dimensions of 42+10+8=60 was obtained, which comprehensively represents the human posture state in the image frame.

[0059] Preferably, the multidimensional features are normalized based on human scale to reduce the impact of individual height and body shape differences on posture features. The normalization process uses human height as a benchmark. Specifically, the vertical distance from the top of the head to the sole of the foot is calculated as the height H, which is the absolute value of the difference between the y-coordinate of the top of the head and the y-coordinate of the sole. All keypoint coordinate features and key joint distance features are divided by the height H to obtain relative values; key joint angle features retain their original values ​​because angle information is not directly affected by height and body shape differences.

[0060] For example, the formula for normalization is as follows: for coordinate features , For distance features For angular features ,in, , These are the original keypoint coordinates. This represents the original inter-joint distance. The original joint angle, , , , These are the normalized feature values. All normalized features are arranged in a fixed order to form the attitude feature vector. For example, the attitude feature vector can be represented as V=[ , , , ,…, , , , ,…, , , ,…, ],in to Normalized coordinates for 21 key points, to For the angles of 10 key joints, to The normalized distance between the eight key joints is represented by a 60-dimensional vector. This pose feature vector primarily reflects the relative relationships between different parts of the human body, rather than absolute scale, thereby improving the universality and robustness of the features.

[0061] In some embodiments, for step S3, firstly, the pose feature vectors corresponding to each time position are arranged in chronological order to form a pose temporal feature sequence. Assume the video frame sequence has n frames, corresponding to time positions t0 to t... n-1 The pose feature vector corresponding to each time position is V0 to V n-1Then the attitude temporal feature sequence can be represented as S=[V0,V1,V2,…,V n-1 The sequence of temporal features, S, has a dimension of n×D, where D is the dimension of a single attitude feature vector, for example, D=60. This temporal feature sequence fully reflects the evolution of the worker's attitude over time during pole climbing, and is used for subsequent temporal modeling and stage boundary identification. It integrates discrete single-frame attitude features into continuous temporal data, meeting the input requirements of the Transformer model for sequence data.

[0062] Since the attitude feature vector itself does not contain explicit temporal sequence information, direct temporal modeling may cause the model to be unable to distinguish features at different time positions. Therefore, it is necessary to introduce temporal position encoding to integrate time information into the attitude feature vector and generate a temporal encoding sequence that integrates time information.

[0063] In this embodiment, the time position encoding adopts a learnable parameterized encoding method, which is implemented as follows:

[0064] First, define a trainable positional encoding matrix P∈R^(N_max×D), where N_max is the maximum allowed sequence length during training and inference, which can be set according to the dataset and actual application scenario, for example, 1024. Each row of this matrix corresponds to a positional encoding vector for a time step. During training and inference, based on the actual length n of the current input sequence (n≤N_max), the first n rows are taken from matrix P to obtain the positional encoding sequence [P0,P1,...,P...]. n-1 The initial values ​​of the location encoding matrix P are randomly initialized and follow a normal distribution with a mean of 0 and a variance of 0.02, denoted as t for each time position. i (i=0,1,…,n-1) Assign a position encoding vector P of dimension D. i ;

[0065] Secondly, the position encoding vector P i With the corresponding attitude feature vector V i Element-wise addition is performed to obtain the pose feature vector fused with temporal information. V i +P i The element-wise addition method enables deep fusion of temporal information and pose features without changing the feature dimensions; finally, the fused feature vector is... to Form a new time-series coding sequence The sequence dimension remains n×D, which is used for subsequent temporal feature association modeling.

[0066] Alternatively, the time position code can also be generated using a fixed method, such as using sine and cosine functions to generate the position code, with the specific formula being... Where i is the temporal position index, k is the encoding dimension index, and D is the dimension of the pose feature vector. When using a fixed generation method, the position encoding vector does not require training optimization; it is directly calculated according to the above formula and then fused with the pose feature vector. Its advantages are simple calculation, no need for additional training parameters, and the ability to effectively distinguish features at different time positions, making it particularly suitable for scenarios with limited data. Regardless of whether learnable parameterized encoding or fixed generation encoding is used, the goal is to enable the model to recognize the temporal relationship of pose features, ensuring the accuracy of temporal modeling. Both encoding methods can achieve the core idea of ​​this invention; the specific choice can be determined based on the actual application scenario and performance requirements.

[0067] In some embodiments, for step S4, the temporal feature association modeling unit adopts a multi-layer Transformer structure based on the temporal attention mechanism to process the temporal encoded sequence and output a high-level feature sequence containing global temporal associations. This solves the problem that the prior art cannot effectively capture the temporal dependency of posture and realizes deep modeling of the evolution process of human motion state.

[0068] Please see Figure 2 , Figure 2 This is a schematic diagram of the temporal feature association modeling unit structure provided in an embodiment of this disclosure. The temporal feature association modeling unit includes an L-layer Transformer encoder layer, and each encoder layer sequentially includes a multi-head self-attention mechanism module, a feedforward neural network module, and a layer normalization and residual connection module.

[0069] In this embodiment, L can be selected as 6 layers. This layer count ensures the model's ability to capture long-term temporal dependencies while controlling the model's computational complexity and training cost. The input to each encoder layer is the output feature sequence of the previous layer, where the input to the first encoder layer is a temporal encoded sequence. The output is the feature sequence after processing by this layer.

[0070] Specifically, for the l-th encoder layer (l=1,2,…,L), its input feature sequence is X. (l-1)∈R (n×D), the output feature sequence is X. (l)∈R (n×D), where n is the number of time steps, i.e., the number of video frames, and D is the feature dimension, for example, D=60. The stacked structure of multi-layer encoders can progressively capture short-term correlations in pose temporal features, such as pose changes between adjacent frames, and long-term dependencies, such as pose similarity across multiple frames, thus achieving hierarchical modeling of the evolution of human motion states.

[0071] The multi-head self-attention mechanism module is responsible for calculating the correlation weights between the features at each time point and the features at all other time points, thereby fusing global temporal information.

[0072] Specifically, firstly, for the input feature sequence X (l-1) Linear projection is performed to obtain the query vector Q, key vector K, and value vector V. Assuming the multi-head self-attention mechanism has h heads, and the feature dimension of each head is d_v = D / h, in this embodiment, h can be selected as 8, and d_v = 7 or 8, which can be adjusted according to the specific value of D. For example, when D = 60, d_v = 7. The remaining dimensions are padded with zeros, and the result is obtained through three different linear projection matrices W_Q∈R. (D×h×d_v) W_K∈R (D×h×d_v) W_V∈R (D×h×d_v) X (l-1) Projected as Q∈R (n×h×d_v) K∈R (n×h×d_v)、 V∈R (n×h×d_v) The purpose of linear projection is to map the original features to the attention space for subsequent association weight calculation. The parameters of the three projection matrices are obtained through model training and optimization.

[0073] Next, the attention weights for each attention head are calculated. For the k-th head (k=1,2,…,h), the query vector Q_k∈R (n×d_v)、 Key vector K_k∈R (n×d_v) Value vector V_k∈R (n×d_v) Attention weight matrix A_k∈R (n×n) The calculation formula is in The scaling factor is used to mitigate the problem of excessively large inner product results and vanishing gradients in the Softmax function caused by a large d_v, ensuring a reasonable distribution of attention weights. The element A_k(i,j) in the attention weight matrix A_k represents the attention level of the feature at time position i to the feature at time position j, with a value range of [0,1]. The sum of all column elements is 1, reflecting the correlation strength between the posture features of two time positions. For example, during the ascent phase of pole climbing, the posture changes of adjacent time positions are continuous, and the corresponding attention weight A_k(i,i+1) will be significantly higher than the weights of other non-adjacent positions. In the stable phase, the posture features of each time position are more similar, and the corresponding attention weight distribution is relatively uniform.

[0074] Then, the output features of each attention head are calculated. The attention weight matrix A_k is multiplied by the value vector V_k to obtain the output features of the k-th head. ∈R (n×d_v) : This step integrates global temporal information into the features of each time location by weighted summation, so that the output features of each time location not only contain its own pose information, but also integrate the pose features of other related time locations.

[0075] Finally, the output features of the h attention heads are concatenated to obtain the intermediate output O∈R of the multi-head self-attention mechanism. (n×D) , where D=h×d_v, which can be understood as adjusting for dimensional mismatches through linear interpolation or zero padding; To further optimize the feature representation, the concatenated output O is projected through a linear projection matrix W_O∈R. (D×D) The transformation yields the final output of the multi-head self-attention mechanism module. ∈R (n×D) : , where the linear projection matrix Its function is to fuse and adjust the dimensions of the output features of multiple attention heads to ensure the consistency and expressive power of the output features. Its parameters are determined through training optimization.

[0076] The feedforward neural network module performs a nonlinear transformation on the output features of the multi-head self-attention mechanism module, enhancing the model's feature representation ability and addressing the problem that linear models struggle to fit complex pose temporal features. This module is implemented using a two-layer fully connected neural network, with nonlinearity introduced through the ReLU activation function.

[0077] Specifically, the first fully connected layer has an input dimension of D and an output dimension of 4D. For example, when D=60, the output dimension is 240, and the weight matrix W1∈R (D×4D) Bias vector b1∈R (4D) The formula for calculating the output feature F1 is: The ReLU activation function is expressed as ReLU(x)=max(0,x), which can effectively introduce nonlinearity while avoiding the gradient vanishing problem and enhancing the model's ability to fit complex features.

[0078] The second fully connected network has an input dimension of 4D, an output dimension of D, and a weight matrix W2∈R. (4D×D) Bias vector b2∈R (D) The formula for calculating the output feature F2 is: The second fully connected layer maps the high-dimensional nonlinear features back to the original feature dimensions, ensuring consistency in feature dimensions for subsequent layer normalization and residual connection operations. The output F2 of the feedforward neural network module is the feature sequence after nonlinear transformation, which contains richer pose temporal correlation information and nonlinear feature patterns.

[0079] Each encoder layer contains a layer normalization and residual connection module, which accelerates model training, avoids gradient vanishing or gradient exploding problems, and improves the training stability and convergence speed of the model.

[0080] The layer normalization module is used to normalize the input features, making the mean of the features 0 and the variance 1, thus eliminating the influence of feature scale differences on training. The specific formula is as follows: Where μ is the mean of the feature, σ 2 ε is the variance of the feature, ε is the minimum value (e.g., 1e-6) to avoid a denominator of 0, and γ and β are learnable scaling and offset parameters obtained through training optimization. Their function is to preserve the non-linear expressiveness of the feature and avoid the loss of feature information caused by normalization.

[0081] The residual connection module is used to add the input features of the encoder layer to the features processed by the multi-head self-attention mechanism and the feedforward neural network. It alleviates the vanishing gradient problem through skip connections. Specifically, in the multi-head self-attention mechanism module, the residual connection is as follows: In this embodiment, the residual connections are performed first, followed by layer normalization. This order effectively improves the training stability and convergence speed of the model, and is more suitable for modeling temporal features compared to the approach of normalization followed by residual connections. In the feedforward neural network module, the residual connections are as follows: After the above processing That is, the output feature sequence X of the l-th encoder layer. (l) By combining layer normalization and residual connections, we can ensure that the model maintains a stable training process even with deep stacking layers, while preserving the feature representation ability of each layer.

[0082] After processing by the L-layer Transformer encoder, the final output feature sequence X is obtained. (L) This refers to a high-level feature sequence that incorporates global temporal correlations. Each feature vector in this high-level feature sequence not only contains posture information at the corresponding time position but also integrates temporal correlation information across the entire scope, enabling it to accurately characterize the evolution of human movement states. For example, during the ascending phase of pole climbing, the high-level feature sequence captures the temporal patterns of periodic upper and lower limb movements; during the stable phase, the high-level feature sequence exhibits characteristics of small posture changes and stable features. This high-level feature sequence provides rich and effective feature evidence for subsequent phase boundary discrimination, solving the problem of inaccurate boundary discrimination caused by existing technologies relying solely on single-frame or local frame features.

[0083] In some embodiments, for step S5, the stage boundary discrimination output unit generates a discrimination quantity sequence based on the high-level feature sequence. Each discrimination quantity represents the possibility of whether the corresponding time position is the boundary of the action stage, realizing the integrated processing of posture feature modeling and stage boundary discrimination, and solving the problem of poor adaptability caused by the separation of stage division and behavior recognition in the prior art.

[0084] Preferably, the stage boundary discrimination output unit is a structure containing at least one fully connected neural network. In this embodiment, a two-layer fully connected neural network structure is used, which can control the model complexity while ensuring discrimination accuracy.

[0085] The input to the stage boundary discrimination output unit is the high-level feature sequence X output by the temporal feature association modeling unit. (L) ∈R (n×D) The output is the discriminant sequence P∈R corresponding to each time position. (n×1) Where n is the number of time steps and D is the feature dimension. The input dimension of the first fully connected layer is D, and the output dimension is D / 2. For example, when D=60, the output dimension is 30, and the weight matrix W_p1∈R (D×D / 2) Bias vector b_p1∈R (D / 2) The formula for calculating the output feature P1 is: The ReLU activation function is used to introduce nonlinearity and enhance the model's ability to fit boundary features.

[0086] The second fully connected network has an input dimension of D / 2 and an output dimension of 1, with a weight matrix of... ∈R (D / 2×1) Bias vector ∈R (1) The formula for calculating the output discriminant P is: The Sigmoid activation function maps the range of the discriminant P to [0,1], giving the discriminant a probabilistic meaning. The closer the value of P is to 1, the greater the probability that the time position is in the transition interval between two different action stages; the closer the value of P is to 0, the greater the probability that the time position is within a certain action stage.

[0087] The discriminant P(i) (i=0,1,…,n-1) at each time point reflects the probability of whether that time point is the boundary of an action phase. Its physical meaning lies in characterizing the degree of significant change in the human body's motion state. The discriminant P changes over time in a continuous curve, and the trend of this curve is closely related to the phase transition process of pole climbing: within a certain action phase, the human body's motion state is relatively stable, the amplitude of posture changes is small, and the feature patterns in the high-level feature sequence are relatively consistent. At this time, the value of the discriminant P usually remains at a low level, such as 0.1-0.3. When approaching the phase boundary, the human body's motion state begins to change, the amplitude of posture changes gradually increases, the feature patterns in the high-level feature sequence show significant changes, and the value of the discriminant P rises accordingly. At the phase boundary, the human body's motion state changes significantly, the feature patterns in the high-level feature sequence undergo abrupt changes, and the value of the discriminant P reaches a peak, such as 0.7-1.0. After crossing the phase boundary, the human body enters a new action phase, the motion state tends to stabilize again, the feature patterns in the high-level feature sequence tend to become consistent again, and the value of the discriminant P gradually decreases to a low level.

[0088] For example, during the transition from the pole-raising phase to the ascent phase in pole climbing operations, the operator's posture gradually changes from a ground-ready posture to a climbing posture. The upper limbs change from a drooping state to a gripping state on the pole, and the lower limbs change from a standing state to a force-pushing state. The rate of posture change and joint movement patterns change significantly. At this time, the value of the discriminant P will rise rapidly from a low level in the pole-raising phase to a peak value, and then remain at a low level in the ascent phase. When the ascent phase transitions to the positioning phase, the operator's climbing speed gradually slows down, and the posture changes from a dynamic climbing state to a relatively fixed positioning state. The value of the discriminant P rises to a peak value again, and so on.

[0089] The generation of the discriminant sequence is based on the attitude temporal correlation information contained in the high-level feature sequence, and is achieved through nonlinear mapping and probability transformation of a fully connected neural network.

[0090] Specifically, firstly, the high-level feature sequence X (L) It contains global temporal correlation information and the evolution law of human motion state. The fully connected neural network performs dimensionality reduction and nonlinear transformation on high-level features through the first layer network to extract key features related to the stage boundary, such as posture change rate, joint motion pattern transformation, posture stability and other features.

[0091] Secondly, a second-layer network maps key boundary features to a single probability value, i.e., a discriminant, achieving the transformation from high-dimensional features to boundary probabilities. Finally, a sigmoid activation function constrains the transformation result to the [0,1] interval, giving the discriminant a clear probabilistic meaning, facilitating subsequent boundary point extraction. This generation logic ensures that the discriminant sequence accurately reflects changes in human motion states, providing a reliable quantitative basis for the adaptive division of stage boundaries and solving the problem of inaccurate boundary discrimination caused by existing technologies relying on manual rules or fixed thresholds.

[0092] In some embodiments, for step S6, based on the changing characteristics of the discriminant sequence, the boundary points of multiple action stages are adaptively determined to complete the division of the action stages of the work process. This process does not require predefining the number, order, or duration of stages, and can adapt to the individual differences of different workers, thus solving the problems of strong subjectivity and poor adaptability in the stage division of the prior art.

[0093] Please see Figure 3 , Figure 3 This is a schematic diagram of the action phase division process provided in an embodiment of this disclosure. In step S301, the discriminant sequence undergoes smoothing filtering.

[0094] Specifically, to reduce noise interference and improve the accuracy of stage boundary identification, the change curve corresponding to the discriminant sequence P is first smoothed. For example, a moving average filtering algorithm is used, with a sliding window size of k (e.g., k = 5 time steps), to smooth the discriminant sequence P, resulting in a smoothed discriminant curve. The calculation formula is: In this process, when i - ⌊k / 2⌋ < 0, t = 0; when i + ⌊k / 2⌋ > n-1, t = n-1, to ensure the effectiveness of the sliding window at the sequence boundaries. Moving average filtering effectively filters high-frequency noise in the discriminant curve, which may be caused by minor posture fluctuations of the operator or model prediction errors. The smoothed discriminant curve more clearly presents the peak features corresponding to the stage boundaries, laying the foundation for subsequent candidate point extraction.

[0095] Alternatively, Gaussian filtering can be used for smoothing. The kernel size of Gaussian filtering can be selected as 3×1 or 5×1, and the standard deviation σ can be selected as 0.8-1.2. Its smoothing effect is more gentle and can better preserve the shape of the peak characteristics. The specific selection can be determined according to the noise situation of the discriminant curve.

[0096] In step S302, candidate points for stage boundaries are extracted. Candidate points for stage boundaries are the time positions in the discriminant curve that meet preset peak conditions. The extraction steps may include: first, setting a peak threshold T, where T ranges from 0.5 to 0.7 (e.g., T=0.6). This threshold can be adjusted based on the actual training data and application scenario to ensure effective differentiation between stage boundary peaks and noise peaks. Setting the threshold too high may lead to missed detection of true boundaries, while setting it too low may lead to false detections. Therefore, in practical applications, the optimal threshold can be determined by optimizing the threshold using validation set data.

[0097] Secondly, iterate through the smoothed discriminant sequence. For each time position i (1≤i≤n-2), determine whether the following three conditions are met: (1) >T;(2 > (3) > If all three conditions above are met simultaneously, then time position i is determined as a candidate point for the stage boundary. This condition ensures that the candidate point is a local peak in the discriminant curve, and the peak height exceeds a preset threshold, conforming to the discriminant characteristics corresponding to the stage boundary. For the start position i=0 of the sequence, only... >T and > If the starting position has no left adjacent point, its peak characteristic is significantly higher than that of the right adjacent point; for the ending position i=n-1 of the sequence, only the following is considered: >T and > If a point is found to be a candidate point, then it can be identified. Similarly, if there is no adjacent point to the right at the end position, its peak value is significantly higher than that of the adjacent point to the left.

[0098] In S303, candidate point selection and optimization for stage boundaries are performed. Since each action stage of pole climbing has a certain time span, the time interval between adjacent stage boundaries should not be too small, otherwise it will not conform to the actual operation logic. Therefore, it is necessary to select the extracted candidate points for stage boundaries, remove overly dense candidate points, and ensure the rationality of stage division.

[0099] Specifically, a minimum stage time interval threshold Δt is set. The value of Δt is determined based on the actual situation of the pole climbing operation. For example, it can be set to 2 seconds, corresponding to 60 time steps when the video frame rate is 30fps. The setting of this threshold can refer to the minimum reasonable duration of each stage of the pole climbing operation. For example, the minimum duration of the pole raising stage is usually 3 seconds, and the minimum duration of the ascent stage is usually 5 seconds. Therefore, the setting of Δt should be less than the minimum reasonable duration of each stage to avoid filtering out the true continuous boundaries.

[0100] Iterate through all candidate boundary points of each stage, and arrange them in chronological order as C1, C2, ..., C m (m is the number of candidate points), for any two adjacent candidate points C j and C j+1 (j=1,2,…,m-1), calculate the time interval Δt between the two. j =C j+1 -C j If Δt j If the value is less than Δt, then compare the discriminant values ​​corresponding to the two candidate points. and Candidate points with larger discriminant values ​​are retained, while those with smaller discriminant values ​​are deleted, because candidate points with larger discriminant values ​​are more likely to be the true stage boundaries; if Δt j If the distance is greater than or equal to Δt, both candidate points are retained because their corresponding stage intervals align with the actual operational logic and may represent continuous transitions between different stages. This screening process removes false boundary candidate points caused by local attitude fluctuations, ensuring that the retained candidate points are the true stage boundary locations. For example, during the ascent phase, operators may make brief attitude adjustments, causing a small peak in the discriminant curve. If the interval between this peak and the adjacent true boundary is less than Δt, it will be filtered out and deleted.

[0101] In S304, the final division of the action phases. Based on the selected phase boundary candidate points and combined with the action logic of the pole climbing operation, the entire temporal feature sequence is divided into phases. The specific steps are as follows: First, the selected phase boundary candidate points are arranged in chronological order as B0, B1, ..., B k (k is the final number of boundary points), where B0 is the first stage boundary, B k This marks the boundary of the final stage.

[0102] Secondly, determine the start and end times of each action phase. The start time of the first action phase is t0, and the end time is B0; the start time of the j-th action phase (2≤j≤k) is B0. j-1 The end time position is B. j The starting time position of the last action phase is B. k The end time position is t n-1 Finally, based on the posture characteristics of each stage and in accordance with the standard requirements for pole climbing operations, the divided stages were named, resulting in five typical stages for pole climbing operations: pole raising stage, ascent stage, positioning stage, stabilization stage, and operation stage.

[0103] For example, the features of each stage and the corresponding pose patterns are as follows:

[0104] Pole-raising phase: From the moment the operator begins to prepare to climb the pole to the moment they first grasp the tower and begin to exert force, the posture characteristics of this phase are frequent posture changes and low stability. The discriminant curve rises from a low level to the first peak.

[0105] Ascending phase: From the end of pole raising to the operator reaching the predetermined working height, the posture characteristics of this phase are periodic upper and lower limb movements, with a large range of posture changes and regularity, and the discriminant curve remains at a low level.

[0106] Positioning phase: from reaching the predetermined height to adjusting to the posture required for operation. The posture characteristics of this phase are: the climbing speed slows down, the posture is adjusted frequently, and the discriminant curve rises to the second peak.

[0107] Stable phase: From the end of positioning to the start of pole operation, the attitude characteristics of this phase are that the amplitude of attitude change is significantly reduced, the motion state is relatively stable, and the discriminant curve remains at a low level.

[0108] Operation phase: From the start of pole work to the completion of work, the postural characteristics of this phase are frequent upper limb movements, stable lower limb support, and the discriminant curve may fluctuate slightly, but overall it remains at a low level.

[0109] It is important to emphasize that the number and duration of each stage are not predetermined, but rather adaptively generated based entirely on the changing patterns of posture and temporal characteristics. This allows the method to adapt to differences in the movement rhythms of different workers and variations in stage duration under different working environments. For example, experienced workers may have a shorter ascent stage and fewer adjustment movements during the positioning stage; while novice workers may have a longer ascent stage and more frequent adjustment movements during the positioning stage. This method can accurately identify and classify each stage in both cases, demonstrating its strong adaptability and robustness.

[0110] In some embodiments, to ensure that the temporal feature association modeling unit and the stage boundary discrimination output unit can accurately achieve pose feature extraction, temporal association modeling and stage boundary discrimination, the entire model needs to be trained and optimized.

[0111] Specifically, the training dataset consists of a large number of pole-climbing operation video samples with labeled stage boundary positions. Each sample includes a video frame sequence, a corresponding posture feature sequence, and stage boundary annotation information. Data collection: Videos of pole-climbing operations by different personnel (including different heights, body types, and skill levels) under different working environments (including different lighting conditions and background complexity) were collected. The number of videos was no less than 1000, and the length of each video was 3-10 minutes, ensuring the diversity and representativeness of the dataset and covering various situations that may occur in actual operations. Posture feature extraction: Posture feature sequences were extracted from each video according to the steps of "Video Data Acquisition and Preprocessing" and "Posture Feature Vector Construction" described above, forming the input features of the training data.

[0112] Stage Boundary Labeling: Professionals with experience in pole climbing operation training and assessment manually label the stage boundaries of each video segment. The labeled stage boundaries include the pole-raising-ascent boundary, ascent-positioning boundary, positioning-stabilization boundary, and stabilization-operation boundary. Each boundary label corresponds to a specific time position in the video frame sequence, serving as training label data. During the labeling process, the labelers must refer to the pole climbing operation specifications and combine this with the movement status of the workers in the video to accurately determine the stage boundary positions, ensuring the accuracy of the label data.

[0113] Dataset partitioning: The constructed dataset was divided into training, validation, and test sets in a 7:2:1 ratio. The training set was used for model parameter training, the validation set for hyperparameter tuning and model performance validation during training, and the test set for evaluating the final model performance. Stratified sampling was used during the partitioning process to ensure a consistent sample proportion across different workers and working environments in each dataset, avoiding model bias caused by imbalanced data distribution.

[0114] The model training uses stage boundary discrimination loss, which is used to optimize the discrimination accuracy of the stage boundary discrimination output unit for stage boundaries.

[0115] Stage boundary discrimination loss: Binary cross-entropy (BCE) loss is used. Assume the discriminant sequence output by the model is... The real-stage boundary label sequence is Where the label corresponding to the stage boundary position is 1, and the label corresponding to the non-boundary position is 0, the formula for calculating the stage boundary discrimination loss is:

[0116]

[0117] This loss function can effectively measure the model's classification accuracy between boundaries and non-boundaries. By minimizing the loss value, the discriminant sequence output by the model is made as close as possible to the true label sequence.

[0118] The model training employs the Adaptive Moment Estimation (Adam) optimizer, which optimizes the model parameters by minimizing the stage boundary discrimination loss L. For example, the specific training parameter settings are as follows:

[0119] Learning rate: The initial learning rate is set to 1e-4, and a cosine annealing learning rate scheduling strategy is adopted. The learning rate decays once every 10 epochs of training, and the decay coefficient is 0.9; Batch size: set to 32.

[0120] Training epochs: Set to 100 epochs. Training stops when the loss value on the validation set no longer decreases for 10 consecutive epochs. Regularization parameter: L2 regularization is introduced in the fully connected layer and the Transformer encoder layer.

[0121] The training process is as follows: Initialize model parameters; input training set samples into the model according to batch size, and obtain the discriminant sequence through forward propagation; calculate the loss value according to the loss function; calculate the gradient using the backpropagation algorithm and update the parameters using the Adam optimizer; evaluate the model performance using the validation set after each training epoch; evaluate the final stage boundary recognition accuracy of the model using the test set after training is completed.

[0122] Once the model is trained, it can be used for phase division in actual pole climbing operations. The reasoning process is as follows:

[0123] The video of the pole climbing operation to be processed is subjected to frame sequence extraction and preprocessing according to the above "Video Data Acquisition and Preprocessing" steps;

[0124] Human key point information is extracted from each frame of image to construct pose feature vector and pose temporal feature sequence;

[0125] The pose temporal feature sequence is input into the trained model, and the discriminant sequence is obtained through time-position encoding, temporal feature correlation modeling, and stage boundary discrimination. ;

[0126] The discriminant sequence is smoothed, candidate points are extracted, and optimized to determine the final stage boundary position.

[0127] The action phases are divided according to the phase boundary positions, and the start and end times, duration and name of each phase are output.

[0128] In some embodiments, the method further includes operational standardization judgment. Specifically, based on each divided action stage, statistical features or variation features of the posture feature vector within that stage are extracted for subsequent standardization judgment. Statistical features include the mean, standard deviation, maximum value, and minimum value of each dimension of posture features within that stage, such as the mean of the elbow joint angle during the pole-raising stage and the standard deviation of the posture change rate during the ascent stage; variation features include the trend, amplitude, and frequency of change of posture features within that stage, such as the amplitude of the shoulder joint angle change during the positioning stage and the frequency of upper limb movements during the operation stage.

[0129] During feature extraction, for each stage, the attitude feature vectors at all time points within that stage are traversed, and the statistical and change indices of each dimension of the features are calculated to form the feature vector for that stage. For example, the feature vector for a certain stage can be represented as F_stage=[μ1,σ1,max1,min1,μ2,σ2,max2,min2,…,μ_D,σ_D,max_D,min_D,Δ1,Δ2,…,Δ_D], where μ_d, σ_d, max_d, and min_d are the mean, standard deviation, maximum, and minimum values ​​of the d-th dimension feature, respectively, and Δ_d is the change magnitude of the d-th dimension feature. This feature vector comprehensively characterizes the distribution and change patterns of the attitude features in that stage.

[0130] The extracted stage features are compared with preset normative indicators for the corresponding action stages to determine whether the operator's operation is standardized during that stage. The normative indicators are based on pole climbing operation specifications and safety operation standards. For each action stage, allowable ranges for each dimension of features are established, for example:

[0131] Standard indicators for the pole lifting phase: elbow joint angle range of 90°-120°, pole lifting duration of 3-8 seconds, and shoulder joint and neck joint angle range of 30°-60°.

[0132] Normative indicators for the ascent phase: hip joint angle range of 60°-100°, knee joint angle range of 90°-135°, and average rate of change of posture of 0.5-1.0 (after normalization).

[0133] Standard indicators for the positioning phase: positioning duration is 5-12 seconds, and the maximum change in the angle of key joints does not exceed 30°;

[0134] Standard indicators for the stable phase: the standard deviation of the attitude change rate does not exceed 0.2 (after normalization), and the fluctuation range of the angles of each key joint does not exceed 10°;

[0135] Standardized indicators for the operational phase: Lower limb joint angles remain stable, with fluctuations not exceeding 5°; upper limb movement frequency is 0.3-0.8 times / second. During the comparison process, the degree of deviation between the phase characteristics and the standardized indicators is calculated. If the deviation of all dimensions is within the allowable range (e.g., deviation ≤ 10%), the operation is deemed standardized for that phase. If the deviation of any dimension exceeds the allowable range, the operation is deemed non-standard for that phase, and the specific non-standard item and deviation value are recorded. For example, "The average elbow joint angle during the pole-lifting phase is 135°, exceeding the standardized range of 90°-120°, with a deviation of 12.5%, indicating non-standard operation."

[0136] The above-mentioned standardized judgments can provide objective and quantitative basis for pole climbing operation training and assessment, help operators identify non-standard practices in operation, and provide clear evaluation criteria for assessors to ensure the objectivity and consistency of assessment results.

[0137] In summary, this method effectively solves the problems of strong subjectivity, poor adaptability, and weak generalization ability in the existing technology in pole climbing operation stage division and standardization judgment by integrating attitude and temporal feature modeling, stage boundary discrimination, and stage division. It provides reliable technical support for the training, evaluation, and operation monitoring of pole climbing operations in power systems and has important practical application value.

[0138] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a human body recognition system based on an interactive spatiotemporal Transformer, provided in an embodiment of this application. Figure 4 As shown, system 400 includes:

[0139] The preprocessing module 401 is used to acquire continuous video data of the workers, extract and preprocess the video data to obtain an ordered image frame sequence;

[0140] The pose feature vector construction module 402 is used to extract the position information of human key points in each frame of image, and construct the pose feature vector corresponding to the frame of image based on the position information of human key points.

[0141] The temporal coding sequence generation module 403 is used to construct an attitude temporal feature sequence from the attitude feature vectors arranged in chronological order, and to perform time position encoding on the attitude temporal feature sequence to generate a temporal coding sequence that integrates time information.

[0142] The advanced feature sequence generation module 404 is used to input the temporal encoded sequence into the temporal feature association modeling unit, and output an advanced feature sequence containing global temporal association relationships after processing, wherein the temporal feature association modeling unit is based on a multi-layer Transformer structure with a temporal attention mechanism;

[0143] The discriminant sequence generation module 405 is used to input the high-level feature sequence to the stage boundary discrimination output unit. The stage boundary discrimination output unit generates a discriminant sequence based on the high-level feature sequence. Each discriminant in the discriminant sequence represents the probability of whether the corresponding time position is the boundary of the action stage.

[0144] The stage division module 406 is used to adaptively determine the boundary points of multiple action stages based on the change characteristics of the discriminant sequence, thereby realizing the division of action stages of the operation process recorded by continuous video data.

[0145] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0146] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0147] Please see Figure 5 It shows a schematic diagram of the structure of an electronic device according to an embodiment of this application, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 5 As shown, the electronic device 500 may include:

[0148] The system includes at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502. The communication bus 502 is used to enable connection and communication between the components. The user interface 503 may include buttons, and optionally include a standard wired or wireless interface. The network interface 504 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0149] The processor 501 may include one or more processing cores and connect to various parts within the device 500 via various interfaces and lines. It implements the various functions and data processing of the device 500 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 505, and by accessing data in the memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 501 may also integrate one or more combinations of CPU, GPU, and modem. The CPU is mainly used to handle the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem is used for wireless communication. It is understood that the modem may not be integrated into the processor 501, but may be implemented through a separate chip.

[0150] Memory 505 may include random access memory (RAM) or read-only memory (ROM). Optionally, memory 505 includes a non-transitory computer-readable medium for storing instructions, programs, code, code sets, or instruction sets. Memory 505 may be divided into a program storage area and a data storage area, wherein the program storage area may be used to store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, audio playback functionality, image playback functionality, etc.), and instructions for implementing the aforementioned method embodiments; the data storage area may be used to store data involved in the relevant method embodiments. Memory 505 may also be at least one storage device located remotely from processor 501. Figure 5 As shown, the memory 505, which serves as a computer storage medium, may contain an operating system, a network communication module, a user interface module, and program instructions.

[0151] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by processor 501, it performs the functions defined in the methods of this application.

[0152] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0153] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0154] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

Claims

1. A human body recognition method based on interactive spatiotemporal Transformer, characterized in that, include: Continuous video data of the workers is acquired, and the video data is subjected to frame sequence extraction and preprocessing to obtain an ordered image frame sequence; For each frame of image, extract the position information of human key points, and construct the pose feature vector corresponding to the frame of image based on the position information of human key points. The attitude feature vectors arranged in chronological order are used to form an attitude temporal feature sequence, and the attitude temporal feature sequence is encoded with time position to generate a temporal encoded sequence that integrates time information. The temporal encoded sequence is input into the temporal feature association modeling unit, and after processing, a high-level feature sequence containing global temporal associations is output. The temporal feature association modeling unit is based on a multi-layer Transformer structure with a temporal attention mechanism. The high-level feature sequence is input into the stage boundary discrimination output unit, which generates a discrimination quantity sequence based on the high-level feature sequence. Each discrimination quantity in the discrimination quantity sequence represents the probability of whether the corresponding time position is the boundary of the action stage. Based on the changing characteristics of the discriminant sequence, the boundary points of multiple action stages are adaptively determined, thereby realizing the division of action stages in the operation process recorded by continuous video data.

2. The human body recognition method based on interactive spatiotemporal Transformer according to claim 1, characterized in that, The pose feature vector corresponding to this frame image is constructed by: Based on the location information of the key points of the human body, multi-dimensional features representing the human posture are calculated. The multi-dimensional features include at least the coordinates of the key points, the relative angles between key joints, or the distances between key joints. The multidimensional features are normalized based on human scale to obtain the pose feature vector.

3. The human body recognition method based on interactive spatiotemporal Transformer according to claim 1, characterized in that: The temporal feature association modeling unit includes an L-layer Transformer encoder layer. Each encoder layer includes a multi-head self-attention mechanism module, a feedforward neural network module, and a layer normalization and residual connection module. The multi-head self-attention mechanism module is used to calculate the association weights between features at different time positions in the temporal encoded sequence, where L is an integer greater than 1.

4. The human body recognition method based on interactive spatiotemporal Transformer according to claim 3, characterized in that, The stage boundary discrimination output unit is a structure containing at least one fully connected neural network, and the discrimination sequence is obtained by performing nonlinear mapping and probability transformation on the high-level feature sequence.

5. The human body recognition method based on interactive spatiotemporal Transformer according to claim 1, characterized in that, Based on the changing characteristics of the discriminant sequence, the boundary points of multiple action stages are adaptively determined, including: The discriminant sequence is smoothed by filtering to obtain a smoothed discriminant curve; Extract the local extreme points that satisfy the peak condition in the smoothed discriminant curve and use them as candidate points for the stage boundary; Based on a preset minimum stage time interval threshold, the candidate points for the stage boundary are filtered, and candidate points with too close intervals are removed to obtain the final stage boundary points.

6. The human body recognition method based on interactive spatiotemporal Transformer according to claim 1, characterized in that, After completing the action phase division, it also includes: Based on each action stage after division, extract the statistical features or change features of the posture feature vector within that stage; The statistical characteristics or change characteristics are compared with the preset normative indicators for the corresponding action stage to determine whether the operator's operation in that action stage is standardized.

7. A human body recognition system based on an interactive spatiotemporal Transformer, characterized in that, include: The preprocessing module is used to acquire continuous video data of the workers, extract and preprocess the video data to obtain an ordered image frame sequence; The pose feature vector construction module is used to extract the position information of human key points in each frame of image, and construct the pose feature vector corresponding to the frame of image based on the position information of human key points. The temporal coding sequence generation module is used to construct an attitude temporal feature sequence from the attitude feature vectors arranged in chronological order, and to perform time position encoding on the attitude temporal feature sequence to generate a temporal coding sequence that integrates time information. The advanced feature sequence generation module is used to input the temporal encoded sequence into the temporal feature association modeling unit, and output an advanced feature sequence containing global temporal association relationships after processing. The temporal feature association modeling unit is based on a multi-layer Transformer structure with a temporal attention mechanism. The discriminant sequence generation module is used to input the high-level feature sequence to the stage boundary discrimination output unit. The stage boundary discrimination output unit generates a discriminant sequence based on the high-level feature sequence. Each discriminant in the discriminant sequence represents the probability of whether the corresponding time position is the boundary of the action stage. The stage division module is used to adaptively determine the boundary points of multiple action stages based on the change characteristics of the discriminant sequence, thereby realizing the division of action stages of the operation process recorded by continuous video data.

8. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions can be executed by a processor to implement the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images