Video action recognition method and system based on skeleton key points

Abstract

The invention discloses a video action recognition method and system based on skeleton key points. Processing a to-be-identified video, generating a video frame sequence arranged according to a time sequence, and extracting human skeleton key point coordinates of each frame to construct a key point sequence; performing continuous time memory coding on the key point sequence to generate a corresponding memory state sequence; determining an action boundary based on the memory state change at adjacent moments, and dividing the key point sequence into a plurality of action stages; determining a joint weight according to the memory state of each stage, and generating a weighted joint trajectory; performing matching calculation on the weighted track and the corresponding template track in each stage range to obtain a stage matching result; and performing continuity judgment in combination with the matching result, the memory state and the stage sequence relationship, and outputting an action category corresponding to the video. The action segmentation processing and time sequence information fusion are realized, and the accuracy and stability of action recognition are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent video technology, and in particular to a video action recognition method and system based on skeletal key points. Background Technology

[0002] With the development of applications such as intelligent security, human-computer interaction and sports analysis, video-based human motion recognition has become an important research direction in the field of computer vision. Currently, most mainstream methods adopt time-series modeling based on image features or skeletal key points, and use deep learning models to model the video sequence as a whole to determine the action category.

[0003] Existing video action recognition technologies still have significant limitations in complex action scenes, multi-stage actions, and uneven temporal rhythms. On the one hand, traditional methods typically use fixed time windows or overall sequence modeling, lacking precise characterization of action boundaries and failing to effectively distinguish stage changes in continuous actions. This can easily lead to interference between different action segments, thus reducing recognition accuracy. On the other hand, existing methods rely heavily on short-term memory or discrete sampling methods in temporal feature representation, which are insufficiently adaptable to long-term dependencies and non-uniform time intervals, making it difficult to stably represent the dynamic changes of complex actions. Furthermore, existing skeletal keypoint-based recognition methods typically assign fixed or simple weights to each joint, failing to fully reflect the differences in the importance of key joints in different action stages, resulting in insufficiently refined feature representation. At the same time, conventional trajectory matching methods lack stage and structural constraints, making it difficult to achieve high-precision matching under action segmentation conditions, easily leading to matching biases, and affecting the stability and reliability of the final classification results.

[0004] Therefore, how to provide a video motion recognition method and system based on skeletal key points is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a video action recognition method and system based on skeletal key points. By constructing temporal data of skeletal key points and introducing a continuous time memory coding mechanism, it realizes automatic division of action boundaries and multi-stage action modeling. At the same time, by combining adaptive allocation of joint weights and segmented constraint trajectory matching calculation, it completes the fine recognition of complex actions and has the advantages of strong temporal modeling capability, accurate action segmentation and high stability of recognition results.

[0006] A video motion recognition method and system based on skeletal key points according to an embodiment of the present invention includes the following steps: The video acquisition module acquires the video to be identified and generates a sequence of video frames arranged in chronological order. The key point sequence construction module performs pose analysis on the video frame sequence, extracts the coordinates of key points of the human skeleton, and generates a skeleton key point sequence according to the time order of the video frame sequence. The memory state generation module inputs the skeletal key point sequence into the Legendre memory unit, performs continuous time memory encoding, and generates a memory state sequence corresponding to each moment of the skeletal key point sequence. The action boundary determination module determines action boundary points based on the state changes of adjacent moments in the memory state sequence, and divides the skeletal key point sequence according to the action boundary points to generate action stage sub-sequences arranged in chronological order. The weighted trajectory generation module determines the joint weights of skeletal key points based on the memory state sequence corresponding to each action stage subsequence, and generates a weighted joint trajectory subsequence by combining the coordinates of skeletal key points in the action stage subsequence. The matching and recognition module defines the matching interval of each weighted joint trajectory subsequence based on the action boundary points, and uses the Fraser distance action recognition method within the matching interval to perform matching calculations between each weighted joint trajectory subsequence and the corresponding stage template trajectory to generate a stage matching distance value. The category determination module performs cross-stage continuity determination based on the stage matching distance value, the corresponding memory state sequence, and the temporal order of action boundary points, and outputs the action category corresponding to the video to be identified.

[0007] Optionally, modules can be integrated using the following methods: Acquire the video to be identified and generate a sequence of video frames arranged in chronological order; Human pose analysis is performed on the video frame sequence, the coordinates of human skeletal key points corresponding to each video frame are extracted, and a skeletal key point sequence is constructed according to the time order of the video frame sequence. The sequence of skeletal key points is input into the Legendre memory unit and continuously encoded in time memory to generate a memory state sequence that corresponds one-to-one with each moment of the skeletal key point sequence. The action boundary points are determined based on the state changes of adjacent moments in the memory state sequence, and the skeletal key point sequence is divided according to the action boundary points to generate action stage subsequences arranged in chronological order. The joint weights of the skeletal key points are determined based on the memory state sequence corresponding to each action stage subsequence, and a weighted joint trajectory subsequence is generated by combining the coordinates of the skeletal key points in each action stage subsequence. The matching interval of each weighted joint trajectory subsequence is defined based on the action boundary points. Within the matching interval, the weighted joint segmentation constraint discrete Frescher distance is used to match and calculate the matching distance between each weighted joint trajectory subsequence and the corresponding stage template trajectory, generating the stage matching distance value corresponding to each action stage subsequence. Based on the stage matching distance value corresponding to each action stage subsequence, the memory state sequence corresponding to each action stage subsequence, and the temporal order relationship of the action boundary points, cross-stage continuity determination is performed, and the action category corresponding to the video to be identified is output.

[0008] Optionally, constructing the skeletal keypoint sequence includes: Receive a sequence of video frames arranged in chronological order, and read the first frame one by one. The frame image is scaled proportionally and filled with boundaries according to the preset input size. The scaling ratio and filling offset parameters are recorded to generate the pose analysis input frame. Human target detection is performed on the pose parsing input frame. Human detection boxes are extracted, and single target regions are selected according to the detection confidence. The original image is then cropped based on the detection boxes to generate corresponding human image patches, and the frame index is labeled. ; Input human image blocks into the keypoint detection network, output a preset number of two-dimensional coordinates of human skeleton keypoints and their confidence scores, and establish a frame-level keypoint coordinate set according to the keypoint topology. For keypoints with a confidence level below a preset threshold, structural constraint completion is performed: when there are adjacent keypoints in the same frame that have a skeleton connection with the keypoint, the coordinates of the keypoint are generated according to a preset skeleton length ratio. When the same frame cannot be completed, retrieve the coordinates of the same key point in the previous and next frames, and use linear interpolation to generate the coordinates of the key point. The coordinates of frame-level key points are inversely mapped from the human image block coordinate system to the original video frame coordinate system. During the inverse mapping process, the real spatial position is restored based on the recorded scaling ratio and fill offset parameters. For key points with the same name in adjacent video frames, the coordinate difference is calculated and compared with a preset jump threshold. When the coordinate difference exceeds the jump threshold, the key point is marked as an abnormal point and replaced with the coordinates of the key point with the same name in the previous frame, generating a key point coordinate set with consistent time sequence. The keypoint coordinate sets of each frame are arranged sequentially according to the time sequence of the video frame sequence to construct a skeletal keypoint sequence, where the first... Frame number The coordinates of the key points are represented as follows: The sequence of skeletal key points is represented as The sequence of key skeletal points is then output.

[0009] Optionally, generating a sequence of memory states includes: Receive skeletal keypoint sequence , will the The coordinates of all key points in the frame are concatenated into an input vector according to a fixed key point index order. and the first The frame's rendering timestamp is written to the timestamp queue to form a sequence with... Bound timing input records; The order of Legendre's memory cells is determined to be... In the interval Fixed selection Each sampling point will be used to transfer from order 0 to... The values ​​of the Legendre polynomials at each sampling point are written into the basis function table in order, and the values ​​of the first derivatives of each basis function at each sampling point are written into the derivative table in order. The basis function table and the derivative table serve as fixed lookup data in the construction process. A continuous-time memory kernel is constructed based on the basis function table and derivative table. The construction process includes: for each pair of order indices Using pre-defined Gaussian-Legend quadrature nodes and weights, for the "first" The first-order basis function and the second-order basis function "The product of the derivatives of the basis functions of order" The integral over is evaluated and written into the state coupling coefficient matrix, for the "first" The product of the first-order basis function and the input basis function is in The integral over the time step is evaluated and written into the input projection coefficient matrix; Write the state coupling coefficient matrix into the continuous-time state transition matrix. Write the input projection coefficient matrix into the continuous-time input matrix. And combine the product nodes, weights, basis function table, derivative table and Store together in a fixed state; Will Discretization is performed into frame-by-frame callable discrete operators. The discretization process includes: reading the nominal frame interval value corresponding to the video frame sequence, and discretizing the continuous-time state transition matrix according to the frame interval to obtain a discrete state transition matrix. The continuous-time input matrix is ​​then equivalently discretized to obtain the discrete input matrix. ,Will Write the parameters of the Legendre memory cell as a fixed operator; In the construction process of Legendre memory units, the input injection structure is constrained based on the skeletal topology. The constraint implementation steps include: reading the keypoint topological edge set and generating an adjacency table, where each joint in the adjacency table contains only its own index and the indices of the joints connected to it in the topological edge set; and inputting the input vector... The joint is divided into joint channel blocks according to the joint index, and a connection is generated based on the adjacency table. Isomorphic mask matrix The setting rule for the mask matrix is: "set to 1 when the corresponding joint channel block in the state dimension satisfies the adjacency relationship with the corresponding joint channel block in the input dimension, otherwise set to 0"; apply the mask matrix to the discrete input matrix to obtain the topological constraint input matrix and write it into the parameter area for replacement. : ; in, Represents element-wise product; In the construction process of Legendre memory cells, a multi-step advancement operator table is established to address the non-uniform time advance caused by frame drops and compensation frames. The steps for constructing the operator table include: setting an upper bound on the maximum allowed number of advancement steps. ,by Pre-compute and store the matrix exponentiation sequence for the basis matrix. ,in Take the satisfaction The smallest integer; during the runtime phase, the time stamp difference between two adjacent frames is quantized to obtain the number of propulsion steps. ,Will The multi-step advancement matrix is ​​obtained by expanding the binary representation and multiplying the expanded results together. ; in, To advance the number of steps The binary bits; the multi-step advancement matrix is ​​used as the current frame state advancement operator to participate in the memory state calculation; Initialize the memory state vector And write to the state cache, traversing in ascending order of frame index. It is bound to a timestamp, and state progression and input injection are performed frame by frame: first based on the number of progression steps. Generate multi-step advancement matrix and perform Execution status advancement, then Input matrix with topological constraints The mapped result is injected into the advanced state to obtain ,Will With frame index Bind to write memory cache; Index by Frame Sequential output of memory buffer forms a memory state sequence ,in The first of the skeletal key point sequence Each frame corresponds to a specific state, and the memory state sequence is output to the subsequent action boundary determination step.

[0010] Optionally, generating action phase subsequences includes: Receive memory state sequence With skeletal keypoint sequence The memory state vector at each time step The Legendre order index is used to divide the components into lower-order and higher-order components. The lower-order components are determined by a predefined order. The first dimension is composed of the second dimension, and the higher-order components are composed of the remaining dimensions. The higher-order component energy and lower-order component energy are calculated time-by-time, and the increase in higher-order energy between adjacent time-times is calculated to form a boundary score sequence. The boundary score is calculated at the t-th time step. The time is recorded as: ; in, Indicates the first Time-of-flight higher-order components, Indicates the first Low-order components at any given time It is a fixed positive constant; Perform candidate boundary detection on the scoring sequence: with a fixed window length. Performing a local maximum search will satisfy the condition that "the current time step is the maximum value within the window and is continuous for at least 1000 seconds". The moment when the frame remains positive is recorded as the candidate boundary moment; Apply rigid interval constraints to the candidate boundaries: set the minimum boundary interval to... Frame, scan the candidate boundary list in chronological order, for any two candidate boundaries with an interval of less than In the case of frames, only candidate boundaries with larger score values ​​are retained; Perform a skeleton consistency check on each retained candidate boundary: take a fixed length of frame segment before and after the candidate boundary, calculate and compare the inter-frame displacement cumulative value of each joint in the two segments, and delete the candidate boundary whose displacement cumulative value does not meet the preset minimum change amount, thus obtaining the action boundary point sequence. Add the first frame index and the last frame index to the motion boundary point sequence, and then process the skeletal keypoint sequence according to the temporal order of the boundary points. By dividing the interval, multiple action stage subsequences are obtained, arranged in chronological order.

[0011] Optionally, generating weighted joint trajectory subsequences includes: Receive the The skeletal keypoint sequence corresponding to each action phase subsequence With memory state subsequence Read the starting frame index of the stage. With Termination Frame Index and the stage length Write the stage parameter table; Memory state vector Split into pre-fixed joint channel mapping tables Each joint channel segment corresponds one-to-one with a skeletal keypoint index; within each stage, the preceding frame set is extracted. With the set of subsequent frames For each joint channel segment, the average value of the L2 norm is calculated frame by frame in the front and back segments and written into the joint mean buffer. Regarding the first For each joint, read the mean of its rear segment and the mean of its front segment, calculate the original joint score, and write it into the joint score vector. Perform topology unification on the joint score vectors; Normalization is performed on the standardized joint score vectors; Within a stage, the coordinates of the skeletal keypoints are traversed frame by frame. The joint weights are multiplied by each coordinate component of the keypoint coordinates to generate the first... Frame number The coordinates of each weighted key point are calculated and written to the weighted frame buffer. From frame index to The weighted frame buffer is written to the stage trajectory buffer in sequence to obtain the first... The weighted joint trajectory subsequences for each action phase are generated; the weighted joint trajectory subsequences are output in chronological order of the action phases.

[0012] Optionally, the stage matching distance values ​​corresponding to each action stage subsequence can be generated as follows: Receive the Weighted joint trajectory subsequences for each action phase No. Phase fixed joint weight vector and the first Stage Template Trajectory Read the phase start frame index given by the action boundary point. With Termination Frame Index Limit the set of trajectory indexes to be matched to And limit the template trajectory index set to ; Establish a frame-to-frame distance lookup table; the frame-to-frame distance table is located in... The distance at point is denoted as: ; in, For the first Joint weights For the number of joints, For the skeleton topological edge set, For the trajectory to be matched Frame number Joint coordinates For template trajectory number Frame number Joint coordinates; In the construction process of discrete Fraser matching, a piecewise constraint for mandatory passage at boundary anchor points is established, specifically as follows: Treat the matching trajectory in the interval The system calculates a frame-by-frame cumulative displacement table. The cumulative displacement is calculated by summing the weighted displacement distances across all joints in two adjacent frames and then accumulating the sums. The total cumulative displacement from the start point to the end point is then divided into fixed equal parts. Divide the data into equal parts. For each division threshold, scan the cumulative displacement table from front to back, and record the frame index that first reaches the threshold, thus obtaining the sequence of anchor point frame indices to be matched. ; Calculate the cumulative displacement table for the template trajectory according to the same rules and generate the template anchor point frame index sequence. Pair anchor points as The anchor point pairs are then written into the anchor point constraint table, which specifies that the matching path must follow the rules. The ascending order passes through each pair of anchor points sequentially. In the construction process of discrete Fraser matching, an index constraint for hard pruning of the banded reachability domain is established, specifically as follows: Calculate the length of the trajectory to be matched With template length Template indexes are determined by integer division and remainders. Mapping to the center index of the match ; For each template index Extend fixed width on both sides of the central index. The frame positions are obtained to obtain the... The corresponding allowed index range to be matched ,in , ; All Write to the reachable domain table, and specify that it is only possible when... Time frame pair For reachable frame pairs, unreachable frame pairs are not included in the recursive calculation; The overall matching interval is divided into sections based on the anchor point constraint table. There are n consecutive subintervals, with the endpoints of the subintervals being, in order: The piecewise constrained discrete Fréchet distance is calculated segment by segment for each sub-interval. The calculation process is as follows: A two-dimensional recursive table is established within each sub-interval. The row index of the recursive table is the index of the frame to be matched allowed in the sub-interval, and the column index is the index of the template frame allowed in the sub-interval. The recursive table is filled in ascending order of rows and columns. Each cell only allows transitions from three adjacent cells: "top", "left", and "top left". The cell cost is taken from the distance table value of the cell frame pair. The larger of the source path cost and the smaller of the costs of the three adjacent source cells; for unreachable frame pairs Mark directly as unreachable and skip padding; initialize the starting cell of each sub-interval with the distance of the corresponding frame pair; After each sub-interval is filled, the cost value of the end cell of the sub-interval is read as the segmented Frescher value of the sub-interval. The maximum value of all segmented Frescher values ​​of the sub-intervals is taken as the first... The stage matching distance value for each stage; The first Stage matching distance value and stage index Bind and write the stage matching result table, and output the set of matching distance values ​​for each stage in chronological order of the action stages.

[0013] The beneficial effects of this invention are: This invention introduces Legendre memory units to encode the skeletal keypoint sequence in a continuous temporal memory manner, which can characterize the long-term dependencies and dynamic evolution characteristics in the action process under a unified temporal framework, and adapt to uneven time intervals, frame drops or compensation frames in video sequences, thereby improving the integrity and stability of action temporal information modeling.

[0014] This invention employs a joint-weighted segmented constraint discrete Fraser distance action recognition method. Within a matching interval defined by the action boundary, it performs matching calculations on the trajectories of each stage. This method can simultaneously take into account the temporal sequence relationship and spatial morphological differences of the action trajectories, and combine joint weights to highlight the changing characteristics of key parts. This reduces stage mismatches and local deviations in the overall matching, thereby improving the accuracy and reliability of complex action recognition. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a video motion recognition method and system based on skeletal key points proposed in this invention; Figure 2 This is a flowchart illustrating the continuous-time memory encoding of the Legendre memory unit proposed in this invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0017] refer to Figure 1 - Figure 2A video action recognition method and system based on skeletal key points includes the following steps: The video acquisition module acquires the video to be identified and generates a sequence of video frames arranged in chronological order. The key point sequence construction module performs pose analysis on the video frame sequence, extracts the coordinates of key points of the human skeleton, and generates a skeleton key point sequence according to the time order of the video frame sequence. The memory state generation module inputs the skeletal key point sequence into the Legendre memory unit, performs continuous time memory encoding, and generates a memory state sequence corresponding to each moment of the skeletal key point sequence. The action boundary determination module determines action boundary points based on the state changes of adjacent moments in the memory state sequence, and divides the skeletal key point sequence according to the action boundary points to generate action stage sub-sequences arranged in chronological order. The weighted trajectory generation module determines the joint weights of skeletal key points based on the memory state sequence corresponding to each action stage subsequence, and generates a weighted joint trajectory subsequence by combining the coordinates of skeletal key points in the action stage subsequence. The matching and recognition module defines the matching interval of each weighted joint trajectory subsequence based on the action boundary points, and uses the joint weighted segmented constraint discrete Fraser distance action recognition method within the matching interval to perform matching calculations between each weighted joint trajectory subsequence and the corresponding stage template trajectory, generating a stage matching distance value; The category determination module performs cross-stage continuity determination based on the stage matching distance value, the corresponding memory state sequence, and the temporal order of action boundary points, and outputs the action category corresponding to the video to be identified.

[0018] In this embodiment, the modules are interconnected using the following method: Acquire the video to be identified and generate a sequence of video frames arranged in chronological order; Human pose analysis is performed on the video frame sequence, the coordinates of human skeletal key points corresponding to each video frame are extracted, and a skeletal key point sequence is constructed according to the time order of the video frame sequence. The sequence of skeletal key points is input into the Legendre memory unit and continuously encoded in time memory to generate a memory state sequence that corresponds one-to-one with each moment of the skeletal key point sequence. The action boundary points are determined based on the state changes of adjacent moments in the memory state sequence, and the skeletal key point sequence is divided according to the action boundary points to generate multiple action stage subsequences arranged in chronological order. The joint weights of the skeletal key points are determined based on the memory state sequence corresponding to each action stage subsequence, and a weighted joint trajectory subsequence is generated by combining the coordinates of the skeletal key points in each action stage subsequence. The matching interval of each weighted joint trajectory subsequence is defined based on the action boundary points. Within the matching interval, the weighted joint segmentation constraint discrete Frescher distance is used to match and calculate the matching distance between each weighted joint trajectory subsequence and the corresponding stage template trajectory, generating the stage matching distance value corresponding to each action stage subsequence. Based on the stage matching distance value corresponding to each action stage subsequence, the memory state sequence corresponding to each action stage subsequence, and the temporal order relationship of the action boundary points, cross-stage continuity determination is performed, and the action category corresponding to the video to be identified is output.

[0019] In this embodiment, generating a video frame sequence includes: Call the decapsulation interface to open the video to be identified, lock a single video track, read the time base, average frame rate and the first renderable timestamp, set the frame interval to the reciprocal of the average frame rate, and set the first renderable timestamp to the start time; The video stream packets are read sequentially and sent to the decoder. Each time the decoder outputs a frame, it reads the frame presentation timestamp and binds it to the image frame, writing it into the buffer. When the presentation timestamp is missing, the presentation timestamp of the previous frame plus the frame interval is used to generate the presentation timestamp of the current frame and written into the binding record. Perform timestamp verification and cleaning in the order of writing, delete duplicate frames whose presentation timestamp is the same as the previous frame, delete back frames whose presentation timestamp is less than the previous frame, and retain valid frame records with strictly increasing presentation timestamps. The target timeline is generated with the start time as the base point and the frame interval as the step size. The end of the target timeline is taken as the timestamp of the last frame in the buffer and quantized down to the time after the frame interval is aligned, and the total number of target frames is determined accordingly. Scan the target timeline moment by moment, locate the first record in the valid frame records whose presentation timestamp is not less than the target time as the subsequent frame, and take the record before it as the preceding frame; calculate the time difference between the target time and the preceding and subsequent frames respectively, and select the frame with the smaller time difference as the output frame corresponding to the target time; When the time span between two adjacent valid frames at the target time exceeds twice the frame interval, the output frame of the previous target time is copied as the output frame of the current target time and marked as a compensation frame. The output frames corresponding to each target time point are sequentially written into the frame sequence buffer according to the target timeline, forming a video frame sequence arranged in chronological order and then output.

[0020] In this embodiment, constructing the skeletal keypoint sequence includes: Receive a sequence of video frames arranged in chronological order, and read the first frame one by one. The frame image is scaled proportionally and filled with boundaries according to the preset input size. The scaling ratio and filling offset parameters are recorded to generate the pose analysis input frame. Human target detection is performed on the pose parsing input frame. Human detection boxes are extracted, and single target regions are selected according to the detection confidence. The original image is then cropped based on the detection boxes to generate corresponding human image patches, and the frame index is labeled. ; Input human image blocks into the keypoint detection network, output a preset number of two-dimensional coordinates of human skeleton keypoints and their confidence scores, and establish a frame-level keypoint coordinate set according to the keypoint topology. For keypoints with a confidence level below a preset threshold, structural constraint completion is performed: when there are adjacent keypoints in the same frame that have a skeleton connection with the keypoint, the coordinates of the keypoint are generated according to a preset skeleton length ratio. When the same frame cannot be completed, retrieve the coordinates of the same key point in the previous and next frames, and use linear interpolation to generate the coordinates of the key point. The coordinates of frame-level key points are inversely mapped from the human image block coordinate system to the original video frame coordinate system. During the inverse mapping process, the real spatial position is restored based on the recorded scaling ratio and fill offset parameters. For key points with the same name in adjacent video frames, the coordinate difference is calculated and compared with a preset jump threshold. When the coordinate difference exceeds the jump threshold, the key point is marked as an abnormal point and replaced with the coordinates of the key point with the same name in the previous frame, generating a key point coordinate set with consistent time sequence. The keypoint coordinate sets of each frame are arranged sequentially according to the time sequence of the video frame sequence to construct a skeletal keypoint sequence, where the first... Frame number The coordinates of the key points are represented as follows: The sequence of skeletal key points is represented as The skeletal key point sequence is then output to the Legendre memory unit as input data for continuous-time memory encoding.

[0021] In this embodiment, generating the memory state sequence includes: Receive skeletal keypoint sequence , will the The coordinates of all key points in the frame are concatenated into an input vector according to a fixed key point index order. and the first The frame's rendering timestamp is written to the timestamp queue to form a sequence with... Bound timing input records; The order of Legendre's memory cells is determined to be... In the interval Fixed selection Each sampling point will be used to transfer from order 0 to... The values ​​of the Legendre polynomials at each sampling point are written into the basis function table in order, and the values ​​of the first derivatives of each basis function at each sampling point are written into the derivative table in order. The basis function table and the derivative table serve as fixed lookup data in the construction process. A continuous-time memory kernel is constructed based on the basis function table and derivative table. The construction process includes: for each pair of order indices Using pre-defined Gaussian-Legend quadrature nodes and weights, for the "first" The first-order basis function and the second-order basis function "The product of the derivatives of the basis functions of order" The integral over is evaluated and written into the state coupling coefficient matrix, for the "first" The product of the first-order basis function and the input basis function is in The integral over the time step is evaluated and written into the input projection coefficient matrix; Write the state coupling coefficient matrix into the continuous-time state transition matrix. Write the input projection coefficient matrix into the continuous-time input matrix. And combine the product nodes, weights, basis function table, derivative table and Store together in a fixed state; Will Discretization is performed into frame-by-frame callable discrete operators. The discretization process includes: reading the nominal frame interval value corresponding to the video frame sequence, and discretizing the continuous-time state transition matrix according to the frame interval to obtain the discrete state transition matrix. The continuous-time input matrix is ​​then equivalently discretized to obtain the discrete input matrix. ,Will Write the parameters of the Legendre memory cell as a fixed operator; The first improvement introduced in the construction process of the Legendre memory unit is to constrain the input injection structure based on the skeletal topology. The constraint implementation steps include: reading the keypoint topological edge set and generating an adjacency table, where each joint in the adjacency table contains only its own index and the index of the joint connected to it in the topological edge set; and inputting the input vector... The joint is divided into joint channel blocks according to the joint index, and a connection is generated based on the adjacency table. Isomorphic mask matrix The setting rule of the mask matrix is ​​"set to 1 when the corresponding joint channel block in the state dimension satisfies the adjacency relationship with the corresponding joint channel block in the input dimension, otherwise set to 0"; the mask matrix is ​​applied to the discrete input matrix to obtain the topological constraint input matrix and written into the parameter area to replace : ; in, Represents element-wise product; A second improvement is introduced into the Legendre memory cell construction process: a multi-step advance operator table is established to address the non-uniform time advance caused by frame drops and compensation frames. The operator table construction steps include: setting an upper bound on the maximum allowed number of advance steps. ,by Pre-compute and store the matrix exponentiation sequence for the basis matrix. ,in Take the satisfaction The smallest integer; during the runtime phase, the time stamp difference between two adjacent frames is quantized to obtain the number of propulsion steps. ,Will The multi-step advancement matrix is ​​obtained by expanding the binary representation and multiplying the expanded results together.

[0022] in, To advance the number of steps The binary bits; the multi-step advancement matrix is ​​used as the current frame state advancement operator to participate in the memory state calculation; Initialize the memory state vector And write to the state cache, traversing in ascending order of frame index. It is bound to a timestamp, and state progression and input injection are performed frame by frame: first based on the number of progression steps. Generate multi-step advancement matrix and perform Execution status advancement, then Input matrix with topological constraints The mapped result is injected into the advanced state to obtain ,Will With frame index Bind to write memory cache; Index by Frame Sequential output of memory buffer forms a memory state sequence ,in The first of the skeletal key point sequence Each frame corresponds to a specific state, and the memory state sequence is output to the subsequent action boundary determination step.

[0023] In this embodiment, generating the action phase sub-sequence includes: Receive memory state sequence With skeletal keypoint sequence The memory state vector at each time step The Legendre order index is used to divide the components into lower-order and higher-order components. The lower-order components are determined by a predefined order. The first dimension is composed of the second dimension, and the higher-order components are composed of the remaining dimensions. The higher-order component energy and lower-order component energy are calculated time-by-time, and the increase in higher-order energy between adjacent time-times is calculated to form a boundary score sequence. The boundary score is calculated at the t-th time step. The time is recorded as: ; in, Indicates the first Time-of-flight higher-order components, Indicates the first Low-order components at any given time It is a fixed positive constant; Perform candidate boundary detection on the scoring sequence: with a fixed window length. Performing a local maximum search will satisfy the condition that "the current time step is the maximum value within the window and is continuous for at least 1000 seconds". The moment when the frame remains positive is recorded as the candidate boundary moment; Apply rigid interval constraints to the candidate boundaries: set the minimum boundary interval to... Frame, scan the candidate boundary list in chronological order, for any two candidate boundaries with an interval of less than In the case of frames, only candidate boundaries with larger score values ​​are retained; Perform a skeleton consistency check on each retained candidate boundary: take a fixed length of frame segment before and after the candidate boundary, calculate and compare the inter-frame displacement cumulative value of each joint in the two segments, and delete the candidate boundary whose displacement cumulative value does not meet the preset minimum change amount, thus obtaining the action boundary point sequence. Add the first frame index and the last frame index to the motion boundary point sequence, and then process the skeletal keypoint sequence according to the temporal order of the boundary points. The interval is segmented to obtain multiple action stage subsequences arranged in chronological order, and each action stage subsequence and its corresponding frame index interval are output for subsequent joint weighting and segment matching calculations.

[0024] In this embodiment, generating the weighted joint trajectory subsequence includes: Receive the The skeletal keypoint sequence corresponding to each action phase subsequence With memory state subsequence Read the starting frame index of the stage. With Termination Frame Index and the stage length Write the stage parameter table; Memory state vector Split into pre-fixed joint channel mapping tables Each joint channel segment corresponds one-to-one with a skeletal keypoint index; within each stage, the preceding frame set is extracted. With the set of subsequent frames For each joint channel segment, the average value of the L2 norm is calculated frame by frame in the front and back segments and written into the joint mean buffer. Regarding the first For each joint, the mean values ​​of its posterior and anterior segments are read, the raw joint score is calculated, and written into the joint score vector. The raw joint score is denoted as: ; in, Indicates the first The first frame in the memory state vector Each joint channel segment, This means taking the average value within the previous frame set. This indicates that the average value is taken within the set of subsequent frames; Perform topology unification on the joint score vector: read the skeleton topology edge set and generate a fixed adjacency list for each joint; update the score one joint at a time according to the joint index order. The update rule is to take the arithmetic mean of the current joint score and the scores of the joints in its adjacency list and write it back to the current joint score, while keeping the adjacency list unchanged during the update process. Normalization is performed on the standardized joint score vector. Specifically, the scores of all joints are summed as the normalization benchmark. Each joint score is then divided by the normalization benchmark to obtain the stage-fixed joint weight vector. This weight vector is then compared with the stage index. The binding storage is used as a stage weight table record; Within a stage, traverse the coordinates of the skeletal keypoints frame by frame and read the first... Frame number The coordinates of the first key point are read, and the record of the first key point in the stage weight table is retrieved. The joint weights are multiplied by each coordinate component of the keypoint coordinates to generate the first joint weight. Frame number The coordinates of each weighted key point are calculated and written to the weighted frame buffer. From frame index to The weighted frame buffer is written to the stage trajectory buffer in sequence to obtain the first... The weighted joint trajectory subsequences for each action phase are generated; the weighted joint trajectory subsequences are output in chronological order of the action phases.

[0025] In this embodiment, generating the stage matching distance value corresponding to each action stage subsequence includes: Receive the Weighted joint trajectory subsequences for each action phase No. Phase fixed joint weight vector and the first Stage Template Trajectory Read the phase start frame index given by the action boundary point. With Termination Frame Index Limit the set of trajectory indexes to be matched to And limit the template trajectory index set to ; Establish a frame pair distance lookup table. The lookup process is as follows: for each pair of indexes... Read sequentially The Middle The joint coordinates of the frame and The Middle For each joint coordinate in a frame, the Euclidean distance between the coordinates of corresponding joints in two frames is calculated and multiplied by the corresponding joint weight. The weighted distances of all joints are summed to obtain a joint term. Simultaneously, the coordinate differences between the joints at both ends of each bone segment are read according to the skeleton topology edge set to form a bone segment vector. The Euclidean distance between the differences of corresponding bone segment vectors in two frames is weighted by the arithmetic mean of the joint weights at both ends of the bone segment and summed to obtain a bone segment term. The joint term and the bone segment term are added together and written into the frame pair distance table. The frame pair distance table is... The distance at point is denoted as: ; in, For the first Joint weights For the number of joints, For the skeleton topological edge set, For the trajectory to be matched Frame number Joint coordinates For template trajectory number Frame number Joint coordinates; The first improvement is implemented in the construction process of discrete Fraser matching, establishing a piecewise constraint of "boundary anchor point forced passage", specifically: Treat the matching trajectory in the interval The system calculates a frame-by-frame cumulative displacement table. The cumulative displacement is calculated by summing the weighted displacement distances across all joints in two adjacent frames and then accumulating the sums. The total cumulative displacement from the start point to the end point is then divided into fixed equal parts. Divide the data into equal parts. For each division threshold, scan the cumulative displacement table from front to back, and record the frame index that first reaches the threshold, thus obtaining the sequence of anchor point frame indices to be matched. ; Calculate the cumulative displacement table for the template trajectory according to the same rules and generate the template anchor point frame index sequence. Pair anchor points as The anchor point pairs are then written into the anchor point constraint table, which specifies that the matching path must follow the rules. The ascending order passes through each pair of anchor points sequentially. The second improvement is implemented in the construction process of discrete Fraser matching, establishing an index constraint of "hard pruning of the banded reachability domain", specifically: Calculate the length of the trajectory to be matched With template length Template indexes are determined by integer division and remainders. Mapping to the center index of the match ; For each template index Extend fixed width on both sides of the central index. The frame positions are obtained to obtain the... The corresponding allowed index range to be matched ,in , ; All Write to the reachable domain table, and specify that it is only possible when... Time frame pair For reachable frame pairs, unreachable frame pairs are not included in the recursive calculation; The overall matching interval is divided into sections based on the anchor point constraint table. There are n consecutive subintervals, with the endpoints of the subintervals being, in order: The piecewise constrained discrete Fréchet distance is calculated segment by segment for each sub-interval. The calculation process is as follows: A two-dimensional recursive table is established within each sub-interval. The row index of the recursive table is the index of the frame to be matched allowed in the sub-interval, and the column index is the index of the template frame allowed in the sub-interval. The recursive table is filled in ascending order of rows and columns. Each cell only allows transitions from three adjacent cells: "top", "left", and "top left". The cell cost is taken from the distance table value of the cell frame pair. The larger of the source path cost and the smaller of the costs of the three adjacent source cells; for unreachable frame pairs Mark directly as unreachable and skip padding; initialize the starting cell of each sub-interval with the distance of the corresponding frame pair; After each sub-interval is filled, the cost value of the end cell of the sub-interval is read as the segmented Frescher value of the sub-interval. The maximum value of all segmented Frescher values ​​of the sub-intervals is taken as the first... The stage matching distance value for each stage; The first Stage matching distance value and stage index Bind and write the stage matching result table, and output the set of matching distance values ​​for each stage in chronological order of the action stages.

[0026] In this embodiment, the action categories corresponding to the video to be identified include: Receive a set of action stages arranged in chronological order of action boundary points, and read the stage matching distance value, the memory state subsequence, and the start and end frame index range of each stage. For each stage, the memory state vector of the first frame of the stage is extracted as the stage start state vector, and the memory state vector of the last frame of the stage is extracted as the stage end state vector, and written into the stage state table in the stage order. Based on the temporal order of the action boundary points, the stage state table is checked for sequential consistency. If the starting frame index of any stage is not less than its ending frame index, or if the ending frame index of any adjacent stage is not less than the starting frame index of the next stage, the current stage division result is marked as invalid and subsequent category calculations are terminated. Provided that the phase sequence is valid, for each pair of adjacent phases, calculate the Euclidean distance between the termination state vector of the previous phase and the starting state vector of the next phase, and use it as the cross-phase continuity deviation value. Write the continuity deviation values ​​of all adjacent phases into the continuity table. Establish a set of candidate action categories, and for each candidate action category, read its preset number of stages, the upper bound of the stage matching distance corresponding to each stage template, and the template state transition vector between each adjacent stage. When the preset number of stages for a candidate action category is inconsistent with the current number of action stages, the candidate category is marked as unavailable; when the stage matching distance value of any stage exceeds the upper bound of the matching distance of the corresponding stage of the candidate category, the candidate category is marked as unavailable; when the continuity deviation value of any adjacent stage is greater than the preset continuity threshold, the candidate category is marked as unavailable. For candidate action categories that are not marked as unavailable, the comprehensive decision cost of the category is calculated. The comprehensive decision cost is obtained by weighting the maximum value of the matching distance value of all stages and the maximum value of the continuity deviation value of all adjacent stages in a fixed proportion. The candidate action category with the lowest overall judgment cost among all candidate action categories is selected as the action category corresponding to the video to be identified, and the action category is output.

[0027] Example: To verify the effectiveness of this invention, it is applied to a video action recognition method and system based on skeletal key points. In this scenario, monitoring equipment continuously collects videos of people's activities. These videos often contain continuous actions such as walking, standing still, turning, waving, squatting, and falling. Furthermore, the same person often transitions continuously from one action to another within a short period. Existing technologies typically classify the entire video segment uniformly, easily mixing multiple action stages. In cases of partial occlusion of personnel, video frame rate fluctuations, or temporary loss of key points, problems such as unclear action boundaries, increased trajectory matching deviations, and unstable recognition results may arise. Therefore, in this embodiment, the video action recognition method based on skeletal key points described in the claims is used to perform staged recognition and classification processing of continuous action videos.

[0028] In practical applications, after the front-end camera outputs the video to be recognized, the video acquisition module decapsulates and decodes the video stream, reads the time base, average frame rate, and presentation time information of each frame from the video track, and generates a video frame sequence in chronological order. For videos with duplicate frames, timestamp rollback, or partially dropped frames, the system first clears the time anomaly records, then constructs a target timeline based on the set frame interval, and remaps the original frames onto a unified timeline. When the span between adjacent valid frames is too large, a compensation frame is generated by copying the previous output frame, thus obtaining a video frame sequence with consistent chronological order. This processing avoids timing errors caused by unstable acquisition of the original video, providing a unified input for subsequent motion modeling.

[0029] After obtaining the video frame sequence, the keypoint sequence construction module performs human pose analysis on each frame. The system first scales the image proportionally and fills the boundaries according to a preset input size. Then, it extracts the target human body region through a human detection network, performs keypoint detection on the target human body, and outputs the coordinates of multiple human skeletal keypoints, including the head, shoulders, elbows, wrists, hips, knees, and ankles. For keypoints with a confidence level below a threshold, the system prioritizes structural completion based on skeletal connections and bone segment length ratios. If completion is not possible in the same frame, linear interpolation is performed using keypoints with the same name from adjacent frames. For keypoints with abnormal coordinate changes, the system verifies them based on jump thresholds in adjacent frames and replaces the abnormal points with the coordinates of the same keypoint from the previous frame, ultimately forming a temporally consistent skeletal keypoint sequence. This process reduces the impact of occlusion, jitter, and detection errors on action recognition.

[0030] The skeletal keypoint sequence is input into the Legendre memory unit for continuous-time memory encoding. Specifically, the system concatenates the coordinates of all keypoints in each frame according to a fixed index order to form an input vector, while recording the timestamp associated with that frame. Then, a Legendre basis function table and derivative table are established according to a preset order to construct continuous-time state transition relationships and input projection relationships, which are discretized into discrete state update operators called frame-by-frame based on the video frame interval. To adapt to the characteristics of the skeletal structure, the system further applies mask constraints to the input injection matrix based on the keypoint topological relationships, ensuring that each joint channel is primarily coupled with itself and adjacent joint channels. To adapt to non-uniform time progression, a multi-step progression matrix is ​​generated based on the time difference between adjacent frames to advance the state as needed. Through frame-by-frame state progression and input injection, a memory state sequence corresponding one-to-one with each frame is obtained. This technical feature enables the system not only to save historical change information of actions but also to maintain continuous temporal representation even with uneven frame rates or the presence of compensation frames.

[0031] After the memory state sequence is generated, the action boundary determination module determines the action boundaries based on the changes in memory states at adjacent time points. The system divides the memory states into low-order and high-order components, calculates their energy changes separately, constructs a boundary scoring sequence based on the increment of high-order energy at adjacent time points, and then extracts candidate action boundaries by combining local maximum search, continuity as a positive constraint, and minimum boundary interval constraint. For candidate boundaries, the system also extracts fixed-length skeletal frame segments before and after the boundary, compares the cumulative values ​​of each joint displacement, and eliminates false boundaries that do not meet the minimum change requirement. After filtering, the first and last frames are added to the boundary set, and the skeletal keypoint sequence is divided into multiple action stage subsequences according to the boundary position. This decomposes a continuous action into several clearer local stages, solving the problem of stage aliasing during unified recognition of the entire segment.

[0032] After the action phases are divided, the weighted trajectory generation module determines the joint weights based on the memory state sequences corresponding to each phase. The system first divides the memory states by joint channels, statistically analyzes the state intensity changes of each joint in the pre- and post-phase segments, and generates an initial score for each joint. Then, based on the skeleton topology, it performs adjacency averaging on the joint scores to obtain topologically consistent scores, which are then normalized to form a fixed joint weight vector for that phase. The system multiplies the coordinates of each keypoint in each frame by the corresponding joint weight to generate a weighted joint trajectory subsequence for that phase. Since different action phases have varying degrees of dependence on parts such as hands, legs, and torso, this processing highlights the contribution of key joints and enhances the feature representation ability within each phase.

[0033] In the matching and recognition phase, the system defines the matching intervals for each weighted joint trajectory subsequence based on the action boundaries, and performs matching within the intervals using the joint-weighted piecewise constrained discrete Fraser distance method. Specifically, the system first establishes a frame pair distance table between the trajectory to be matched and the stage template trajectory. The distance values ​​are composed of the coordinate differences of corresponding joints and the differences in bone segment vectors, weighted according to joint weights. Then, an anchor point frame sequence is constructed based on the cumulative displacement of each stage, ensuring that the matching path passes through the corresponding anchor point pairs in sequence. Simultaneously, a strip-shaped reachable domain is established for the template frames based on the trajectory length relationship, allowing only frame pairs satisfying the index constraints to enter the recursive calculation. Finally, piecewise discrete Fraser recursion is performed within each sub-interval, and the maximum value among the results of each sub-interval is used as the matching distance value for that stage. Compared to the overall matching method without piecewise constraints, this technique significantly reduces stage mismatch and path drift.

[0034] In the category determination phase, the system reads the matching distance values ​​of each action phase, the memory states corresponding to the first and last frames of each phase, and the boundary time sequence relationships. It first checks the validity of the phase sequence and then calculates the continuity deviation between adjacent phases. For each candidate action category, the system compares its preset number of phases, the upper bound of the matching distance for each phase, and the state transition relationships between adjacent phases, eliminating categories that do not meet the conditions. For the remaining candidate categories, the system calculates a comprehensive determination cost based on the phase matching results and the continuity deviation, ultimately selecting the category with the lowest cost as the action category of the video to be identified. This approach not only utilizes the local matching results within each phase but also leverages the overall continuity between phases, improving the reliability of the final classification results.

[0035] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A video motion recognition system based on skeletal key points, characterized in that, include: The video acquisition module acquires the video to be identified and generates a sequence of video frames arranged in chronological order. The key point sequence construction module performs pose analysis on the video frame sequence, extracts the coordinates of key points of the human skeleton, and generates a skeleton key point sequence according to the time order of the video frame sequence. The memory state generation module inputs the skeletal key point sequence into the Legendre memory unit, performs continuous time memory encoding, and generates a memory state sequence corresponding to each moment of the skeletal key point sequence. The action boundary determination module determines action boundary points based on the state changes of adjacent moments in the memory state sequence, and divides the skeletal key point sequence according to the action boundary points to generate action stage sub-sequences arranged in chronological order. The weighted trajectory generation module determines the joint weights of skeletal key points based on the memory state sequence corresponding to each action stage subsequence, and generates a weighted joint trajectory subsequence by combining the coordinates of skeletal key points in the action stage subsequence. The matching and recognition module defines the matching interval of each weighted joint trajectory subsequence based on the action boundary points, and uses the Fraser distance action recognition method within the matching interval to perform matching calculations between each weighted joint trajectory subsequence and the corresponding stage template trajectory to generate a stage matching distance value. The category determination module performs cross-stage continuity determination based on the stage matching distance value, the corresponding memory state sequence, and the temporal order of action boundary points, and outputs the action category corresponding to the video to be identified.

2. A video action recognition method based on skeletal key points, characterized in that, The modules are connected in the following way: Acquire the video to be identified and generate a sequence of video frames arranged in chronological order; Human pose analysis is performed on the video frame sequence, the coordinates of human skeletal key points corresponding to each video frame are extracted, and a skeletal key point sequence is constructed according to the time order of the video frame sequence. The sequence of skeletal key points is input into the Legendre memory unit and continuously encoded in time memory to generate a memory state sequence that corresponds one-to-one with each moment of the skeletal key point sequence. The action boundary points are determined based on the state changes of adjacent moments in the memory state sequence, and the skeletal key point sequence is divided according to the action boundary points to generate action stage subsequences arranged in chronological order. The joint weights of the skeletal key points are determined based on the memory state sequence corresponding to each action stage subsequence, and a weighted joint trajectory subsequence is generated by combining the coordinates of the skeletal key points in each action stage subsequence. The matching interval of each weighted joint trajectory subsequence is defined based on the action boundary points. Within the matching interval, the weighted joint segmentation constraint discrete Frescher distance is used to match and calculate the matching distance between each weighted joint trajectory subsequence and the corresponding stage template trajectory, generating the stage matching distance value corresponding to each action stage subsequence. Based on the stage matching distance value corresponding to each action stage subsequence, the memory state sequence corresponding to each action stage subsequence, and the temporal order relationship of the action boundary points, cross-stage continuity determination is performed, and the action category corresponding to the video to be identified is output.

3. The video action recognition method based on skeletal key points according to claim 2, characterized in that, Constructing the skeletal keypoint sequence includes: Receive a sequence of video frames arranged in chronological order, and read the first frame one by one. The frame image is scaled proportionally and filled with boundaries according to the preset input size. The scaling ratio and filling offset parameters are recorded to generate the pose analysis input frame. Human target detection is performed on the pose parsing input frame. Human detection boxes are extracted, and single target regions are selected according to the detection confidence. The original image is then cropped based on the detection boxes to generate corresponding human image patches, and the frame index is labeled. ; Input human image blocks into the keypoint detection network, output a preset number of two-dimensional coordinates of human skeleton keypoints and their confidence scores, and establish a frame-level keypoint coordinate set according to the keypoint topology. For keypoints with confidence levels below a preset threshold, structural constraint completion is performed: when there are adjacent keypoints in the same frame that have a skeleton connection with the keypoint, the coordinates of the keypoint are generated according to a preset skeleton length ratio. When the same frame cannot be completed, retrieve the coordinates of the same key point in the previous and next frames, and use linear interpolation to generate the coordinates of the key point. The coordinates of frame-level key points are inversely mapped from the human image block coordinate system to the original video frame coordinate system. During the inverse mapping process, the real spatial position is restored based on the recorded scaling ratio and fill offset parameters. For key points with the same name in adjacent video frames, the coordinate difference is calculated and compared with a preset jump threshold. When the coordinate difference exceeds the jump threshold, the key point is marked as an abnormal point and replaced with the coordinates of the key point with the same name in the previous frame, generating a key point coordinate set with consistent time sequence. The keypoint coordinate sets of each frame are arranged sequentially according to the time sequence of the video frame sequence to construct a skeletal keypoint sequence, where the first frame... Frame number The coordinates of the key points are represented as follows: The sequence of skeletal key points is represented as The sequence of key skeletal points is then output.

4. The video action recognition method based on skeletal key points according to claim 2, characterized in that, Generating a memory state sequence includes: Receive skeletal keypoint sequence , will the The coordinates of all key points in the frame are concatenated into an input vector according to a fixed key point index order. and the first The frame's rendering timestamp is written to a timestamp queue to form a sequence with the frame's timestamp. Bound timing input records; The order of Legendre's memory cells is determined to be... In the interval Fixed selection Each sampling point will be used to transfer from order 0 to... The values ​​of the Legendre polynomials at each sampling point are written into the basis function table in order, and the values ​​of the first derivatives of each basis function at each sampling point are written into the derivative table in order. The basis function table and the derivative table serve as fixed lookup data in the construction process. A continuous-time memory kernel is constructed based on the basis function table and derivative table. The construction process includes: for each pair of order indices Using pre-defined Gaussian-Legend quadrature nodes and weights, for the "1st"... The first-order basis function and the second-order basis function "The product of the derivatives of the basis functions of order" The integral over is evaluated and written into the state coupling coefficient matrix, for the "th The product of the first-order basis function and the input basis function is in The integral over the time matrix is ​​evaluated and written into the input projection coefficient matrix; Write the state coupling coefficient matrix into the continuous-time state transition matrix. Write the input projection coefficient matrix into the continuous-time input matrix. And combine the product nodes, weights, basis function table, derivative table and Store together in a fixed state; Will Discretization is performed into frame-by-frame callable discrete operators. The discretization process includes: reading the nominal frame interval value corresponding to the video frame sequence, and discretizing the continuous-time state transition matrix according to the frame interval to obtain a discrete state transition matrix. The continuous-time input matrix is ​​then equivalently discretized to obtain the discrete input matrix. ,Will Write the Legendre memory cell parameter area as a fixed operator; In the construction process of Legendre memory units, the input injection structure is constrained based on the skeletal topology. The constraint implementation steps include: reading the keypoint topological edge set and generating an adjacency table, where each joint in the adjacency table contains only its own index and the indices of the joints connected to it in the topological edge set; and inputting the input vector... The joint is divided into joint channel blocks according to the joint index, and a connection is generated based on the adjacency table. Isomorphic mask matrix The setting rule of the mask matrix is: "set to 1 when the corresponding joint channel block in the state dimension satisfies the adjacency relationship with the corresponding joint channel block in the input dimension, otherwise set to 0"; apply the mask matrix to the discrete input matrix to obtain the topological constraint input matrix and write it into the parameter area for replacement. : ； in, Represents element-wise product; In the construction process of Legendre memory cells, a multi-step advancement operator table is established to address the non-uniform time advance caused by frame drops and compensation frames. The steps for constructing the operator table include: setting an upper bound on the maximum allowed number of advancement steps. ,by Pre-compute and store the matrix exponentiation sequence for the basis matrix. ,in Take the satisfaction The smallest integer; during the runtime phase, the time stamp difference between two adjacent frames is quantized to obtain the number of propulsion steps. ,Will The multi-step advancement matrix is ​​obtained by expanding the binary representation and multiplying the expanded results together. ； in, To advance the number of steps The binary bits; the multi-step advancement matrix is ​​used as the current frame state advancement operator to participate in the memory state calculation; Initialize the memory state vector And write to the state cache, traversing in ascending order of frame index. It is bound to a timestamp, and state progression and input injection are performed frame by frame: first based on the number of progression steps. Generate multi-step advancement matrix and perform Execution status advancement, then Input matrix with topological constraints The mapped result is injected into the advanced state to obtain ,Will With frame index Bind to write memory cache; Index by Frame Sequential output of memory buffer forms a memory state sequence ,in The first of the skeletal key point sequence Each frame corresponds to a specific state, and the memory state sequence is output to the subsequent action boundary determination step.

5. The video action recognition method based on skeletal key points according to claim 2, characterized in that, The generated action phase subsequence includes: Receive memory state sequence With skeletal keypoint sequence The memory state vector at each time step The Legendre order index is used to divide the components into lower-order and higher-order components. The lower-order components are determined by a predefined order. The first dimension is composed of the second dimension, and the higher-order components are composed of the remaining dimensions. The higher-order component energy and lower-order component energy are calculated time-by-time, and the increase in higher-order energy between adjacent time-times is calculated to form a boundary score sequence. The boundary score is calculated at the t-th time step. The time is recorded as: ； in, Indicates the first Time-of-flight higher-order components, Indicates the first Low-order components at any given time It is a fixed positive constant; Perform candidate boundary detection on the scoring sequence: with a fixed window length. Performing a local maximum search will satisfy the condition that "the current time step is the maximum value within the window and is continuous for at least 1000 seconds". The moment when the frame remains positive is recorded as the candidate boundary moment; Apply rigid interval constraints to the candidate boundaries: set the minimum boundary interval to... Frame, scan the candidate boundary list in chronological order, for any two candidate boundaries with an interval of less than In the case of frames, only candidate boundaries with larger score values ​​are retained; Perform a skeleton consistency check on each retained candidate boundary: take a fixed length of frame segment before and after the candidate boundary, calculate and compare the inter-frame displacement cumulative value of each joint in the two segments, and delete the candidate boundary whose displacement cumulative value does not meet the preset minimum change amount, thus obtaining the action boundary point sequence. Add the first frame index and the last frame index to the motion boundary point sequence, and then process the skeletal keypoint sequence according to the temporal order of the boundary points. By dividing the interval, multiple action stage subsequences are obtained, arranged in chronological order.

6. The video action recognition method based on skeletal key points according to claim 2, characterized in that, Generating weighted joint trajectory subsequences includes: Receive the The skeletal keypoint sequence corresponding to each action phase subsequence With memory state subsequence Read the starting frame index of the stage. With Termination Frame Index and the stage length Write the stage parameter table; Memory state vector Split into pre-fixed joint channel mapping tables Each joint channel segment corresponds one-to-one with a skeletal keypoint index; within each stage, the preceding frame set is extracted. With the set of subsequent frames For each joint channel segment, the average value of the L2 norm is calculated frame by frame in the front and back segments and written into the joint mean buffer. Regarding the first For each joint, read the mean of its rear segment and the mean of its front segment, calculate the original joint score, and write it into the joint score vector. Perform topology unification on the joint score vectors; Normalization is performed on the standardized joint score vectors; Within a stage, the coordinates of the skeletal keypoints are traversed frame by frame. The joint weights are multiplied by each coordinate component of the keypoint coordinates to generate the first... Frame number The coordinates of each weighted key point are calculated and written to the weighted frame buffer. From frame index to The weighted frame buffer is written to the stage trajectory buffer in sequence to obtain the first... The weighted joint trajectory subsequences for each action phase are generated; the weighted joint trajectory subsequences are output in chronological order of the action phases.

7. The video action recognition method based on skeletal key points according to claim 2, characterized in that, The generated stage matching distance values ​​for each action stage subsequence include: Receive the Weighted joint trajectory subsequences for each action phase No. Phase fixed joint weight vector and the first Stage Template Trajectory Read the phase start frame index given by the action boundary point. With Termination Frame Index Limit the set of trajectory indexes to be matched to And limit the template trajectory index set to ; Establish a frame-to-frame distance lookup table; the frame-to-frame distance table is located in... The distance at point is denoted as: ； in, For the first Joint weights For the number of joints, For the skeleton topological edge set, For the trajectory to be matched Frame number Joint coordinates For template trajectory number Frame number Joint coordinates; In the construction process of discrete Fraser matching, a piecewise constraint for mandatory passage at boundary anchor points is established, specifically as follows: Treat the matching trajectory in the interval The system calculates a frame-by-frame cumulative displacement table. The cumulative displacement is calculated by summing the weighted displacement distances across all joints in two adjacent frames and then accumulating the sums. The total cumulative displacement from the start point to the end point is then divided into fixed equal parts. Divide the data into equal parts. For each division threshold, scan the cumulative displacement table from front to back, and record the frame index that first reaches the threshold, thus obtaining the sequence of anchor point frame indices to be matched. ; Calculate the cumulative displacement table for the template trajectory according to the same rules and generate the template anchor point frame index sequence. Pair anchor points as The anchor point pairs are then written into the anchor point constraint table, which specifies that the matching path must follow the rules. The ascending order passes through each pair of anchor points sequentially. In the construction process of discrete Fraser matching, an index constraint for hard pruning of the banded reachability domain is established, specifically as follows: Calculate the length of the trajectory to be matched With template length Template indexes are determined by integer division and remainders. Mapping to the center index of the match ; For each template index Extend fixed width on both sides of the central index. The frame positions are obtained to obtain the... The corresponding allowed index range to be matched ,in , ; All Write to the reachable domain table, and specify that only when Time frame pair For reachable frame pairs, unreachable frame pairs are not included in the recursive calculation; The overall matching interval is divided into sections based on the anchor point constraint table. There are n consecutive subintervals, with the endpoints of the subintervals being, in order: The piecewise constrained discrete Fréchet distance is calculated segment by segment for each sub-interval. The calculation process is as follows: A two-dimensional recursive table is established within each sub-interval. The row index of the recursive table is the index of the frame to be matched allowed in the sub-interval, and the column index is the index of the template frame allowed in the sub-interval. The recursive table is filled in ascending order of rows and columns. Each cell only allows transitions from three adjacent cells: "top", "left", and "top left". The cell cost is taken from the distance table value of the cell frame pair. The larger of the source path cost and the smaller of the costs of the three adjacent source cells; for unreachable frame pairs Mark directly as unreachable and skip padding; initialize the starting cell of each sub-interval with the distance of the corresponding frame pair; After each sub-interval is filled, the cost value of the end cell of the sub-interval is read as the segmented Frescher value of the sub-interval. The maximum value of all segmented Frescher values ​​of the sub-intervals is taken as the first... The stage matching distance value for each stage; The first Stage matching distance value and stage index Bind and write the stage matching result table, and output the set of matching distance values ​​for each stage in chronological order of the action stages.

Images