Method for constructing, updating and retrieving action memory bank

Abstract

The invention discloses a method for constructing, updating and retrieving an action memory library, which belongs to the technical field of computer vision and comprises the following steps of: constructing a training data source with time sequence diversity; initializing an action memory library containing a plurality of learnable prototype matrixes and a double-flow interaction network; memory bank evolution is executed, an action prototype is retrieved by utilizing a query stream, a current memory state is dynamically generated by combining a memory state updating gate mechanism with a historical state, and dynamic memory is injected into a feature space by utilizing memory driving graph convolution; synchronously updating parameters based on multi-target loss, and driving a memory bank to evolve into optimal structured prior; and finally, performing structured reasoning on the to-be-detected sequence by using the optimal memory bank. According to the method, structured priori is constructed by mining a spatio-temporal topology mode of a human body action sequence, and hierarchical memory evolution and double-flow depth interaction are combined, so that the problem of depth ambiguity in a monocular vision task is effectively solved, geometric structure distortion is corrected, and the accuracy of action posture estimation is remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, and in particular to a method for constructing, updating and retrieving an action memory bank. Background Technology

[0002] The digital reconstruction and understanding of human movements is a core topic in computer vision and graphics. Although the trajectory of the human skeleton in high-dimensional space appears complex and unpredictable, in-depth manifold analysis reveals that human movement data is not inherently randomly distributed, but rather embedded in a low-dimensional manifold space. This means that massive long-sequence movements exhibit high sparsity and reusability in their topological structure. Whether walking, grasping, or obstacle avoidance, these complex macroscopic behaviors are essentially composed of a finite number of action primitives or action atoms with specific topological structures. These action patterns with standard topological structures demonstrate strong structural invariance across different subjects and environmental contexts. Therefore, based on these theoretical findings, constructing a mechanism capable of accurately capturing, storing, and retrieving these action primitives has become crucial for understanding the laws governing human movement.

[0003] However, existing deep learning techniques generally suffer from limitations such as implicit modeling and static rigidity when dealing with this problem. Current mainstream methods tend to implicitly encode prior human actions into the massive weight parameters of neural networks. This "black box" memorization method means that the model lacks a visual, discretizable reference standard. During training, the model often memorizes specific sample distributions rather than learning general topological rules, making it not only difficult to interpret but also prone to forgetting sparse samples. Especially in the ill-posed inverse problem of monocular video 3D human pose estimation, which involves endogenous depth ambiguity, traditional models can only rely on trained parameters for blind inference when the input video encounters severe occlusion, self-occlusion, or noise interference. This easily leads to depth ambiguity phenomena such as disproportionate limbs or joint angles violating physiological limitations. Furthermore, existing technologies lack a hierarchical evolution mechanism for prior knowledge, resulting in a static and fragmented state of memory priors between deep and shallow network layers. This fails to guarantee that deep action states are progressively optimized and confirmed based on shallow retrieval results. Moreover, there is a lack of deep bidirectional interaction and injection mechanisms between structured action priors and visual observation features, hindering the deep correction of visual ambiguity by manifold constraints. Therefore, there is an urgent need to improve existing technologies to address the problems of depth ambiguity and geometric distortion caused by the lack of explicit manifold guidance and hierarchical dynamic evolution in monocular vision tasks. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of implicit modeling in existing technologies, which lack interpretability and static priors cannot adapt to complex manifold changes. It provides a method for constructing, updating, and retrieving action memory databases to solve the problems of depth ambiguity and structural distortion in monocular 3D human pose estimation.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for constructing, updating, and retrieving an action memory bank, wherein the method constructs a structured prior by mining the spatiotemporal topological patterns of human action sequences, specifically including the following steps: Training data source for building action memory bank By performing temporal sampling on the original human action sequence, action slice sequences with temporal diversity and their corresponding spatial topological truth values ​​are generated. Initialize the action memory M With the driving network, wherein the action memory library M Include K A learnable prototype matrix is ​​used to store potential topological patterns, and the driving network is a two-stream interactive network that includes a spatiotemporal feature stream and a memory-induced pose query stream. Execution action memory M The evolution and state update of the training data source D Import the two-stream interactive network and use the memory-induced posture query stream to retrieve data from the action memory bank. M The system retrieves matching action prototypes by establishing a memory state update gate mechanism and combining historical states. Dynamically generate the current memory state And utilize the memory-driven graph convolution module to convert the current memory state Inject into the feature space; Synchronously update network parameters and prototype matrices in the action memory database, calculate gradients based on multi-target loss, and drive the action memory database using the backpropagation algorithm. Evolves from an initial random distribution to the optimal structured prior. ; Based on the best structured prior Perform structured reasoning and load the optimal structured prior. While maintaining the operation of the memory state update gate mechanism, geometric correction and three-dimensional pose inference are performed on the action sequence to be tested.

[0006] Preferably, the training data source for constructing the action memory bank D The specific steps include: Set the action memory library M The standard input frame length required for pattern mining; Generate uniformly distributed time anchor points for the original video sequence; For each time anchor point, a random resampling strategy is adopted to select the frame corresponding to its rounded-down or rounded-up index with random probability to construct a sequence, generating motion slices with different rates and phases as input material for mining motion topology patterns.

[0007] Furthermore, the initialization action memory library M The specific steps for configuring the driver network include: Define the structural capacity of the action memory library, and set the action memory library. M Number of learnable prototype matrices K This parameter determines the upper limit of the diversity of different human motion topology patterns stored in the memory bank; Define the input resolution of the memory bank and set the standard frame length of the action slice sequence. T Number of nodes in the human skeleton J This ensures that the memory can capture long-range temporal dependencies and fine-grained spatial structures; Define the hyperparameters of the evolution-driven network, and set the feature embedding dimension, the latent state dimension of the state space model, and the local convolutional kernel size.

[0008] Preferably, the spatiotemporal feature stream specifically includes parallel selective state-space model branches and self-attention branches: The selective state space model branch is connected in series with the spatial selective scanning module and the temporal selective scanning module. It utilizes the linear recursive characteristics of the selective state space model to capture the local spatial topology and local dynamic evolution features of adjacent joints in the input action sequence within a short time. The self-attention branch is connected in series with the temporal self-attention module and the spatial self-attention module. It uses a multi-head self-attention mechanism to capture the global long-range dependency of the input action sequence in the time dimension and the global joint association in the spatial dimension. The spatiotemporal feature stream also includes a feature fusion unit, which is used to calculate the adaptive fusion weights of the two branches and generate enhanced visual features.

[0009] Furthermore, the execution action memory library M In the evolutionary construction and state update steps, from the action memory bank M The specific action prototypes for searching and matching in the middle include: The memory-induced pose query stream is used to generate pose query vectors through adaptive pooling; The pose query vector and the action memory are calculated through a learnable network layer. M middle K Matching weights for each prototype matrix; Based on the matching weights, all prototype matrices are weighted and aggregated to generate the instantaneous retrieval pattern for the current moment. .

[0010] Furthermore, the establishment of the memory state update gate mechanism specifically includes: An inter-layer state evolution strategy is introduced to adaptively generate and update the gating coefficients based on the current query context features. Combined with the historical state of the previous network layer With the instantaneous retrieval mode Calculate the current memory state of the current network layer. The calculation formula is: ; Wherein, the updated gating coefficient g This is used to dynamically adjust the fusion ratio between newly retrieved action topology patterns and historical memory states, thereby ensuring the consistency of the action memory database during the deep feature extraction process of the network.

[0011] More specifically, the memory-driven graph convolution module is used to process the current memory state. The process of injecting the feature space specifically includes: Define learnable blend parameters Construct a dynamic adjacency matrix ; The dynamic adjacency matrix Based on a predefined static anatomical adjacency matrix With the current memory state It is formed by weighted fusion, and the fusion formula is: ; Using the dynamic adjacency matrix A graph convolution operation is performed on the pose query vector, and the propagation path of information between human skeletal nodes is adjusted according to dynamic memory to generate pose query features enhanced by secondary structuring.

[0012] Preferably, the execution action memory library M The evolutionary construction and state update steps also include: A bidirectional cross-attention mechanism is constructed to enable deep interaction between the memory-induced pose query stream and the spatiotemporal feature stream. On the one hand, the enhanced visual features extracted from the spatiotemporal feature stream are used as queries, and the pose query features enhanced by secondary structuring are used as keys to inject structured priors into the visual representation. On the other hand, reverse interaction is used to correct the bias in memory retrieval by utilizing the enhanced visual features.

[0013] Furthermore, the step of synchronously updating the network parameters and the prototype matrix in the action memory specifically includes: A multi-objective weighted total loss function is constructed, which is a weighted sum of the average joint position error of constraint geometric accuracy, the scale normalization error of constraint scale consistency, and the velocity smoothing error of constraint time continuity. The gradient is calculated using the backpropagation algorithm, which not only updates the weights of the two-stream interaction network but also directly affects the action memory. M Update the parameters of all prototype matrices in the dataset; At the end of each training cycle, if the error on the validation set is lower than the historical minimum, the optimized action memory at the current time step is saved as the best structured prior. .

[0014] Furthermore, the aforementioned optimal structured prior... The specific steps involved in performing structured reasoning include: Obtain the sequence of actions to be tested, and load the optimal structured sequence. ; During inference, gradient updates for parameters and prototypes are stopped, but the memory state update gate mechanism remains active. Based on the input of the action sequence to be tested, the current memory state of each layer is dynamically generated using the memory state update gate mechanism. And combined with the aforementioned optimal structured prior The provided topological constraints are used to perform geometric corrections on the sequence of actions to be tested, and output a three-dimensional attitude sequence.

[0015] This invention discloses a method for constructing, updating, and retrieving an action memory, the core of which lies in explicitly constructing a visualized action memory as a structured prior. Compared with existing technologies, this invention has significant advantages: First, explicit modeling of the human action manifold structure and manifold projection inference based on dynamic references are achieved. Unlike existing technologies that implicitly encode action priors in black-box weights, this invention constructs a visualized graph-structured action memory library that conforms to the distribution laws of the manifold. During inference, the model no longer relies on blind parameter fitting but actively retrieves high-confidence typical action prototypes. By explicitly retrieving high-confidence action prototypes as dynamic references, and utilizing explicit topological constraints, damaged visual observations are projected back into the standard action manifold space, thus effectively solving the problems of geometric ambiguity and structural distortion caused by depth loss in monocular vision tasks.

[0016] Secondly, a hierarchical memory evolution mechanism was established to ensure the robustness of feature abstraction. Addressing the problem of existing technologies using fixed constraints at different feature levels, this invention introduces a memory state update gate mechanism. This mechanism establishes an inter-layer memory transfer channel, allowing deep networks to reuse and correct action priors retrieved from shallow layers, realizing memory evolution and confirmation from coarse-grained to fine-grained, thereby ensuring the accuracy and consistency of retrieval results in complex feature spaces.

[0017] Furthermore, the accuracy of the memory index is maximized. To ensure that the most accurate action template can be retrieved from the memory, this invention designs a heterogeneous dual-stream driven network that combines a selective state-space model with a self-attention mechanism. This design cleverly balances local high-frequency dynamic details and global long-range spatial relationships in human motion, providing a high-quality visual index with rich spatiotemporal context for memory retrieval, ensuring a high degree of matching between the retrieved action prototype and the current motion state. The heterogeneous dual-stream architecture combining a selective state-space model and a self-attention mechanism balances local high-frequency dynamic details and global long-range spatiotemporal relationships, providing a high-quality visual index with rich context for the memory and significantly improving retrieval accuracy.

[0018] Furthermore, a deep fusion of structured priors and visual features is achieved. This invention does not simply treat priors as external constraints, but rather constructs a dynamic adjacency matrix through memory-driven graph convolution. It then uses learnable parameters to deeply and organically integrate data-driven dynamic semantic associations with physically defined static anatomical connections. A bidirectional cross-attention mechanism is used to inject this strongly structured information into the feature space, achieving geometric correction and enhancement of visual features, thereby significantly improving reconstruction accuracy and generalization ability in uncontrolled outdoor scenarios. By constructing a dynamic adjacency matrix through memory-driven graph convolution, the deep injection of structured priors is achieved, significantly improving the model's generalization ability and reconstruction accuracy in uncontrolled outdoor scenarios.

[0019] Finally, a general motion topology ecosystem foundation was constructed, enabling a leap from discrimination tasks to generation tasks. The motion memory library constructed in this invention is essentially a general human motion memory library with topological completeness and periodic closed-loop characteristics. This standardized motion primitive not only solves the pose estimation problem but also provides a reusable underlying graph for downstream tasks such as digital human driving, motion generation, and efficient data compression, possessing extremely high general value and potential for ecological expansion. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the overall process of constructing, updating, and retrieving an action memory bank, as provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the overall network model based on the memory-induced dual-stream interaction architecture in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram illustrating the error comparison and performance improvement of the action memory bank memory retrieval matching strategy in an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram illustrating the performance distribution of each independent memory unit and the combined memory bank in an embodiment of the present invention.

[0025] Figure 5 This is a visualization heatmap of the spatial topology patterns contained in the action memory library in this embodiment of the invention.

[0026] Figure 6 This is a schematic diagram of the skeletal topology of the action prototypes contained in the action memory library in an embodiment of the present invention.

[0027] Figure 7 This is a visualization comparison of the method of this invention and existing mainstream technologies on the Human 3.6M dataset.

[0028] Figure 8 This is a graph showing the generalization ability test results of the method of this invention in a real field scenario on the 3DPW dataset.

[0029] Figure 9 This is a visual comparison chart of the characteristic response patterns of different branches in the dual-branch architecture of this invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0031] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0032] To address the core shortcomings of existing data-driven human motion modeling techniques, such as implicit black-boxing and lack of manifold structure, existing techniques generally implicitly encode human motion priors in the massive weight parameters of deep neural networks, resulting in a lack of discretizable indexable standard references for model inference.

[0033] Based on this, in order to improve the problems in related technologies, such as Figure 1 As shown in the figure, this application provides a method for constructing, updating, and retrieving an action memory bank, which mainly includes the following steps: Step S1: Construct a training data source for mining action topology patterns.

[0034] Establish a dataset containing human action sequences and their corresponding spatial topological truths. ;in Indicates the total number of samples; Indicates the first A sequence of two-dimensional skeletal keypoints serving as the context for querying the action memory bank; Indicates the first The ground truth values ​​of 3D skeleton keypoints are used as topological supervision signals.

[0035] Input context definition: Input sequence ,in The standard frame length for action slices (for example, set in this embodiment) ), The number of skeletal joints (set in this embodiment) ), For the input channel dimension. The first frame Each joint, its data is represented as (Right now (two-dimensional plane coordinates) or (Right now (Coordinates including confidence level), this embodiment preferentially uses The standardized two-dimensional coordinates plus confidence scores are used as input features for memory retrieval. Topological truth definition: Target truth value. For the first The first frame Each joint, its truth value Composed of three elements of a Cartesian coordinate system, it represents the absolute three-dimensional spatial position of a human joint relative to the root node, and is used to guide the evolution direction of the prototype matrix in the motion memory bank.

[0036] Temporal enhancement strategy: Preset length for original video sequences with insufficient frame count. In cases where this is not possible, or to uncover action patterns at different rates, a random resampling strategy is employed: generating... The algorithm uses uniformly distributed time anchors and randomly selects an index for sampling at each anchor, either rounded down or rounded up. This strategy not only achieves fixed-length sequence processing but, more importantly, enhances the temporal diversity of the data by simulating different action phases and rates, thereby making the constructed action memory bank more robust.

[0037] One specific implementation method is as follows: First, the Human3.6M dataset is selected as the baseline material library for constructing the motion memory bank. This dataset was collected by a motion capture system and contains video footage of seven professional actors in an indoor environment, totaling approximately 3.6 million frames. Second, the training and test sets are divided according to a standard protocol: data from subjects S1, S5, S6, S7, and S8 are selected to construct the training set, which drives the evolution of the memory bank; data from subjects S9 and S11 are selected to construct the test set, which is used to verify the effectiveness of the memory bank. The dataset contains 15 typical daily action categories, covering a rich variety of human movement postures, which can provide diverse prototype materials for the memory bank. Third, the original materials are preprocessed. The video frames are downsampled to 50Hz, and the motion memory of each frame is extracted. Two-dimensional coordinates of individual body joints are used as input features. Finally, a random resampling strategy is implemented to generate standardized action slices. Specifically, this involves generating... Using uniformly distributed time anchor points, for each anchor point, the frame corresponding to its rounded-down or rounded-up index is randomly selected to complete the sequence. Through this process, the training data source for the action memory bank is constructed, achieving not only tensor standardization of the data but also laying a solid data foundation for subsequent action topology pattern mining. At this point, This indicates a completed training data source; This represents the preprocessed and resampled two-dimensional skeletal input tensor, which serves as the query for memory retrieval. This represents the corresponding 3D pose truth value label; ,in This refers to the total number of processed samples, for example, approximately 1.56 million frames in the H36M training set. To remember the window length, The number of key points in the human skeleton is defined using the H36M standard skeleton. Representing two-dimensional plane coordinates Add confidence level; By the spatial Cartesian coordinate system Composition. The above definition ensures consistency between the input data and the interface dimension of the memory-driven network, providing a standardized data foundation for subsequent batch memory evolution.

[0038] Step S2: Initialize the action memory library and construct a two-stream interaction network model that drives its evolution.

[0039] Specifically, first construct a system containing A graph-structured action memory library with a learnable prototype matrix. This serves as a container for storing typical human movement topological patterns. Secondly, to drive the evolution of this memory bank from a randomly initialized state to an optimal structured prior state, a two-stream interactive network model integrating visual perception and memory evolution mechanisms was constructed. This model consists of a spatiotemporal feature stream providing multi-scale visual context, a memory-induced pose query stream performing retrieval and evolution, and a bidirectional cross-attention interaction module for deep feature injection.

[0040] Building upon the spatiotemporal feature stream, an adaptive fusion unit is first used to weightedly fuse the two heterogeneous features, generating enhanced visual features that balance local details and global correlations. Simultaneously, a memory-induced pose query stream integrating retrieval and evolution functions is constructed. This stream first utilizes the query vector generated by adaptive pooling to retrieve the action memory database during the network's forward propagation. The most suitable action prototype; then, a memory state update gate mechanism is introduced, combining the memory state of the previous network layer. Dynamically generate new states through hierarchical evolution based on the current search results. This establishes a cross-level memory transfer channel, enabling progressive confirmation and optimization of action priors from coarse-grained to fine-grained. Subsequently, the memory-driven graph convolution module is used to process the memory states representing dynamic semantic associations. The static anatomical adjacency matrix, representing the physical skeleton, is deeply fused with learnable parameters to construct an adaptive dynamic graph structure for secondary structural enhancement of query features. Finally, a bidirectional cross-attention interaction module is used to make the spatiotemporal feature stream of visual context and the enhanced post-query stream, which serves as memory prior, mutually act as query and key, achieving depth alignment and feature injection. Ultimately, a geometrically reasonable 3D pose is output via a regression head, while multi-target loss is calculated to drive the parameter update and evolution of the action memory bank.

[0041] The specific steps of the implementation process are as follows: The first step is to initialize the action memory and build a two-stream interaction network architecture. First, build a network containing... A graph-structured action memory library with a learnable prototype matrix. This serves as a container for storing typical human movement topological patterns. Secondly, to drive the evolution of this memory bank from a randomly initialized state to an optimal structured prior state, a two-stream interactive network model integrating visual perception and memory evolution mechanisms was constructed. This model consists of a spatiotemporal feature stream providing multi-scale visual context, a memory-induced pose query stream performing retrieval and evolution, and a bidirectional cross-attention interaction module for deep feature injection.

[0042] The second step involves constructing a heterogeneous dual-branch feature encoding module for extracting multi-scale visual context. This module serves as the visual perception foundation of the action memory bank, responsible for capturing high-fidelity spatiotemporal features, aiming to provide a high-quality visual index rich in context for subsequent memory retrieval. To balance the heterogeneous characteristics of local high-frequency dynamics and global long-range dependencies in human motion, this module employs a heterogeneous parallel architecture composed of selective state-space model branches, self-attention branches, and adaptive fusion units.

[0043] Specifically, a selective state-space model branch is first constructed, focusing on capturing local high-frequency dynamic details. This branch employs a spatial-first, temporal-second processing logic, leveraging the linear recursive properties of the selective state-space model to efficiently capture the subtle movement trajectories of limbs within a short timeframe. Its computational process is described as follows: ;in, As input features, and These represent the spatially selective scanning module for capturing physical connections and the temporally selective scanning module for capturing dynamic changes, respectively. Simultaneously, to compensate for the limitations of local features in capturing macroscopic structures, a parallel self-attention branch is constructed. This branch focuses on capturing global long-range spatial dependencies, employing a temporal-first-space processing logic. It utilizes the global receptive field of the self-attention mechanism to establish semantic associations between distant joints. Its computational process is described as follows: ;in, and These represent the temporal and spatial self-attention modules, respectively. Finally, an adaptive fusion unit is used to integrate the two heterogeneous features. This unit dynamically calculates the confidence weights of the two branches through a gating mechanism. Based on this, weighted enhanced features are generated. ;in For learnable projection matrices, This involves feature concatenation. Through the above steps, the model can adaptively adjust its focus on local details or global structure based on the complexity of the actions, ultimately outputting enhanced full-resolution spatiotemporal features that take into account both micro and macroscopic aspects.

[0044] The third step involves retrieving, updating, and injecting structured data into the action memory. At this point, enhanced visual features extracted from the spatiotemporal feature stream are fed into the interaction module as multi-scale visual observation evidence, ready for deep interaction and fusion with structured prior knowledge from the memory-induced pose query stream. The entire memory's operational mechanism is as follows: First, the memory retrieval and hierarchical state evolution mechanism is initiated. The network uses the pose query vector to address within the action memory bank and calculates its relation to... The similarity weights of a learnable prototype matrix are used to retrieve the instantaneous action pattern, which serves as a data-driven structured prior, at the current feature level. Next, to address potential local biases in single-level retrieval and maintain consistency in feature abstraction, a memory state update gate mechanism is introduced. This mechanism utilizes adaptive gating coefficients. Dynamically adjust the memory state of the previous network layer The degree of retention and the new retrieval mode of the current layer The degree of acceptance is determined by the formula. Generate the hierarchically modified current memory state This establishes a cross-level memory transfer channel. Following this, the memory-driven structured injection stage begins. This stage aims to transform abstract memory states into concrete geometric constraints, its core being the construction of an adaptive dynamic adjacency matrix using a memory-driven graph convolutional network. This matrix is ​​obtained through the formula. This will represent a predefined static anatomical adjacency matrix that represents inviolable physical connection constraints. With dynamic memory states representing semantic priors that change with posture Deep fusion is performed; then, the dynamic graph is used to convolve the features, strongly injecting high-confidence topological priors into the feature stream, outputting structured enhanced features. Based on this, bidirectional cross-attention interaction is executed to achieve deep alignment between vision and memory. On one hand, the spatiotemporal feature stream is used as the query, and the structured features enhanced by graph convolution are used as the key, injecting structured priors into the visual representation; on the other hand, reverse interaction is used to correct the bias in memory retrieval using rich visual details, ultimately outputting fused features that combine high fidelity of visual observation with the rationality of anatomical structure, fundamentally solving the problem of depth ambiguity. Finally, an evolution-driven regression head maps this fused feature into a three-dimensional coordinate sequence. During the training phase, the prediction error output by the regression head is transformed into a gradient, which is used to synchronously update the parameters of the prototype matrix in the action memory through the backpropagation algorithm until it converges from the initial random state to the optimal structured prior that can accurately represent the laws of human movement. .

[0045] Step 4: Define the multi-objective loss function and perform joint evolutionary training. The final step in constructing the action memory bank lies in guiding the randomly initialized prototype matrix to converge into a meaningful topological pattern. Therefore, this embodiment first defines a multi-objective weighted total loss function to constrain the evolution of the action memory bank. .

[0046] Specifically, the function consists of three parts: first, the geometric evolution constraint term. The first constraint uses the mean joint position error (MPJPE) as a measure to constrain the accuracy of motion patterns stored in the memory bank in absolute spatial coordinates; the second constraint is the topological consistency constraint. The method employs PA-MPJPE error aligned with Procrustes to eliminate individual scale differences, forcing the memory bank to focus on learning pure topological structures that do not change with height; thirdly, it incorporates temporal coherence constraints. The norm of the joint displacement difference between adjacent frames is used as a metric to penalize unnatural abrupt changes and force the memory to learn smooth motion transition patterns. After establishing the evolutionary goal, the parameter environment of the evolution-driven network is configured. Specific settings are as follows: standard frame length of action slices. Number of skeletal joints Number of prototype memory The AdamW optimizer was selected (initial learning rate). (In conjunction with exponential decay). During training, the gradient is calculated using the aforementioned total loss function, and the network parameters are updated synchronously via backpropagation. With the prototype of the action memory bank When the evaluation error on the validation set reaches its historical minimum, a fixation operation is performed, saving the optimized action memory at the current moment as the best action memory. And save the best network parameters simultaneously. This completes the entire process of constructing a standardized prior library from a random distribution.

[0047] Step S3: Load the optimized action memory library and execute structured reasoning based on prior memory. This step aims to optimize the action memory library constructed in step S2. This approach is applied to practical 3D human pose estimation tasks, resolving depth ambiguity in monocular vision through explicit structured priors. The specific implementation process is as follows: First, load the optimal model state and the test data. Then, obtain the monocular video action sequence to be tested. And load the best motion memory library that was solidified and saved in step S2. and the corresponding optimal network weights At this point, the network enters inference mode and stops processing the parameters. and prototype Gradient updates are performed, but all internal dynamic gating mechanisms remain active. Secondly, hierarchical evolution-based dynamic reference inference is executed. The test sequence is input into the network, and during layer-by-layer feature extraction and interaction, the hierarchical memory evolution mechanism remains active. This means that, despite the prototype of the memory bank... It is fixed, but the network will still calculate the instantaneous retrieval pattern for each layer based on the current input. And combined with the memory state of the previous network layer Through formula The current memory state is dynamically generated and hierarchically modified. This mechanism ensures that the inference process is not a static lookup, but a progressive confirmation of action priors from coarse to fine granular. Finally, structured priors are injected and high-precision poses are output. Using a memory-driven graph convolution module, the evolved dynamic memory state is transformed into specific geometric constraints and forcefully injected into the feature space of the test sequence. This process is equivalent to using explicit standard action references to fill in the missing depth information in monocular vision, effectively correcting structural distortions caused by occlusion or blurring. Finally, the regression head outputs a 3D pose sequence based on the structured enhanced features. Experiments show that this inference process can significantly reduce the average joint position error in predicted postures and output high-fidelity 3D motion that conforms to the laws of human kinematics, thus fully verifying the effectiveness of the motion memory bank as an explicit prior.

[0048] To more intuitively illustrate the specific execution logic of the methods described in steps S1 to S3 above, especially how the dual-stream interactive network constructed in step S2 works collaboratively under the evolution drive of step S3, the embodiments of the present invention provide the following algorithm flow.

[0049] First, Algorithm 1 demonstrates the overall process of jointly updating network parameters and the memory. As the evolution-driven master control logic, this algorithm details how to connect the aforementioned data preprocessing, forward inference based on a two-stream architecture, and multi-objective loss calculation, driving the network parameters through the backpropagation algorithm. With the prototype of the memory bank Achieve synchronous convergence.

[0050] To further clarify the specific operational mechanism of the two-stream interaction network invoked in step 12 of Algorithm 1, the overall process of its forward propagation and memory interaction is shown in Algorithm 2. This algorithm demonstrates how the input action slice, serving as the query context, undergoes position encoding, two-stream visual context extraction, and memory retrieval and structured injection to ultimately complete the feature enhancement and inference process. In this process, two core modules are given clear functional definitions: the two-stream spatiotemporal feature encoding module is defined as a multi-scale visual index generator in this algorithm. It is no longer just a feature extractor but is responsible for extracting high-fidelity spatiotemporal features, providing a reliable visual context for accurate addressing in the action memory bank. The memory-induced interaction module is defined as the core of the memory bank operation. It is responsible for coordinating the execution of three key tasks: retrieving and matching action prototypes based on visual indexes, performing hierarchical evolution of memory states by combining information from the previous network layer, and transforming the updated dynamic memory into structured constraints and injecting them into the feature stream.

[0051] Regarding the dual-branch feature encoding step in step 16 of Algorithm 2, its internal parallel processing logic is shown in Algorithm 3. This algorithm details how to construct a high-quality visual retrieval index for the action memory bank. Specifically, it executes the selective state-space model branch and the self-attention branch in parallel to capture the local high-frequency dynamic evolution and global long-range spatiotemporal dependence required for matching the prototype in the memory bank, respectively. Based on this, an adaptive gating unit is used to achieve dynamic weighted fusion of the two visual evidence streams, thereby generating an enhanced visual context with rich spatiotemporal semantics, ensuring that the most matching topological pattern can be accurately retrieved from the action memory bank subsequently.

[0052] Regarding the memory-induced interaction module called in step 19 of Algorithm 2, its specific memory retrieval, hierarchical evolution, and structured injection logic are shown in Algorithm 4. This algorithm demonstrates how action priors are transformed from abstract memory states into concrete geometric constraints. Its core process is as follows: First, the current action prototype is retrieved through global pooling and similarity calculation. Then, a new state with hierarchical correction is dynamically generated using the memory state update gate mechanism and combined with the state of the previous network layer. Finally, an adaptive adjacency matrix that integrates physical skeleton connections and dynamic semantic associations is constructed, and the structured priors are deeply injected into the visual feature stream through graph convolution and cross attention.

[0053] In summary, through the multi-level collaborative work of Algorithms 1 to 4, the evolutionary construction process of this method constitutes a rigorous closed-loop system: Algorithm 3 is responsible for constructing a high-quality visual retrieval index, ensuring the richness of the context; Algorithm 2 and its sub-module Algorithm 4 are responsible for the retrieval of execution memory, hierarchical state evolution, and structured injection, providing reliable dynamic priors; while Algorithm 1, as the overall controller of evolution-driven processes, ensures the synchronous convergence and optimization of network parameters and prototype matrices in the action memory bank through backpropagation.

[0054] like Figure 3 to Figure 9 As shown, the embodiments of the present invention have conducted comprehensive simulation experiments and performance evaluations on the proposed action memory bank construction, updating and retrieval methods.

[0055] First, the effectiveness of the internal operating mechanism of the action memory bank was quantitatively verified. Figure 3 The ablation experiments of the memory retrieval matching strategy are presented in detail. By comparing the error curves of dynamically learned weights and static uniform weights, it is confirmed that the retrieval mechanism proposed in this invention can adaptively focus on the most relevant memory prototypes, thereby significantly improving the accuracy of retrieval. Based on this, Figure 4 Further, the performance distribution of each independent memory unit and the combined memory bank was revealed. The statistical results of MPJPE and PA-MPJPE indicators show that the evolved memory prototype, while maintaining high diversity, can cover the complex and ever-changing human action space as a whole, verifying the synergistic effect of the best individual memory and the complete memory bank in feature expression.

[0056] Secondly, a visual analysis was conducted to examine the interpretability of the model and the heterogeneity of feature extraction. To demonstrate the explicit existence of action priors, Figure 5 and Figure 6 Heatmaps of 48 spatial topological patterns from the motion memory bank and their corresponding skeletal visualizations are presented. These clear, structured patterns intuitively demonstrate that the model has successfully overcome the interpretability bottleneck of traditional implicit parameter fitting, concretizing the abstract motion laws latent in the data into explicit and readable structured reference standards. Meanwhile... Figure 9 By comparing the feature response patterns of different branches in the two-stream network, the results clearly show that the selective state space model branch is highly focused on the local high-frequency changes of the limb extremities, while the self-attention branch extensively covers the long-range correlations of joints throughout the body. This significant heterogeneous complementary characteristic perfectly matches the original design intention of this invention for multi-scale visual indexing.

[0057] Finally, a qualitative comparison was made on the accuracy and generalization ability of the final generated pose. Figure 7The visualization comparison of the proposed method with existing mainstream techniques on the Human 3.6M dataset is presented. The results show that when encountering occlusion or complex poses, the proposed method can effectively avoid the depth ambiguity and structural distortion problems common in other methods, generating geometrically more rigorous and reasonable 3D poses; furthermore, Figure 8 The model's performance in real-world scenarios on the 3DPW dataset is further demonstrated, proving that even when faced with unfamiliar and complex backgrounds and uncontrolled actions, the feature stream incorporating structured priors still maintains strong robustness and generalization ability.

[0058] Table 1 compares the performance of the action memory bank construction method proposed in this invention with other existing mainstream technologies on the Human3.6M test set.

[0059] Table 1 The data in the table clearly shows that, thanks to the strongly structured priors provided by the explicit action memory base, this method significantly outperforms existing mainstream methods on several key evaluation metrics. Specifically, this method reduces MPJPE to 37.1 mm and PA-MPJPE to 31.4 mm. This result strongly demonstrates that, compared to traditional methods such as MotionBERT and MotionAGFormer, which rely on implicitly encoding prior knowledge into network weights, the dynamic reference reasoning mechanism based on explicit retrieval adopted in this invention can more effectively utilize external knowledge bases to fill in the gaps in visual information, thereby fundamentally solving the problem of deep ambiguity.

[0060] Table 2 shows the cumulative contribution of the action memory bank construction, update, and retrieval strategies in this invention to model performance. To verify the necessity of each core evolution step of the memory bank, the experiment adopted a strategy of gradually stacking components starting from the baseline model. The data shows that: First, after introducing the basic memory retrieval mechanism, the error index decreased significantly, which directly proves that the explicit introduction of action prototypes can effectively correct posture distortion and provide a reliable reference benchmark; Second, after adding the memory state update gate mechanism, the model accuracy was further improved, and the speed error of the action sequence was significantly reduced, which verifies that this mechanism effectively enhances the consistency of feature abstraction and the stability of the evolution process by establishing memory transfer channels between levels; Finally, after integrating memory-driven graph convolution, the system performance reached its optimal level, which proves that transforming dynamic memory states into specific graph structure geometric constraints is a key step in realizing deep feature injection and shape correction.

[0061] Table 2 Table 3 analyzes the impact of the number of prototype matrices in the action memory bank on performance. This experiment aims to explore the nonlinear relationship between the structural capacity of the memory bank and the model's generalization ability. Experimental results show that when the number of prototypes is set to... K When the value is 48, the model achieves the optimal balance between retrieval accuracy and computational efficiency. Data trends indicate that if... K If the value is too small, the memory capacity is limited and cannot cover a sufficiently diverse range of action topology patterns; while if K If the value is too large, it will lead to the generation of a large number of redundant prototypes, which not only increases the computational cost of retrieval but may also introduce noise interference. This experimental data clarifies the optimal discretization granularity of the action space, providing solid empirical evidence for the setting of memory structure parameters in the claims.

[0062] Table 3 Table 4 highlights the effectiveness of different visual index generation strategies in the driving network. This experiment verifies the superiority and necessity of the proposed heterogeneous dual-branch architecture in providing high-quality retrieval indexes for the action memory. Comparative data shows that using only the selective state space branch or only the self-attention branch results in lower pose accuracy compared to the two-stream fusion architecture. This result strongly demonstrates that the accurate operation of the action memory relies on complete visual evidence containing both local micro-details and global macro-context. This heterogeneous and complementary design ensures that the action memory can accurately match the most suitable topological pattern from complex and varied inputs, thus supporting the scientific nature and irreplaceable nature of this invention in driving network design.

[0063] Table 4 Implementing the embodiments of this invention has the following beneficial effects: Compared with the prior art, the embodiments of this invention achieve the following significant technical advancements by constructing an action memory bank and establishing a corresponding hierarchical evolution and deep injection mechanism: First, addressing the problems of depth ambiguity and loss of geometric details caused by existing single implicit reasoning mechanisms, this invention establishes a dynamic reference reasoning mechanism based on explicit retrieval by explicitly constructing a graph-structured action memory bank. The model no longer relies on implicit fitting using static parameters highly compressed in the neural network weights, but can actively retrieve high-confidence typical action prototypes from the memory bank as dynamic references based on visual cues. This not only provides interpretable structured priors for monocular vision tasks with uncertainty, but more importantly, this invention introduces a hierarchical memory state evolution mechanism. This mechanism establishes a cross-level memory transfer channel by dynamically fusing the memory state of the previous network layer with the retrieval mode of the current layer, realizing progressive confirmation and correction of action priors from coarse-grained to fine-grained, thereby effectively solving the geometric distortion caused by occlusion or feature blurring in complex scenes, and ensuring the consistency and robustness of generated poses at the feature abstraction level. Secondly, addressing the issue that existing single feature extraction paradigms are insufficient for high-precision retrieval, this invention designs a heterogeneous dual-branch multi-scale visual perception architecture. To provide the action memory with a high-quality visual retrieval index rich in context, this invention innovatively deploys a selective state-space model branch and a self-attention mechanism branch in parallel. The former utilizes linear recursion to efficiently capture local high-frequency dynamic details at the extremities, while the latter focuses on capturing the global long-range spatial dependencies of all joints. This heterogeneous complementary design, emphasizing both micro and macro aspects, significantly enhances the expressive power of visual features, ensuring that the model can accurately locate the topological pattern that best fits the current motion state within the vast memory, avoiding erroneous retrievals caused by index bias. Finally, addressing the problem that existing structured constraints are only shallow unidirectional injections, this invention designs a deep fusion mechanism of memory-driven graph convolution and bidirectional cross-attention. This invention does not simply superimpose priors, but rather deeply and organically integrates data-driven dynamic semantic associations with predefined static anatomical constraints through learnable parameters, constructing an adaptive dynamic adjacency matrix. Furthermore, it utilizes a bidirectional cross-attention mechanism to enable structured priors and visual features to act as queries and keys at a deep network level, performing depth alignment and geometric calibration. This deep interaction paradigm successfully transforms abstract memory states into concrete geometric constraints, significantly improving the reconstruction accuracy and generalization ability of 3D human pose estimation in uncontrolled outdoor scenarios.

[0064] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, optical disk, etc.

[0065] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for constructing, updating, and retrieving an action memory database, characterized in that, The method constructs structured priors by mining the spatiotemporal topological patterns of human action sequences, specifically including the following steps: Training data source for building action memory bank By performing temporal sampling on the original human action sequence, action slice sequences with temporal diversity and their corresponding spatial topological truth values ​​are generated. Initialize the action memory M With the driving network, wherein the action memory library M Include K A learnable prototype matrix is ​​used to store potential topological patterns, and the driving network is a two-stream interactive network that includes a spatiotemporal feature stream and a memory-induced pose query stream. Execution action memory M The evolution and state update of the training data source D Import the two-stream interactive network and use the memory-induced posture query stream to retrieve data from the action memory bank. M The system retrieves matching action prototypes by establishing a memory state update gate mechanism and combining historical states. Dynamically generate the current memory state And utilize the memory-driven graph convolution module to convert the current memory state Inject into the feature space; Synchronously update network parameters and prototype matrices in the action memory database, calculate gradients based on multi-target loss, and drive the action memory database using the backpropagation algorithm. Evolves from an initial random distribution to the optimal structured prior. ; Based on the best structured prior Perform structured reasoning and load the optimal structured prior. While maintaining the operation of the memory state update gate mechanism, geometric correction and three-dimensional pose inference are performed on the action sequence to be tested.

2. The method for constructing, updating, and retrieving an action memory bank according to claim 1, characterized in that, The training data source for constructing the action memory bank D The specific steps include: Set the action memory library M The standard input frame length required for pattern mining; Generate uniformly distributed time anchor points for the original video sequence; For each time anchor point, a random resampling strategy is adopted to select the frame corresponding to its rounded-down or rounded-up index with random probability to construct a sequence, generating motion slices with different rates and phases as input material for mining motion topology patterns.

3. The method for constructing, updating, and retrieving an action memory bank according to claim 1, characterized in that, The initialization action memory library M The specific steps for configuring the driver network include: Define the structural capacity of the action memory library, and set the action memory library. M Number of learnable prototype matrices K This parameter determines the upper limit of the diversity of different human motion topology patterns stored in the memory bank; Define the input resolution of the memory bank and set the standard frame length of the action slice sequence. T Number of nodes in the human skeleton J This ensures that the memory can capture long-range temporal dependencies and fine-grained spatial structures; Define the hyperparameters of the evolution-driven network, and set the feature embedding dimension, the latent state dimension of the state space model, and the local convolutional kernel size.

4. The method for constructing, updating, and retrieving an action memory bank according to claim 1, characterized in that, The spatiotemporal feature stream specifically includes parallel selective state-space model branches and self-attention branches: The selective state space model branch is connected in series with the spatial selective scanning module and the temporal selective scanning module. It utilizes the linear recursive characteristics of the selective state space model to capture the local spatial topology and local dynamic evolution features of adjacent joints in the input action sequence within a short time. The self-attention branch is connected in series with the temporal self-attention module and the spatial self-attention module. It uses a multi-head self-attention mechanism to capture the global long-range dependency of the input action sequence in the time dimension and the global joint association in the spatial dimension. The spatiotemporal feature stream also includes a feature fusion unit, which is used to calculate the adaptive fusion weights of the two branches and generate enhanced visual features.

5. The method for constructing, updating, and retrieving an action memory bank according to claim 1, characterized in that, The execution action memory bank M In the evolutionary construction and state update steps, from the action memory bank M The specific action prototypes for searching and matching in the middle include: The memory-induced pose query stream is used to generate pose query vectors through adaptive pooling; The pose query vector and the action memory are calculated through a learnable network layer. M middle K Matching weights for each prototype matrix; Based on the matching weights, all prototype matrices are weighted and aggregated to generate the instantaneous retrieval pattern for the current moment. .

6. The method for constructing, updating, and retrieving an action memory bank according to claim 5, characterized in that, The mechanism for establishing a memory state update gate specifically includes: An inter-layer state evolution strategy is introduced to adaptively generate and update the gating coefficients based on the current query context features. Combined with the historical state of the previous network layer With the instantaneous retrieval mode Calculate the current memory state of the current network layer. The calculation formula is: ; Wherein, the updated gating coefficient g This is used to dynamically adjust the fusion ratio between newly retrieved action topology patterns and historical memory states, thereby ensuring the consistency of the action memory database during the deep feature extraction process of the network.

7. The method for constructing, updating, and retrieving an action memory bank according to claim 6, characterized in that, The memory-driven graph convolution module is used to determine the current memory state. The process of injecting the feature space specifically includes: Define learnable blend parameters Construct a dynamic adjacency matrix ; The dynamic adjacency matrix Based on a predefined static anatomical adjacency matrix With the current memory state It is formed by weighted fusion, and the fusion formula is: ; Using the dynamic adjacency matrix A graph convolution operation is performed on the pose query vector, and the propagation path of information between human skeletal nodes is adjusted according to dynamic memory to generate pose query features enhanced by secondary structuring.

8. The method for constructing, updating, and retrieving an action memory bank according to claim 7, characterized in that, The execution action memory bank M The evolutionary construction and state update steps also include: A bidirectional cross-attention mechanism is constructed to enable deep interaction between the memory-induced pose query stream and the spatiotemporal feature stream. On the one hand, the enhanced visual features extracted from the spatiotemporal feature stream are used as queries, and the pose query features enhanced by secondary structuring are used as keys to inject structured priors into the visual representation. On the other hand, reverse interaction is used to correct the bias in memory retrieval by utilizing the enhanced visual features.

9. The method for constructing, updating, and retrieving an action memory bank according to claim 1, characterized in that, The steps for synchronously updating network parameters and the prototype matrix in the action memory specifically include: A multi-objective weighted total loss function is constructed, which is a weighted sum of the average joint position error of constraint geometric accuracy, the scale normalization error of constraint scale consistency, and the velocity smoothing error of constraint time continuity. The gradient is calculated using the backpropagation algorithm, which not only updates the weights of the two-stream interaction network but also directly affects the action memory. M Update the parameters of all prototype matrices in the dataset; At the end of each training cycle, if the error on the validation set is lower than the historical minimum, the optimized action memory at the current time step is saved as the best structured prior. .

10. A method for constructing, updating, and retrieving an action memory bank according to any one of claims 1 to 9, characterized in that, The basis of optimal structured priors The specific steps involved in performing structured reasoning include: Obtain the sequence of actions to be tested and load the optimal structured prior. ; During inference, gradient updates for parameters and prototypes are stopped, but the memory state update gate mechanism remains active. Based on the input of the action sequence to be tested, the current memory state of each layer is dynamically generated using the memory state update gate mechanism. And combined with the aforementioned optimal structured prior The provided topological constraints are used to perform geometric corrections on the sequence of actions to be tested, and output a three-dimensional attitude sequence.

Images