A dynamic pedestrian three-dimensional reconstruction method based on adaptive prior fusion

Abstract

This invention relates to the field of 3D reconstruction technology, and discloses a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, comprising: acquiring a video frame sequence and initializing a Gaussian ellipsoid; generating a clean kinematic human body mask and an accessory mask; acquiring accessories based on the accessory mask and calculating the global transformation matrix of the accessories; extracting temporal information, instance embedding features, and visual features, concatenating them, and inputting them into an adaptive guidance module to obtain conditional feature vectors and reliability confidence scores, and predicting the non-rigid residual deformation of the Gaussian ellipsoid; calculating the final state of the Gaussian ellipsoid based on the non-rigid residual deformation; inputting the Gaussian ellipsoid in the final state into a differentiable rasterizer to generate a 2D rendered image; calculating the joint loss function and optimizing the Gaussian ellipsoid; and generating a 3D reconstructed image of the dynamic pedestrian based on the optimized Gaussian ellipsoid and the global transformation matrix. This invention can improve the accuracy of dynamic pedestrian 3D reconstruction.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, and particularly relates to a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion. Background Technology

[0002] In autonomous driving simulation and virtual reality applications, high-fidelity reconstruction of dynamic pedestrians outdoors is a core challenge. Existing mainstream methods typically combine parametric human models (such as Skinned Multi-Person Linear Model, SMPL) as priors and utilize 3D Gaussian Splatting (3DGS) for scene representation.

[0003] However, existing technologies suffer from the following drawbacks when handling complex real-world scenes: High-frequency detail loss and prior conflict: Parametric priors, such as parametric human models, only provide a smooth geometric foundation, lacking high-frequency details like clothing wrinkles. When human movements are complex or occlusion exists, parametric models are often inaccurate. Existing methods struggle to balance "macro-geometric correction" with "micro-detail generation," resulting in reconstructions that are either overly smooth or produce artifacts. Human-object structural conflation: Existing data annotations are often coarse, frequently merging pedestrians and their accessories (such as backpacks and suitcases) into a single mask. This causes the model to incorrectly integrate accessories into the human geometry during training, resulting in semantic ambiguity and physically inconsistent deformations, preventing accessories from moving naturally with the human body.

[0004] In summary, existing technologies result in low accuracy in dynamic pedestrian 3D reconstruction. Summary of the Invention

[0005] This invention provides a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, which can improve the accuracy of dynamic pedestrian 3D reconstruction.

[0006] To achieve the above objectives, this invention provides a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, comprising: The video frame sequence of the target pedestrian in the dynamic scene is acquired, and a Gaussian ellipsoid attached to the surface of the human body parameterized mesh is initialized in three-dimensional space based on the initial parameters of the target pedestrian. The attribute parameters of the Gaussian ellipsoid include position, rotation, scale, color and opacity. A pure kinematic human body mask is generated using a parameterized human body model, and a coarse overall mask containing the human body and its appendages is obtained by processing the video frame sequence using a pre-trained general segmentation model. The appendage mask is obtained by performing a difference operation between the coarse overall mask and the pure kinematic human body mask. Construct appendage nodes based on appendages, match the corresponding human parent joints based on the semantic category of the appendage nodes, and calculate the global transformation matrix of the appendages based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint. The temporal information, instance embedding features, and visual features of the target pedestrian are extracted from the video frame sequence at the current moment. The temporal information, instance embedding features, and visual features of the target pedestrian are then concatenated and input into the adaptive guidance module to obtain the conditional feature vector and reliability confidence score. The neural residual deformation field is used to predict the non-rigid residual deformation of the Gaussian ellipsoid based on the conditional feature vector. The final state of the Gaussian ellipsoid is calculated based on the non-rigid residual deformation. The Gaussian ellipsoid in its final state is input into a differentiable rasterizer to generate a 2D rendered image. The appendage mask is used as a supervision signal to constrain the rendering results of appendage nodes in the 2D rendered image. The joint loss function of the 2D rendered image and the real image is calculated, and the regularization loss term in the joint loss function is dynamically adjusted according to the reliability confidence score. The attribute parameters of the Gaussian ellipsoid are optimized through the backpropagation algorithm. Based on the optimized attribute parameters of the Gaussian ellipsoid and the global transformation matrix of the appendages, a 3D reconstructed image of the dynamic pedestrian is generated.

[0007] To address the aforementioned problems, the present invention also provides a dynamic pedestrian 3D reconstruction device based on adaptive prior fusion, the device comprising: The data processing module is used to acquire video frame sequences of target pedestrians in dynamic scenes, and initialize a Gaussian ellipsoid attached to the surface of a human body parameterized mesh in three-dimensional space based on the initial parameters of the target pedestrians. The attribute parameters of the Gaussian ellipsoid include position, rotation, scale, color, and opacity. A pure kinematic human body mask is generated using a parameterized human body model, and a coarse overall mask containing the human body and its appendages is obtained by processing the video frame sequence using a pre-trained general segmentation model. The appendage mask is obtained by performing a difference operation between the coarse overall mask and the pure kinematic human body mask. The Gaussian ellipsoid update module is used to construct appendage nodes based on appendages, match the corresponding human parent joints according to the semantic category of the appendage nodes, and calculate the global transformation matrix of the appendages based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint. It extracts the temporal information of the video frame sequence at the current moment, the instance embedding features of the target pedestrian, and the visual features of the target pedestrian. The temporal information, the instance embedding features of the target pedestrian, and the visual features of the target pedestrian are concatenated and input into the adaptive guidance module to obtain the conditional feature vector and the reliability confidence score. The neural residual deformation field is used to predict the non-rigid residual deformation of the Gaussian ellipsoid based on the conditional feature vector, and the final state of the Gaussian ellipsoid is calculated based on the non-rigid residual deformation. The 3D reconstruction image generation module is used to input the Gaussian ellipsoid in its final state into a differentiable rasterizer to generate a 2D rendered image. It uses an appendage mask as a supervision signal to perform boundary constraints on the rendering results of appendage nodes in the 2D rendered image, calculates the joint loss function of the 2D rendered image and the real image, and dynamically adjusts the regularization loss term in the joint loss function according to the reliability confidence score. It optimizes the attribute parameters of the Gaussian ellipsoid through the backpropagation algorithm, and generates a 3D reconstructed image of a dynamic pedestrian based on the optimized attribute parameters of the Gaussian ellipsoid and the global transformation matrix of the appendage.

[0008] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the aforementioned dynamic pedestrian 3D reconstruction method based on adaptive prior fusion.

[0009] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned dynamic pedestrian 3D reconstruction method based on adaptive prior fusion.

[0010] This invention initializes a Gaussian ellipsoid attached to the surface of a parameterized human body mesh in three-dimensional space based on the initial parameters of the target pedestrian, ensuring that the reconstruction result conforms to human kinematics in its overall structure. Furthermore, by performing a differential operation between the coarse overall mask and the pure kinematic human body mask, an independent accessory mask can be obtained, completely separating the human body and accessories and reducing the probability of structural confusion. Additionally, constructing a hierarchical motion dependency relationship between human skeletons and accessory nodes achieves consistency in the physical motion of accessories and human joints. Moreover, by concatenating temporal information, instance embedding features, and visual features, and inputting them into an adaptive guidance module, a conditional feature vector and a reliability confidence score are obtained. Finally, the neural residual deformation field is used to predict the non-rigidity of the Gaussian ellipsoid based on the conditional feature vector. The system employs a residual deformation method, where the neural residual deformation field captures fine structures such as clothing wrinkles and surface textures, compensating for the smoothness limitations of parametric human models. On the other hand, the reliability confidence score dynamically adjusts the guidance intensity based on the quality of the input features, effectively suppressing artifacts. Finally, the final Gaussian ellipsoid is input into a differentiable rasterizer to generate a 2D rendered image. An appendage mask is used as a supervisory signal to constrain the rendering results of appendage nodes in the 2D rendered image, calculating the joint loss function between the 2D rendered image and the real image. This allows for relaxation of regularization constraints when confidence is low, enabling the parametric human model to fit observations more freely, while strengthening regularization when confidence is high to prevent overfitting, thereby improving the accuracy of generating 3D reconstructed images of dynamic pedestrians. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating an example of a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, provided as an embodiment of the present invention. Figure 3 A schematic diagram illustrating the separation of pure kinematic human body mask and accessory mask in a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, provided as an embodiment of the present invention; Figure 4 This is a reconstruction diagram of a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, provided in an embodiment of the present invention. Figure 5 A functional block diagram of a dynamic pedestrian 3D reconstruction device based on adaptive prior fusion provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device that implements the dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, according to an embodiment of the present invention.

[0012] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0014] This application provides a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the dynamic pedestrian 3D reconstruction method based on adaptive prior fusion can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0015] Reference Figure 1 The diagram shown is a flowchart illustrating a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion according to an embodiment of the present invention. In this embodiment, the dynamic pedestrian 3D reconstruction method based on adaptive prior fusion includes: S1. Obtain the video frame sequence of the target pedestrian in the dynamic scene, and initialize the Gaussian ellipsoid attached to the human body parameterized mesh surface in three-dimensional space based on the initial parameters of the target pedestrian. The attribute parameters of the Gaussian ellipsoid include position, rotation, scale, color and opacity.

[0016] Understandably, dynamic scenarios include, but are not limited to, dynamic driving scenarios.

[0017] As is understandable, a video frame sequence refers to a continuous set of two-dimensional images containing a tracked pedestrian, captured continuously in a real dynamic scene by image acquisition devices such as multi-view camera systems or monocular cameras. Each frame of the video frame sequence records the instantaneous motion posture of the target pedestrian at a specific timestamp, and also includes visual features such as changes in lighting, occlusion relationships, and clothing wrinkles.

[0018] Understandably, human parametric meshes refer to three-dimensional surface mesh models reconstructed in three-dimensional space based on standard parametric human models (such as SMPL, Skinned Multi-Person Linear Model) to represent human anatomical structures. Human parametric meshes are usually generated by calculating fixed template vertices and faces through functions such as linear blending skinning. The parametric human model only provides a smooth geometric basis, and its surface only contains the geometric topological information of human skin, naturally excluding non-human appendage structures such as clothing, backpacks, and suitcases.

[0019] Understandably, a Gaussian ellipsoid refers to the fundamental spatial geometry used to represent 3D scenes and objects in 3D Gaussian Splatting (3DGS) technology. Each Gaussian ellipsoid is an independent, differentiable rendering unit with a set of learnable and optimizable attribute parameters, specifically including: positional attributes to determine its coordinates in 3D space; rotational attributes to control its spatial orientation; scale attributes to control its size in three mutually perpendicular dimensions; color attributes to express appearance texture; and opacity attributes to control occlusion and transparency relationships.

[0020] Understandably, the initial parameters of the target pedestrian include shape parameters and posture parameters. The shape parameters are used to describe the inherent physical characteristics of the target pedestrian, such as height and weight. The posture parameters are used to describe the overall motion state of the target pedestrian at the current moment and the rotation angle of each joint in the human skeleton.

[0021] S2. Generate a pure kinematic human body mask using a parametric human body model, and process the video frame sequence using a pre-trained general segmentation model to obtain a coarse overall mask containing the human body and its appendages. Perform a difference operation between the coarse overall mask and the pure kinematic human body mask to obtain the appendage mask.

[0022] Understandably, a parametric human body model refers to a three-dimensional geometric representation model of the human body that is generated statistically based on massive amounts of real human body scan data and driven by specific parameters. The parametric human body model can dynamically reconstruct a three-dimensional human body mesh with corresponding body shape and posture based on specific input parameters and through positive kinematic algorithms such as linear hybrid skinning.

[0023] Understandably, a pure kinematic human mask refers to a binary image generated using a parametric human model that represents only the anatomical contours of the human body on a two-dimensional image plane.

[0024] Understandably, a pre-trained general segmentation model refers to a deep learning visual neural network that has been pre-trained on a massive image dataset and possesses powerful zero-shot generalization and pixel-level segmentation capabilities. Examples include the Segment Anything Model (SAM). A pre-trained general segmentation model can receive the coordinates of a two-dimensional bounding box as input box cues and accurately retrieve and segment the foreground object region that best matches the cues space based on the visual texture and edge features of the input image, without needing to retrain for a specific scene.

[0025] Understandably, a coarse overall mask refers to the binarized segmentation result image of the foreground object output by a pre-trained general segmentation model. Since the general segmentation model mainly relies on visual texture and physical edges for image segmentation, it lacks the ability to distinguish between "human body" and "personal belongings". Therefore, the mask it generates will merge and cover the target pedestrian's body and all explicit accessories (such as backpacks, handbags, etc.) it carries.

[0026] Understandably, differential operation refers to an image processing operation at the pixel level that uses a coarse overall mask containing complete geometric information of the human body and its appendages to subtract a pure kinematic human body mask containing only the anatomical outline of the human body. Through the above differential operation, the pixels of the human body itself can be accurately separated and removed from the overall foreground area, thereby extracting the pixels of the object area outside the human body, and finally obtaining an independent "appendage mask".

[0027] Specifically, a clean kinematic human mask is generated using a parametric human model, including: The initial parameters of the target pedestrian are input into the parametric human body model to reconstruct a three-dimensional human body mesh containing geometric information of the human skin surface; Obtain the intrinsic and extrinsic parameter matrices of the camera that captures video frame sequences, and use the principle of perspective projection to project the vertices of the 3D human body mesh from the world coordinate system to the 2D image plane to obtain the projected human body mesh. The projected human body mesh is rasterized using a differentiable rasterizer renderer. Pixels falling within the human body mesh area are marked as foreground, and pixels not falling within the human body mesh area are marked as background, resulting in a binarized image. This binarized image is then used as a clean kinematic human body mask.

[0028] Specifically, the step of generating a coarse overall mask containing the human body and its appendages using a pre-trained general segmentation model includes: Obtain the 3D bounding box annotation of the target pedestrian, project the vertices of the 3D bounding box annotation onto the 2D image plane using the camera's intrinsic and extrinsic parameter matrices, and obtain the 2D bounding box by calculating the minimum bounding rectangle; The coordinates of the two-dimensional bounding box are used as box cue input into a pre-trained general segmentation model. The segmentation model is used to retrieve and segment the foreground object region that best matches the spatial range of the two-dimensional bounding box from the image features, and obtain the binarized segmentation result. The binarized segmentation result is used as the coarse overall mask.

[0029] For example, a pure kinematic human body mask is generated using a parameterized human body model, and a coarse overall mask containing the human body and appendages is obtained by processing the video frame sequence using a pre-trained general segmentation model. The appendage mask is obtained by performing a difference operation between the coarse overall mask and the pure kinematic human body mask. This can be achieved using the following implementation steps: Step 1: Obtaining Initial Parameters and Constructing a 3D Mesh: First, obtain the initial parameters of the target pedestrian in the current video frame. The initial parameters include shape parameters. and attitude parameters Inputting the initial parameters into a standard parametric human body model, the vertices of the current 3D human body mesh are reconstructed using the following formula. : ; in, For template vertices, This represents the vertex offset transformation affected by shape and orientation. Represents a linear blending skinning function. This is the skinning weight matrix.

[0030] Step 2, Spatial Projection Transformation: Obtain the camera's intrinsic parameter matrix K and extrinsic parameter matrices (rotation R and translation T) for capturing this video frame. Using the principle of perspective projection, plot the vertices of the aforementioned 3D human body mesh. Projecting from the world coordinate system onto a two-dimensional image plane: ; in This represents a perspective projection operation, which obtains the two-dimensional projection area of ​​the human body mesh onto the image plane.

[0031] Step 3, Differentiable Rasterization and Mask Generation: The projected human body mesh is rasterized using a Differentiable Renderer. For each pixel on the image plane... The algorithm determines whether a pixel is located within the projected human body mesh: if a pixel falls within the human body mesh, its pixel value is marked as 1 (foreground); if a pixel does not fall within the human body mesh, its pixel value is marked as 0 (background). This generates a binarized image with the same resolution as the original input image, which is the clean kinematic human body mask. The mask It accurately outlines the human body in its current posture, without including any pixels of non-human structures.

[0032] Furthermore, the coarse global mask provided by the dataset is utilized. The process of obtaining [the information] includes the following steps: Step 1: Box Projection The 3D bounding box annotations of the target pedestrian are read from the dynamic driving scene dataset (Waymo Open Dataset). Using the camera's intrinsic matrix K and extrinsic matrix (rotation R and translation T), the vertices of the 3D bounding box are projected onto the 2D image plane of the current video frame. The minimum bounding rectangle that can enclose all projected vertices is calculated, thus obtaining a compact 2D bounding box.

[0033] Step 2, Prompt-based Segmentation: A pre-trained general segmentation model (specifically, the Segment Anything Model, SAM) is introduced. The 2D bounding box coordinates generated in the previous steps are used as box prompts and input into the SAM encoder. Based on these prompts, the SAM model retrieves and segments the foreground object region that best matches the spatial range of the bounding box from the image features.

[0034] Step 3: Mask Generation The binarized segmentation result output by the pre-trained general segmentation model is the coarse global mask. Since the pre-trained general segmentation model is based on visual texture and edges for segmentation, the generated mask will completely cover the human body and its explicit accessories (such as backpacks, handbags, etc.) and has pixel-level high edge fitting, providing a basis with complete geometric information for subsequent decoupling operations.

[0035] Using the original annotations provided by the dataset above, an overall mask containing the human body and appendages is obtained. Using the initial parameters, a 3D human body parameterized mesh is projected onto a 2D image plane using a differentiable rasterizer, generating a pure mask containing only human anatomical structures. Subtraction is then performed to obtain the appendage mask. ; Obtain accessory mask Then, it is used as a monitoring signal to pass to subsequent modeling processes. Specifically, the accessory mask is obtained. Then, it is used as a supervision signal to pass on to subsequent modeling processes, including: during the training phase, using... Boundary constraints are applied to the rendering results of "attachment nodes" to ensure that the Gaussian model of the attachments accurately fits the object region outside the human body, thereby achieving geometric and semantic decoupling between "human body" and "object".

[0036] S3. Construct appendage nodes based on appendages, match the corresponding human parent joints based on the semantic category of the appendage nodes, and calculate the global transformation matrix of the appendages based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint.

[0037] Understandably, a learnable relative transformation matrix is ​​a mathematical matrix used to represent the static spatial offset of the center of an accessory relative to its corresponding parent joint of the human body. The learnable relative transformation matrix describes the fixed physical wearing position of the accessory in the local coordinate system of the human body, such as the fixed distance and angle of the backpack center relative to the spinal joint. In this embodiment of the invention, the learnable relative transformation matrix is ​​set as an optimization parameter in the network model, and can be set as an identity matrix or a predicted value based on the centroid of the Gaussian group during initialization.

[0038] Understandably, the joint transformation matrix of the human body's parent joint refers to the real-time pose transformation matrix of a specific joint (such as the spinal joint, wrist joint, or head joint) in the world coordinate system, which is calculated in real time by a parametric human model using a forward kinematics algorithm.

[0039] Understandably, constructing an accessory node based on an accessory means converting the accessory category (e.g., a backpack) that the accessory represents into a node.

[0040] Understandably, by constructing appendage nodes based on appendages and matching the corresponding parent joints of the human body according to the semantic category of the appendage nodes, a hierarchical motion dependency relationship between the human skeleton and appendage nodes can be constructed, including: Based on the pose and shape parameters of the target pedestrian in the current frame, the set of transformation matrices of all joints in the human skeleton in the world coordinate system is calculated through forward kinematics. Based on the semantic category of the appendage node and the preset mapping table, the most relevant joint on the human skeleton is taken as the parent node, and the corresponding parent joint transformation matrix is ​​retrieved from the transformation matrix set. The global transformation matrix of the appendage is calculated by multiplying the parent joint transformation matrix with the relative transformation rigid body matrix representing the static spatial offset of the appendage node relative to the parent joint.

[0041] Understandably, hierarchical motion dependency refers to a hierarchical attachment relationship based on kinematic chains constructed in a 3D spatial scene graph. In hierarchical motion dependency, the human skeleton is defined as the "parent node," while the attachments obtained through decoupling are defined as "child nodes." This dependency relationship ensures through mathematical transformation that the spatial motion trajectory of the child node must strictly follow the macroscopic pose changes of its corresponding parent node. For example, when the human body makes large movements such as walking, turning, or waving, the attachments attached to it will automatically and synchronously generate corresponding translations and rotations.

[0042] Further, the global transformation matrix of the appendage is calculated, including: Based on the pose and shape parameters of the target pedestrian in the current video frame, the set of transformation matrices of all joints in the human skeleton in the world coordinate system is calculated by forward kinematics. Based on the semantic category of the appendage node and the preset mapping table, the most relevant joint in the human skeleton is taken as the human parent joint, and the corresponding joint transformation matrix is ​​extracted from the transformation matrix set. The extracted joint transformation matrix is ​​multiplied by the learnable relative transformation matrix representing the static spatial offset of the appendage node relative to the human parent joint to obtain the global transformation matrix of the appendage.

[0043] For example, a hierarchical motion dependency relationship between the human skeleton and appendage nodes is constructed. Appendage nodes are constructed based on the appendages, and the corresponding human parent joints are matched according to the semantic category of the appendage nodes. The global transformation matrix of the appendages is calculated based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint. This is achieved through the following implementation steps: Step 1: Obtaining the kinematic drive and joint transformation matrix of human nodes: pose parameters based on the current frame and shape parameters The global transformation matrix of all K joints (usually K=24) in the human skeleton in the world coordinate system is calculated by forward kinematics.

[0044] No. Each joint Transformation matrix at time step The calculation is as follows: ; in, Represents the positive kinematic function. This represents the joint return position determined by shape parameters. From this, we obtain the set of joint transformation matrices describing the human motion state. .

[0045] Step 2: Parent node matching based on semantic rules: For each Accessory Node, the most relevant joint on the human skeleton is determined as the "Parent Node" based on its semantic category.

[0046] The specific definition of the "semantic rule" is a pre-defined mapping table: if the semantic category of the accessory is "backpack," then the mapping parent node is the human body's "spine joint" or "torso"; if the semantic category of the accessory is "suitcase" or "handbag," then the mapping parent node is the human body's "wrist joint"; if the semantic category of the accessory is "headwear / hat," then the mapping parent node is "head joint." According to this rule, the corresponding parent joint transformation matrix is ​​retrieved from the set calculated in step 1, denoted as... .

[0047] Step 3: Calculation and optimization of the global transformation matrix of the appendages Define appendages in Global transformation matrix at time step This matrix determines the center position and orientation of the attached Gaussian ellipsoid in world space. Its calculation formula is defined as the product of the parent joint transformation and the relative transformation: ; The parameters are obtained and defined as follows: (Parent Joint Transformation): Directly derived from the retrieved joint transformation matrix, which changes over time. Dynamic changes include not only rotation but also translation, thus giving the appendages a macroscopic trajectory that follows the movement of the human body. (Learnable relative transformation matrix): This is a 4×4 rigid body transformation matrix representing the static spatial offset of the appendage relative to the parent joint (e.g., the fixed distance and angle of the backpack center relative to the spinal joint). This matrix is ​​initialized as the identity matrix or a predicted value based on the centroid of the Gaussian group and serves as a learnable parameter of the network.

[0048] The specific process of the "end-to-end optimization" is as follows: During network training, the Gaussian points of the appendages undergo... After transformation, the image is rasterized and rendered onto the image plane. A loss function (e.g., photometric loss) is calculated between the rendered image and the real-world object mask / image. The loss function is then calculated using backpropagation. The gradients of the translation vector and rotation parameters are continuously updated using the gradient descent method. The value. After several iterations. It converges to a stable optimal value that accurately describes the physical wearing relationship between the accessory and the human body (such as "fitting against the back" or "held in the hand"), thus enabling natural following motion without human intervention in subsequent reasoning.

[0049] Understandably, for accessories, this embodiment of the invention does not use parameterized human model point-driven methods, but rather treats them as independent deformable Gaussian groups attached to the parent skeleton. Their spatial position transformations follow a rigid body kinematics chain: ;in The static offset needs optimization for automatically aligning the backpack / handbags with the body's relative position. The global transformation matrix of the appendage. This is the global transformation matrix of the parent node of the human body.

[0050] S4. Extract the time information, instance embedding features, and visual features of the target pedestrian from the video frame sequence at the current moment. Then, concatenate the time information, instance embedding features, and visual features of the target pedestrian and input them into the adaptive guidance module to obtain the conditional feature vector and reliability confidence score. Then, use the neural residual deformation field to predict the non-rigid residual deformation of the Gaussian ellipsoid based on the conditional feature vector, and calculate the final state of the Gaussian ellipsoid based on the non-rigid residual deformation.

[0051] Understandably, the adaptive guidance module refers to a dynamic feature extraction and evaluation network unit built on a multilayer perceptron (MLP). Its core function is to output two key parameters: one is a conditional feature vector used to drive fine deformation, and the other is a reliability confidence score (usually a scalar between 0 and 1) used to evaluate the accuracy of prior models such as SMPL under the current observation conditions.

[0052] Understandably, the neural residual deformation field refers to a nonlinear mapping function constructed based on a deep neural network (such as an MLP containing multiple fully connected layers and nonlinear activation functions). The neural residual deformation field takes the initial coordinate encoding of the Gaussian ellipsoid in the regular space and the conditional feature vector output by the aforementioned adaptive guidance module as input, and predicts the offset of each Gaussian ellipsoid relative to its initial state through the forward computation of the network.

[0053] Understandably, non-rigid residual deformation refers to the subtle geometric corrections obtained from the neural residual deformation field predictions that exceed rigid kinematic transformations such as linear hybrid skin (LBS). Specifically, non-rigid residual deformation manifests as a set of multidimensional offset vectors, corresponding to the position offset, rotation quaternion offset, and scale offset of the Gaussian ellipsoid, respectively.

[0054] Specifically, the non-rigid residual deformation of a Gaussian ellipsoid is predicted using a neural residual deformation field based on conditional eigenvectors, including: High-frequency position encoding is performed on the initial coordinates of the Gaussian ellipsoid in regular space; The position encoding vector and the conditional feature vector are concatenated and input into the multilayer perceptron network of the neural residual deformation field. The output is the position offset, rotation quaternion offset and scale offset of the Gaussian ellipsoid, and the non-rigid residual deformation of the Gaussian ellipsoid is obtained.

[0055] For example, the temporal information, instance embedding features, and visual features include: Temporal information (t): timestamps or normalized time codes of video frames. Instance embedding (e h `Identifier`: Defined as a learnable high-dimensional vector (a vector of length 32) uniquely corresponding to the ID of each tracked pedestrian in the scene. This vector is randomly generated during network initialization and continuously updated during training. It is used to store identity-specific information such as the pedestrian's body shape and clothing style, enabling the model to distinguish and simultaneously reconstruct multiple different pedestrians in the scene. Visual Features: The acquisition process involves inputting the current video frame image into a pre-trained image feature extractor (feature layer of a shallow convolutional neural network, CNN). Based on the human body's two-dimensional bounding box in the image, region of interest pooling (ROI pooling) or cropping is performed, outputting a feature vector describing the pedestrian's current appearance. This feature vector captures instantaneous visual information such as lighting, occlusion, and clothing wrinkles.

[0056] For example, a specific implementation of neural residual deformation prediction includes: utilizing conditional features c t Given the initial position information of the Gaussian ellipsoid, predict the non-rigid offset of each Gaussian point. The specific implementation process is as follows: Position encoding: First, for the... Initial coordinates of a Gaussian ellipsoid in canonical space Perform high-frequency positional encoding (Positional Encoding) vectors. To capture high-frequency geometric details. Feature concatenation: The position-encoded vector is then... The conditional feature vector c output in step 1 t Concatenation is performed to form a combined input vector. MLP prediction: The combined input vector is then fed into the neural residual deformation field (i.e., a multilayer perceptron network). The network contains several fully connected layers and nonlinear activation functions (such as ReLU). The last layer has no activation function and directly outputs the residual deformation of the Gaussian ellipsoid. These correspond to position offset, rotation quaternion offset, and scale offset, respectively.

[0057] For example, calculating the final state of a Gaussian ellipsoid includes: Combining the rigid LBS transform and the predicted non-rigid residuals, the first... A Gaussian ellipsoid in The final state g at time t i (t): ; in, For global transformation, This indicates the superposition of residuals in a regular space. This indicates the application of linear blending skinning and skeleton transformation.

[0058] S5. Input the Gaussian ellipsoid in its final state into a differentiable rasterizer to generate a 2D rendered image. Use the appendage mask as a supervision signal to perform boundary constraints on the rendering results of appendage nodes in the 2D rendered image. Calculate the joint loss function of the 2D rendered image and the real image. Dynamically adjust the regularization loss term in the joint loss function according to the reliability confidence score. Optimize the attribute parameters of the Gaussian ellipsoid through the backpropagation algorithm. Based on the optimized attribute parameters of the Gaussian ellipsoid and the global transformation matrix of the appendages, generate a 3D reconstructed image of the dynamic pedestrian.

[0059] Understandably, a differentiable rasterizer is an image generation engine that, in the intersection of 3D reconstruction and deep learning, can project geometric primitives in 3D space and convert them into 2D pixel images, while maintaining mathematically complete differentiability (i.e., differentiability) throughout the entire projection and color calculation process.

[0060] Understandably, the backpropagation algorithm refers to the core mathematical and computational mechanism in deep learning networks used to train models and automatically optimize various learnable parameters.

[0061] Understandably, using appendage masks as supervisory signals to constrain the rendering results of appendage nodes in a 2D rendered image can ensure that the Gaussian model of the appendages accurately fits the object region outside the human body, thereby achieving decoupling between the "human body" and the "object" in both geometry and semantics.

[0062] Specifically, in the calculation of the joint loss function of the two-dimensional rendered image and the real image, the joint loss function includes photometric loss, pose loss and regularization loss term dynamically adjusted using reliability confidence score; The photometric loss is obtained by a weighted combination of absolute value error norm and structural similarity loss, and the pixel-level and structural-level similarity between the two-dimensional rendered image and the real image is calculated using the photometric loss. The attitude loss is obtained by calculating the temporal difference of the target pedestrian's attitude parameters. The attitude loss is used to penalize abrupt attitude changes in order to ensure the temporal smoothness of the target pedestrian's human movements. Furthermore, a regularization loss term is dynamically adjusted using the reliability confidence score. The regularization loss term includes: sparsity loss and kinematic consistency loss. When the reliability confidence score is greater than or equal to the preset reliability confidence score threshold, the weight of the sparsity loss is increased, and a regularization constraint is applied to the prediction residual of the Gaussian ellipsoid. When the reliability confidence score is less than the preset reliability confidence score threshold, the weight of the kinematic consistency loss is increased. By applying kinematic constraints, including bone length loss and angle restriction loss, to the posture and shape parameters of the parameterized human model, deviations in joint rotation angle and bone length that exceed the physiological limits of the human body are penalized.

[0063] Furthermore, the formula for calculating the joint loss function is as follows: ; in, The weight coefficients are the hyperparameters of the pose loss. The weighting coefficients of the hyperparameters of the sparsity loss. The weighting coefficients are the hyperparameters of the kinematic consistency loss. This represents the reliability confidence score.

[0064] Furthermore, luminosity loss ( This includes: Physical meaning: Measuring the similarity between a rendered image and a real image in pixel color and structure, which is fundamental to ensuring visual fidelity. Solution formula: A weighted combination of L1 norm and structural similarity (SSIM) loss is used. ;in, : Represents the predicted image pixel value rendered using differentiable rasterization; : Represents the pixel value of the corresponding ground truth training image; : The weight used to balance pixel-level errors and structural-level errors (e.g., 0.2). : Calculate the structural similarity index between two images to capture the differences in human perception of texture structure.

[0065] Furthermore, attitude loss ( This includes: Physical meaning: Constraining the temporal smoothness of human movements to prevent jitter or abrupt changes between adjacent frames, ensuring natural and continuous motion. Solution formula: Calculating the second-order difference (acceleration) or first-order difference (velocity) of the human posture parameter θ in time: ;in, : The optimized SMPL attitude parameter vector at the current time t; : Attitude parameters at the previous moment; : denotes the square of the L2 norm, penalizing abrupt attitude changes.

[0066] Furthermore, the adaptive regularization term (derived from the reliability confidence score) Dynamic control), including: embodiments of the present invention utilize reliability confidence scores. Soft switching is performed between the following two types of losses: A. Sparsity loss ( ): Activation condition: when the prior is reliable This item's weight is increased. Physical meaning: Encourages neural residual deformation fields. Output sparse, small displacements; that is, "avoid making changes unless absolutely necessary, and only modify details (such as wrinkles)" to prevent unnecessary deformation caused by model overfitting. Solution formula: Prediction residuals for all Gaussian primitives. Apply L1 regularization: N: The total number of Gaussian primitives; : The positional and scale offsets of the i-th Gaussian primitive. B. Kinematic consistency loss ( Activation condition: When the prior is unreliable ( This item's weight is increased. Physical meaning: It enforces biological and physical constraints to prevent unnatural distortions of the human body structure (such as bone elongation and joint retraction) due to occlusion or miscalculation. Solution formula: Includes bone length consistency and joint angle constraints. Bone length loss Calculate the optimized bone length Compared to standard template length Deviation: ; Angle-limited loss Punishment exceeding the physiological limits of the human body Joint rotation angle : .

[0067] Furthermore, kinematic consistency constraints include: bone length constraints and joint angle limitations. ;in, These are the weighting coefficients for the bone length constraint. The weighting coefficients for joint angle constraints. To ensure that bone length remains constant during movement, Limiting joint rotation within a reasonable physiological range can prevent structural distortion of the human body in the event of prior failure.

[0068] For example, generating a 3D reconstructed image of a dynamic pedestrian based on an optimized Gaussian ellipsoid and a global transformation matrix includes: for each pixel, calculating the cumulative color according to the sorted Gaussian ellipsoid. : ;in The color is calculated using spherical harmonic functions. It is opaque, and through the backpropagation algorithm, the Gaussian properties and MLP network weights are optimized end-to-end. and the learnable relative transformation matrix of the appendages .

[0069] This invention initializes a Gaussian ellipsoid attached to the surface of a parameterized human body mesh in three-dimensional space based on the initial parameters of the target pedestrian, ensuring that the reconstruction result conforms to human kinematics in its overall structure. Furthermore, by performing a differential operation between the coarse overall mask and the pure kinematic human body mask, an independent accessory mask can be obtained, completely separating the human body and accessories and reducing the probability of structural confusion. Additionally, constructing a hierarchical motion dependency relationship between human skeletons and accessory nodes achieves consistency in the physical motion of accessories and human joints. Moreover, by concatenating temporal information, instance embedding features, and visual features, and inputting them into an adaptive guidance module, a conditional feature vector and a reliability confidence score are obtained. Finally, the neural residual deformation field is used to predict the non-rigidity of the Gaussian ellipsoid based on the conditional feature vector. The system employs a residual deformation method, where the neural residual deformation field captures fine structures such as clothing wrinkles and surface textures, compensating for the smoothness limitations of parametric human models. On the other hand, the reliability confidence score dynamically adjusts the guidance intensity based on the quality of the input features, effectively suppressing artifacts. Finally, the final Gaussian ellipsoid is input into a differentiable rasterizer to generate a 2D rendered image. An appendage mask is used as a supervisory signal to constrain the rendering results of appendage nodes in the 2D rendered image, calculating the joint loss function between the 2D rendered image and the real image. This allows for relaxation of regularization constraints when confidence is low, enabling the parametric human model to fit observations more freely, while strengthening regularization when confidence is high to prevent overfitting, thereby improving the accuracy of generating 3D reconstructed images of dynamic pedestrians.

[0070] Reference Figure 2 The diagram shown is a flowchart illustrating an example of a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, provided by an embodiment of the present invention.

[0071] Reference Figure 3The diagram shown illustrates the separation of the pure kinematic human body mask and accessory mask in a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, according to an embodiment of the present invention.

[0072] Reference Figure 4 The diagram shown is a reconstruction schematic of a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion provided in an embodiment of the present invention.

[0073] like Figure 5 The diagram shown is a functional block diagram of a dynamic pedestrian 3D reconstruction device based on adaptive prior fusion provided in an embodiment of the present invention.

[0074] The dynamic pedestrian 3D reconstruction device 100 based on adaptive prior fusion described in this invention can be installed in an electronic device. Depending on the functions implemented, the dynamic pedestrian 3D reconstruction device 100 based on adaptive prior fusion may include a data processing module 101, a Gaussian ellipsoid update module 102, and a 3D reconstruction image generation module 103.

[0075] The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0076] In this embodiment, the functions of each module / unit are as follows: The data processing module 101 is used to acquire video frame sequences of target pedestrians in a dynamic scene, and initialize a Gaussian ellipsoid attached to the surface of a parameterized human body mesh in three-dimensional space based on the initial parameters of the target pedestrians. The attribute parameters of the Gaussian ellipsoid include position, rotation, scale, color, and opacity. A pure kinematic human body mask is generated using a parameterized human body model, and a coarse overall mask containing the human body and its appendages is obtained by processing the video frame sequence using a pre-trained general segmentation model. The appendage mask is obtained by performing a difference operation between the coarse overall mask and the pure kinematic human body mask.

[0077] The Gaussian ellipsoid update module 102 is used to construct appendage nodes based on appendages, match the corresponding human parent joints based on the semantic category of the appendage nodes, and calculate the global transformation matrix of the appendages based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint. It extracts the time information of the video frame sequence at the current moment, the instance embedding features of the target pedestrian, and the visual features of the target pedestrian. After concatenating the time information, the instance embedding features of the target pedestrian, and the visual features of the target pedestrian, it inputs them into the adaptive guidance module to obtain the conditional feature vector and the reliability confidence score. It then uses the neural residual deformation field to predict the non-rigid residual deformation of the Gaussian ellipsoid based on the conditional feature vector and calculates the final state of the Gaussian ellipsoid based on the non-rigid residual deformation.

[0078] The 3D reconstructed image generation module 103 is used to input the Gaussian ellipsoid in its final state into a differentiable rasterizer to generate a 2D rendered image, and to use an appendage mask as a supervision signal to perform boundary constraints on the rendering results of appendage nodes in the 2D rendered image, calculate the joint loss function of the 2D rendered image and the real image, and dynamically adjust the regularization loss term in the joint loss function according to the reliability confidence score, optimize the attribute parameters of the Gaussian ellipsoid through the backpropagation algorithm, and generate a 3D reconstructed image of a dynamic pedestrian based on the optimized attribute parameters of the Gaussian ellipsoid and the global transformation matrix of the appendage.

[0079] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, according to an embodiment of the present invention.

[0080] The electronic device may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a dynamic pedestrian 3D reconstruction method program based on adaptive prior fusion.

[0081] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a dynamic pedestrian 3D reconstruction method program based on adaptive prior fusion) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0082] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a dynamic pedestrian 3D reconstruction method program based on adaptive prior fusion, but also to temporarily store data that has been output or will be output.

[0083] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0084] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0085] Figure 6 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 6 The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0086] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0087] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0088] The program for a dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, stored in the memory 11 of the electronic device, is a combination of multiple instructions. When run in the processor 10, it can achieve the following: The video frame sequence of the target pedestrian in the dynamic scene is acquired, and a Gaussian ellipsoid attached to the surface of the human body parameterized mesh is initialized in three-dimensional space based on the initial parameters of the target pedestrian. The attribute parameters of the Gaussian ellipsoid include position, rotation, scale, color and opacity. A pure kinematic human body mask is generated using a parameterized human body model, and a coarse overall mask containing the human body and its appendages is obtained by processing the video frame sequence using a pre-trained general segmentation model. The appendage mask is obtained by performing a difference operation between the coarse overall mask and the pure kinematic human body mask. Construct appendage nodes based on appendages, match the corresponding human parent joints based on the semantic category of the appendage nodes, and calculate the global transformation matrix of the appendages based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint. The temporal information, instance embedding features, and visual features of the target pedestrian are extracted from the video frame sequence at the current moment. The temporal information, instance embedding features, and visual features of the target pedestrian are then concatenated and input into the adaptive guidance module to obtain the conditional feature vector and reliability confidence score. The neural residual deformation field is used to predict the non-rigid residual deformation of the Gaussian ellipsoid based on the conditional feature vector. The final state of the Gaussian ellipsoid is calculated based on the non-rigid residual deformation. The Gaussian ellipsoid in its final state is input into a differentiable rasterizer to generate a 2D rendered image. The appendage mask is used as a supervision signal to constrain the rendering results of appendage nodes in the 2D rendered image. The joint loss function of the 2D rendered image and the real image is calculated, and the regularization loss term in the joint loss function is dynamically adjusted according to the reliability confidence score. The attribute parameters of the Gaussian ellipsoid are optimized through the backpropagation algorithm. Based on the optimized attribute parameters of the Gaussian ellipsoid and the global transformation matrix of the appendages, a 3D reconstructed image of the dynamic pedestrian is generated.

[0089] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0090] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0091] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: The video frame sequence of the target pedestrian in the dynamic scene is acquired, and a Gaussian ellipsoid attached to the surface of the human body parameterized mesh is initialized in three-dimensional space based on the initial parameters of the target pedestrian. The attribute parameters of the Gaussian ellipsoid include position, rotation, scale, color and opacity. A pure kinematic human body mask is generated using a parameterized human body model, and a coarse overall mask containing the human body and its appendages is obtained by processing the video frame sequence using a pre-trained general segmentation model. The appendage mask is obtained by performing a difference operation between the coarse overall mask and the pure kinematic human body mask. Construct appendage nodes based on appendages, match the corresponding human parent joints based on the semantic category of the appendage nodes, and calculate the global transformation matrix of the appendages based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint. The temporal information, instance embedding features, and visual features of the target pedestrian are extracted from the video frame sequence at the current moment. The temporal information, instance embedding features, and visual features of the target pedestrian are then concatenated and input into the adaptive guidance module to obtain the conditional feature vector and reliability confidence score. The neural residual deformation field is used to predict the non-rigid residual deformation of the Gaussian ellipsoid based on the conditional feature vector. The final state of the Gaussian ellipsoid is calculated based on the non-rigid residual deformation. The Gaussian ellipsoid in its final state is input into a differentiable rasterizer to generate a 2D rendered image. The appendage mask is used as a supervision signal to constrain the rendering results of appendage nodes in the 2D rendered image. The joint loss function of the 2D rendered image and the real image is calculated, and the regularization loss term in the joint loss function is dynamically adjusted according to the reliability confidence score. The attribute parameters of the Gaussian ellipsoid are optimized through the backpropagation algorithm. Based on the optimized attribute parameters of the Gaussian ellipsoid and the global transformation matrix of the appendages, a 3D reconstructed image of the dynamic pedestrian is generated.

[0092] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0093] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0094] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0095] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0096] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0097] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0098] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0099] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A dynamic pedestrian 3D reconstruction method based on adaptive prior fusion, characterized in that, The method includes: The video frame sequence of the target pedestrian in the dynamic scene is acquired, and a Gaussian ellipsoid attached to the surface of the human body parameterized mesh is initialized in three-dimensional space based on the initial parameters of the target pedestrian. The attribute parameters of the Gaussian ellipsoid include position, rotation, scale, color and opacity. A pure kinematic human body mask is generated using a parameterized human body model, and a coarse overall mask containing the human body and its appendages is obtained by processing the video frame sequence using a pre-trained general segmentation model. The appendage mask is obtained by performing a difference operation between the coarse overall mask and the pure kinematic human body mask. Construct appendage nodes based on appendages, match the corresponding human parent joints based on the semantic category of the appendage nodes, and calculate the global transformation matrix of the appendages based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint. The temporal information, instance embedding features, and visual features of the target pedestrian are extracted from the video frame sequence at the current moment. The temporal information, instance embedding features, and visual features of the target pedestrian are then concatenated and input into the adaptive guidance module to obtain the conditional feature vector and reliability confidence score. The neural residual deformation field is used to predict the non-rigid residual deformation of the Gaussian ellipsoid based on the conditional feature vector. The final state of the Gaussian ellipsoid is calculated based on the non-rigid residual deformation. The Gaussian ellipsoid in its final state is input into a differentiable rasterizer to generate a 2D rendered image. The appendage mask is used as a supervision signal to constrain the rendering results of appendage nodes in the 2D rendered image. The joint loss function of the 2D rendered image and the real image is calculated, and the regularization loss term in the joint loss function is dynamically adjusted according to the reliability confidence score. The attribute parameters of the Gaussian ellipsoid are optimized through the backpropagation algorithm. Based on the optimized attribute parameters of the Gaussian ellipsoid and the global transformation matrix of the appendages, a 3D reconstructed image of the dynamic pedestrian is generated.

2. The dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in claim 1, characterized in that, The method of generating a pure kinematic human mask using a parametric human model includes: The initial parameters of the target pedestrian are input into the parametric human body model to reconstruct a three-dimensional human body mesh containing geometric information of the human skin surface; Obtain the intrinsic and extrinsic parameter matrices of the camera that captures video frame sequences, and use the principle of perspective projection to project the vertices of the 3D human body mesh from the world coordinate system to the 2D image plane to obtain the projected human body mesh. The projected human body mesh is rasterized using a differentiable rasterizer renderer. Pixels falling within the human body mesh area are marked as foreground, and pixels not falling within the human body mesh area are marked as background, resulting in a binarized image. This binarized image is then used as a clean kinematic human body mask.

3. The dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in claim 1, characterized in that, The method of generating a coarse overall mask containing the human body and its appendages using a pre-trained general segmentation model includes: Obtain the 3D bounding box annotation of the target pedestrian, project the vertices of the 3D bounding box annotation onto the 2D image plane using the camera's intrinsic and extrinsic parameter matrices, and obtain the 2D bounding box by calculating the minimum bounding rectangle; The coordinates of the two-dimensional bounding box are used as box cue input into a pre-trained general segmentation model. The segmentation model is used to retrieve and segment the foreground object region that best matches the spatial range of the two-dimensional bounding box from the image features, and obtain the binarized segmentation result. The binarized segmentation result is used as the coarse overall mask.

4. The dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in claim 1, characterized in that, The global transformation matrix of the computational appendage includes: Based on the pose and shape parameters of the target pedestrian in the current video frame, the set of transformation matrices of all joints in the human skeleton in the world coordinate system is calculated by forward kinematics. Based on the semantic category of the appendage node and the preset mapping table, the most relevant joint in the human skeleton is taken as the human parent joint, and the corresponding joint transformation matrix is ​​extracted from the transformation matrix set. The extracted joint transformation matrix is ​​multiplied by the learnable relative transformation matrix representing the static spatial offset of the appendage node relative to the human parent joint to obtain the global transformation matrix of the appendage.

5. The dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in claim 1, characterized in that, The method of predicting the non-rigid residual deformation of a Gaussian ellipsoid based on conditional eigenvectors using neural residual deformation fields includes: High-frequency position encoding is performed on the initial coordinates of the Gaussian ellipsoid in regular space; The position encoding vector and the conditional feature vector are concatenated and input into the multilayer perceptron network of the neural residual deformation field. The output is the position offset, rotation quaternion offset and scale offset of the Gaussian ellipsoid, and the non-rigid residual deformation of the Gaussian ellipsoid is obtained.

6. The dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in claim 1, characterized in that, The joint loss function for calculating the two-dimensional rendered image and the real image includes photometric loss, pose loss, and a regularization loss term dynamically adjusted using the reliability confidence score. The photometric loss is obtained by a weighted combination of absolute value error norm and structural similarity loss, and the pixel-level and structural-level similarity between the two-dimensional rendered image and the real image is calculated using the photometric loss. The posture loss is obtained by calculating the temporal difference of the target pedestrian's posture parameters. The posture loss is used to penalize drastic posture changes in order to ensure the temporal smoothness of the target pedestrian's human movements.

7. The dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in claim 6, characterized in that, The regularization loss term, which is dynamically adjusted using the reliability confidence score, includes: sparsity loss and kinematic consistency loss. When the reliability confidence score is greater than or equal to the preset reliability confidence score threshold, the weight of the sparsity loss is increased, and a regularization constraint is applied to the prediction residual of the Gaussian ellipsoid. When the reliability confidence score is less than the preset reliability confidence score threshold, the weight of the kinematic consistency loss is increased. By applying kinematic constraints, including bone length loss and angle restriction loss, to the posture and shape parameters of the parameterized human model, deviations in joint rotation angle and bone length that exceed the physiological limits of the human body are penalized.

8. A dynamic pedestrian 3D reconstruction device based on adaptive prior fusion, characterized in that, The apparatus is used to implement the dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in any one of claims 1 to 7, the apparatus comprising: The data processing module is used to acquire video frame sequences of target pedestrians in dynamic scenes, and initialize a Gaussian ellipsoid attached to the surface of a human body parameterized mesh in three-dimensional space based on the initial parameters of the target pedestrians. The attribute parameters of the Gaussian ellipsoid include position, rotation, scale, color, and opacity. A pure kinematic human body mask is generated using a parameterized human body model, and a coarse overall mask containing the human body and its appendages is obtained by processing the video frame sequence using a pre-trained general segmentation model. The appendage mask is obtained by performing a difference operation between the coarse overall mask and the pure kinematic human body mask. The Gaussian ellipsoid update module is used to construct appendage nodes based on appendages, match the corresponding human parent joints according to the semantic category of the appendage nodes, and calculate the global transformation matrix of the appendages based on the joint transformation matrix of the human parent joint and the learnable relative transformation matrix of the appendage nodes relative to the human parent joint. It extracts the temporal information of the video frame sequence at the current moment, the instance embedding features of the target pedestrian, and the visual features of the target pedestrian. The temporal information, the instance embedding features of the target pedestrian, and the visual features of the target pedestrian are concatenated and input into the adaptive guidance module to obtain the conditional feature vector and the reliability confidence score. The neural residual deformation field is used to predict the non-rigid residual deformation of the Gaussian ellipsoid based on the conditional feature vector, and the final state of the Gaussian ellipsoid is calculated based on the non-rigid residual deformation. The 3D reconstruction image generation module is used to input the Gaussian ellipsoid in its final state into a differentiable rasterizer to generate a 2D rendered image. It uses an appendage mask as a supervision signal to perform boundary constraints on the rendering results of appendage nodes in the 2D rendered image, calculates the joint loss function of the 2D rendered image and the real image, and dynamically adjusts the regularization loss term in the joint loss function according to the reliability confidence score. It optimizes the attribute parameters of the Gaussian ellipsoid through the backpropagation algorithm, and generates a 3D reconstructed image of a dynamic pedestrian based on the optimized attribute parameters of the Gaussian ellipsoid and the global transformation matrix of the appendage.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the dynamic pedestrian 3D reconstruction method based on adaptive prior fusion as described in any one of claims 1 to 7.

Images