A multi-view 3D skeleton real-time reconstruction method and storage medium

Abstract

The present invention relates to a multi-perspective 3D skeleton real-time reconstruction method and storage medium. The method comprises the following steps: acquiring skeleton data using an RGB camera array to obtain a video stream; preprocessing the video frames of the currently acquired video stream, wherein the preprocessing includes optical flow-guided alignment of the current video frames; inputting the preprocessed current video frames into a spatiotemporal feature pyramid network for processing and outputting a multi-scale dynamic feature map; and inputting the multi-scale dynamic feature map into a physically constrained neural radiation field for joint optimization to obtain a multi-perspective 3D skeleton model. Dynamic details are restored by combining optical flow-guided alignment technology with a spatiotemporal feature pyramid network, overcoming the problem of low-light motion blur. 3D skeleton reconstruction is achieved at low cost, with high precision, and in real time using the hardware configuration of a pure RGB camera or video camera.

Description

Technical Field

[0001] The present application relates to the field of computer vision and artificial intelligence technology, and specifically to a multi-view 3D skeleton real-time reconstruction method and storage medium. Background Art

[0002] 3D skeleton reconstruction is the process of recovering the three-dimensional structure and morphology of human or animal bones from medical images (such as CT, MRI, and X-rays) or other data sources. It has a wide range of applications in medical diagnosis, surgical planning, rehabilitation engineering, motion analysis, virtual reality, and animation.

[0003] However, existing 3D bone reconstruction technology has the following problems:

[0004] 1. Low-light motion blur: When the camera or camcorder is in scenes below 10 lux, the reduced shutter speed causes motion blur, resulting in a positioning error of more than 15 pixels in the key points of the human skeleton when the gesture recognition model (such as YOLO) is used.

[0005] 2. Lack of physical rationality: Existing methods do not model the physical constraints of skeletal movement, and the incidence of abnormal postures (such as ankle inversion >30°) is >35%. Summary of the Invention

[0006] In view of the above problems, the present application provides a multi-perspective 3D skeleton real-time reconstruction method and storage medium to solve the problem that the existing 3D skeleton reconstruction technology uses low-cost cameras or video cameras, which causes motion blur due to the low shutter speed, and then leads to large errors in the positioning of skeleton key points in the posture recognition model.

[0007] To achieve the above object, the inventors provide a multi-view 3D skeleton real-time reconstruction method, comprising the following steps:

[0008] The video stream is obtained by collecting skeleton data through the RGB camera array;

[0009] Preprocessing the video frames of the currently acquired video stream, wherein the preprocessing includes performing optical flow guided alignment on the current video frames;

[0010] The pre-processed current video frame is input into the spatiotemporal feature pyramid network for processing and then a multi-scale dynamic feature map is output;

[0011] The multi-scale dynamic feature map is input into the physically constrained neural radiation field for joint optimization to obtain a multi-view 3D skeleton model.

[0012] In some embodiments, performing optical flow guided alignment on the current video frame comprises the following steps:

[0013] Calculate the optical flow field of adjacent video frames of the video stream through the lightweight dense optical flow field algorithm;

[0014] The dynamic region features of the current video frame are aligned through bilinear interpolation Warp operation.

[0015] In some embodiments, performing low-light enhancement on the current video frame includes:

[0016] Based on the end-to-end U-Net model, Poisson noise is injected into the current video frame during training to simulate dark light scenes.

[0017] In some embodiments, the step of inputting the multi-scale dynamic feature map into the physically constrained neural radiation field for joint optimization to obtain a multi-view 3D skeleton model specifically includes the following steps:

[0018] The multi-scale dynamic feature map is input into the skeleton branch and the radiation field branch of the physical constraint neural radiation field respectively;

[0019] The skeleton branch uses the encoder to calculate multi-head self-attention to capture joint spatial associations, and outputs 3D skeleton coordinates through the decoder. The calculated 3D skeleton coordinates are input into the physical operation engine to calculate the joint torque error and verify the output 3D skeleton coordinates.

[0020] The radiance field branch uses micro-NeRF with depthwise separable convolution and sparse tensor core acceleration to perform volume density and color prediction;

[0021] The prediction results of skeletal branches and radiation field beams are jointly optimized by embedding loss functions based on joint angle soft constraints and torque balance constraints to obtain a multi-view 3D skeletal model.

[0022] In some embodiments, the step of inputting the pre-processed current video frame into the spatiotemporal feature pyramid network for processing and then outputting a multi-scale dynamic feature map specifically includes the following steps:

[0023] Output 3 layers of feature maps through the lightweight ResNet-34 backbone network;

[0024] Capture cross-frame dynamic information through the spatiotemporal convolution layer with 3D convolution embedded in each layer, and output a three-level spatiotemporal feature map;

[0025] Generate dynamic masks;

[0026] The SE-Net structure is applied to each level of spatiotemporal feature map, and the channel weights are adjusted through dynamic masks to generate multi-scale dynamic feature maps.

[0027] Another technical solution is provided, a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are performed:

[0028] The RGB camera array is used to collect skeleton data to obtain a video stream;

[0029] Preprocessing the video frames of the currently acquired video stream, wherein the preprocessing includes performing optical flow guided alignment on the current video frames;

[0030] The pre-processed current video frame is input into the spatiotemporal feature pyramid network for processing and then a multi-scale dynamic feature map is output;

[0031] The multi-scale dynamic feature map is input into the physically constrained neural radiation field for joint optimization to obtain a multi-view 3D skeleton model.

[0032] In some embodiments, performing optical flow guided alignment on the current video frame comprises the following steps:

[0033] Calculate the optical flow field of adjacent video frames of the video stream through the lightweight dense optical flow field algorithm;

[0034] The dynamic region features of the current video frame are aligned through bilinear interpolation Warp operation.

[0035] In some embodiments, performing low-light enhancement on the current video frame includes:

[0036] Based on the end-to-end U-Net model, Poisson noise is injected into the current video frame during training to simulate dark light scenes.

[0037] In some embodiments, the step of inputting the multi-scale dynamic feature map into the physically constrained neural radiation field for joint optimization to obtain a multi-view 3D skeleton model specifically includes the following steps:

[0038] The multi-scale dynamic feature map is input into the skeleton branch and the radiation field branch of the physical constraint neural radiation field respectively;

[0039] The skeleton branch uses the encoder to calculate multi-head self-attention to capture joint spatial associations, and outputs 3D skeleton coordinates through the decoder. The calculated 3D skeleton coordinates are input into the physical operation engine to calculate the joint torque error and verify the output 3D skeleton coordinates.

[0040] The radiance field branch uses micro-NeRF with depthwise separable convolution and sparse tensor core acceleration to perform volume density and color prediction;

[0041] The prediction results of skeletal branches and radiation field beams are jointly optimized by embedding loss functions based on joint angle soft constraints and torque balance constraints to obtain a multi-view 3D skeletal model.

[0042] In some embodiments, the step of inputting the pre-processed current video frame into the spatiotemporal feature pyramid network for processing and then outputting a multi-scale dynamic feature map specifically includes the following steps:

[0043] Output 3 layers of feature maps through the lightweight ResNet-34 backbone network;

[0044] Capture cross-frame dynamic information through the spatiotemporal convolution layer with 3D convolution embedded in each layer, and output a three-level spatiotemporal feature map;

[0045] Generate dynamic masks;

[0046] The SE-Net structure is applied to each level of spatiotemporal feature map, and the channel weights are adjusted through dynamic masks to generate multi-scale dynamic feature maps.

[0047] Different from existing technologies, the above technical solution, when performing 3D skeleton reconstruction, collects skeleton data through an RGB camera array to obtain a video stream, performs optical flow-guided alignment on the video frames in the currently collected video stream, reduces motion blur and dislocation through optical flow-guided alignment, and enhances the video frames in low light conditions. The pre-processed video frames are then input into a spatiotemporal feature pyramid network for processing to obtain a multi-scale dynamic feature map. The multi-scale dynamic feature map is then input into a physically constrained neural radiation field for joint optimization to obtain a multi-view 3D skeleton model. Dynamic details are restored through optical flow-guided alignment technology combined with a spatiotemporal feature pyramid network, overcoming the problem of low-light motion blur. 3D skeleton reconstruction is achieved at low cost, with high precision, and in real time using the hardware configuration of a pure RGB camera or video camera.

[0048] The above-mentioned records related to the content of the invention are only an overview of the technical solution of this application. In order to enable ordinary technicians in this field to understand the technical solution of this application more clearly, and then implement it according to the text of the specification and the contents recorded in the drawings, and to make the above-mentioned purposes and other purposes, features and advantages of this application easier to understand, the following is an explanation in combination with the specific implementation methods and drawings of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, characteristics and effects of the specific embodiments of this application and other related contents, and are not to be considered as limiting this application.

[0050] In the drawings of the specification:

[0051] Figure 1 A schematic structural diagram of the multi-view 3D skeleton real-time reconstruction method described in the specific implementation method;

[0052] Figure 2 A schematic diagram of a process of pretreatment according to a specific embodiment;

[0053] Figure 3 A schematic diagram of a process for performing optical flow guided alignment on a current video frame according to a specific implementation method;

[0054] Figure 4 Schematic diagram of a dual-branch architecture for physically constraining neural radiation fields according to a specific embodiment;

[0055] Figure 5 A schematic diagram of a process for implementing hardware acceleration in the multi-view 3D skeleton real-time reconstruction method described in the specific implementation method;

[0056] Figure 6 A schematic diagram of a process for automatically adjusting the NeRF sampling rate in the multi-view 3D skeleton real-time reconstruction method described in the specific embodiment;

[0057] Figure 7 A structural diagram of the storage medium described in the specific implementation method.

[0058] The reference numerals in the above drawings are described as follows:

[0059] 710. Storage medium,

[0060] 720, processor. DETAILED DESCRIPTION

[0061] In order to explain in detail the possible application scenarios, technical principles, specific solutions that can be implemented, and the purpose and effects of this application, the following is a detailed description of the specific embodiments listed in conjunction with the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this application and are therefore only examples and are not intended to limit the scope of protection of this application.

[0062] References to "embodiments" herein mean that the specific features, structures, or characteristics described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the word "embodiment" in various places in the specification does not necessarily refer to the same embodiment, nor does it particularly limit its independence or relevance to other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the various technical features mentioned in the embodiments can be combined in any manner to form a corresponding implementable technical solution.

[0063] Unless otherwise defined, the technical terms used herein have the same meanings as those generally understood by those skilled in the art to which this application belongs; the use of relevant terms herein is only for describing specific embodiments and is not intended to limit this application.

[0064] In the description of this application, the term "and / or" is used to describe a logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and both A and B exist. In addition, the character " / " in this document generally indicates that the objects before and after are in a logical "or" relationship.

[0065] In this application, terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, priority or sequence relationship between these entities or operations.

[0066] Without further limitations, in this application, the words "include", "comprise", "have" or other similar expressions used in the sentences are intended to cover non-exclusive inclusion. These expressions do not exclude the presence of additional elements in the process, method or product including the elements, so that the process, method or product including a series of elements may include not only those defined elements, but also other elements not explicitly listed, or elements inherent to such process, method or product.

[0067] Consistent with the understanding in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceed" are understood to exclude the number itself; expressions such as "above," "below," and "within" are understood to include the number itself. Furthermore, in the description of the embodiments of this application, "multiple" means two or more (including two), and similar expressions related to "multiple," such as "multiple groups" and "multiple times," are also understood in this manner, unless otherwise specifically defined.

[0068] In the description of the embodiments of the present application, the space-related expressions used, such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "vertical", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicate the orientation or position relationship based on the orientation or position relationship shown in the specific embodiments or drawings, and are only for the convenience of describing the specific embodiments of the present application or facilitating the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, it should not be understood as a limitation on the embodiments of the present application.

[0069] Unless otherwise expressly specified or limited, in the description of the embodiments of the present application, the terms "installed", "connected", "connected", "fixed", "set", etc. used should be understood in a broad sense. For example, the "connection" can be a fixed connection, a detachable connection, or an integrated setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For those skilled in the art of the present application, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0070] See also Figure 1-2 This embodiment provides a multi-view 3D skeleton real-time reconstruction method, including the following steps:

[0071] Step S110: collecting skeleton data through an RGB camera array to obtain a video stream;

[0072] Step S120: preprocessing the video frames of the currently acquired video stream, wherein the preprocessing includes performing optical flow guided alignment on the current video frames;

[0073] Step S130: Input the pre-processed current video frame into the spatiotemporal feature pyramid network for processing and output a multi-scale dynamic feature map;

[0074] Step S140: Input the multi-scale dynamic feature map into the physical constraint neural radiation field for joint optimization to obtain a multi-view 3D skeleton model.

[0075] When performing 3D skeleton reconstruction, skeletal data is collected through an RGB camera array to generate a video stream. Optical flow-guided alignment is then performed on the video frames in the currently captured video stream to reduce motion blur and misalignment, and the video frames are enhanced in low-light conditions. This is then fed into a spatiotemporal feature pyramid network with preprocessed video frames to generate a multi-scale dynamic feature map. This multi-scale dynamic feature map is then fed into a physically constrained neural radiation field for joint optimization, resulting in a multi-view 3D skeleton model. Dynamic details are restored through optical flow-guided alignment combined with the spatiotemporal feature pyramid network, overcoming the challenge of low-light motion blur. This allows for low-cost, high-precision, and real-time 3D skeleton reconstruction using pure RGB camera or video camera hardware.

[0076] See also Figure 3 In some embodiments, performing optical flow guided alignment on the current video frame comprises the following steps:

[0077] Step S310: Calculating the optical flow fields of adjacent video frames of the video stream using a lightweight dense optical flow algorithm;

[0078] Step S320: Align the dynamic region features of the current video frame through a bilinear interpolation Warp operation.

[0079] Use the lightweight RAFT optical flow algorithm (parameter compression to 12M) to calculate the optical flow field of adjacent frames , align dynamic region features through bilinear interpolation Warp operation. Recover low-light motion blur region details through dynamic feature alignment guided by optical flow, the formula is , reducing motion blur dislocation.

[0080] In some embodiments, performing low-light enhancement on the current video frame includes:

[0081] An end-to-end U-Net-based model injects Poisson noise into the current video frame during training to simulate low-light scenes. This model improves SSIM by 0.32 and PSNR by 9dB. SSIM (Structural Similarity) is a metric that measures the similarity between two images. PSNR (Peak Signal-to-Noise Ratio) is an engineering term that represents the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy. Because many signals have a very wide dynamic range, the peak signal-to-noise ratio is often expressed in logarithmic decibels.

[0082] In some embodiments, the step of inputting the multi-scale dynamic feature map into the physically constrained neural radiation field for joint optimization to obtain a multi-view 3D skeleton model specifically includes the following steps:

[0083] The multi-scale dynamic feature map is input into the skeleton branch and the radiation field branch of the physical constraint neural radiation field respectively;

[0084] The skeleton branch uses the encoder to calculate multi-head self-attention to capture joint spatial associations, and outputs 3D skeleton coordinates through the decoder. The calculated 3D skeleton coordinates are input into the physical operation engine to calculate the joint torque error and verify the output 3D skeleton coordinates.

[0085] The radiance field branch uses micro-NeRF with depthwise separable convolution and sparse tensor core acceleration to perform volume density and color prediction;

[0086] The prediction results of skeletal branches and radiation field beams are jointly optimized by embedding loss functions based on joint angle soft constraints and torque balance constraints to obtain a multi-view 3D skeletal model.

[0087] like Figure 4The dual-branch architecture of the physical constraint neural radiation field shown in the figure has a 2-layer Transformer encoder and a 3-layer 256-layer MLP decoder in the skeleton branch. The encoder calculates multi-head self-attention (8 heads, dimension 64) to capture the spatial correlation of joints, and the decoder outputs 3D skeleton coordinates. , perform 3D skeleton prediction, and then convert the 3D skeleton coordinates Input into NVIDIAPhysX engine to calculate joint torque error , for 3D bone coordinates The radiation field branch uses micro NeRF (parameter count: 0.2M) combined with sparse tensor core acceleration to perform volume density and color prediction: , which uses position encoding (10-level sine) + direction encoding (4-level sine) to improve the capture of high-frequency details, and uses the RTX4090 sparse tensor core to dynamically crop voxels, reducing the amount of calculation by 60%. Combining the joint angle soft constraint and the torque balance constraint to embed the loss function, the loss function is obtained:

[0088] ;

[0089] in,

[0090] : The set loss of 3D skeleton coordinate reconstruction is calculated as follows:

[0091] ;

[0092] in, Indicates the number of 3D skeleton key points (integer), such as =17, indicating that 17 key bone points of the human body are selected to calculate the geometric loss of 3D reconstruction bone coordinates; represents the joint weight factor (torso joint = 1.5, end joint = 0.8); Represents the L2 norm, that is, the Euclidean norm, where the Euclidean distance between the predicted 3D skeleton coordinates and the true coordinates is calculated; Represents the predicted 3D bone coordinates (i-th bone point); represents the real 3D joint coordinates (the i-th bone point, generated by multi-view triangulation);

[0093] : Soft constraint of joint angle (e.g. knee flexion 10°~160°, Huber loss);

[0094] : Moment balance constraint ( is the L2 norm).

[0095] represents the rendering loss, which is defined as: , which means measuring the pixel-level error between the RGB image rendered by the neural radiance field (PC-NeRF) and the real image of the input video frame, and calculating the pixel-by-pixel difference through the L2 norm; its function is to ensure that the projection of the reconstructed 3D skeleton model under multiple perspectives is consistent with the input video stream, and to assist in optimizing the volume density and color prediction of the neural radiance field, thereby improving the geometric accuracy and visual realism of 3D reconstruction.

[0096] Traditional NeRF (such as Mip-NeRF 360) focuses on rendering static scenes and cannot handle the physical plausibility of dynamic skeletons. Physical constraints are usually implemented through post-processing (such as IK inverse kinematics), decoupled from the reconstruction model, resulting in increased latency (>30ms).

[0097] The physical constraint neural radiation field adopts a two-branch joint optimization:

[0098] Skeleton branch: The Transformer encoder captures the spatial relationship of joints, and the MLP decoder outputs 3D coordinates and then connects to the PhysX engine in real time to calculate the torque error. .

[0099] Radiation field branch: Micro NeRF (0.2M parameters) uses depthwise separable convolution and is accelerated by Sparse Tensor Cores.

[0100] Physical constraint embedding loss function: kinematic constraint (joint angle Huber loss) + dynamic constraint (torque L2 norm), which significantly improves physical compliance.

[0101] Compared with existing cases: Facebook's DensePose only outputs 2D surfaces without physical constraints; the physically constrained neural radiation field realizes dual modeling of geometry + physics.

[0102] In some embodiments, the step of inputting the pre-processed current video frame into the spatiotemporal feature pyramid network for processing and then outputting a multi-scale dynamic feature map specifically includes the following steps:

[0103] Output 3 layers of feature maps through the lightweight ResNet-34 backbone network;

[0104] Capture cross-frame dynamic information through the spatiotemporal convolution layer with 3D convolution embedded in each layer, and output a three-level spatiotemporal feature map;

[0105] Generate dynamic masks;

[0106] The SE-Net structure is applied to each level of spatiotemporal feature map, and the channel weights are adjusted through dynamic masks to generate multi-scale dynamic feature maps.

[0107] The structure of the spatiotemporal feature pyramid network is a three-scale pyramid:

[0108] Backbone network: Use lightweight ResNet-34 to output 3 layers of feature maps (resolution 1 / 4, 1 / 8, 1 / 16, number of channels [64, 128, 256]).

[0109] Spatiotemporal convolution layer: Each layer embeds 3D convolution (kernel size 3×3×3, step size 1×2×2), captures cross-frame dynamic information, and outputs spatiotemporal features .

[0110] Dynamic mask generation:

[0111] ;

[0112] in is the optical flow amplitude map, It is Sigmoid activation, and the area with mask value > 0.5 is considered as a moving target.

[0113] Channel attention enhancement: Apply the SE-Net structure in each level of feature map and adjust the channel weights through dynamic masks:

[0114] ;

[0115] Output: Multi-scale dynamic feature maps and 2D skeleton point heatmaps (an improved version of YOLOv8-Pose, using a spatiotemporal feature pyramid as input). Based on the motion target mask, a weighted summation is performed on the feature maps of each channel to enhance the representation of dynamic region features while suppressing static background interference. This effectively highlights key skeletal motion information and provides high-quality feature input for subsequent 2D skeleton point heatmap generation and 3D reconstruction.

[0116] Traditional multi-view 3D reconstruction often uses feature pyramid networks (FPNs) (such as Mask R-CNN) to extract multi-scale features, but lacks specific processing capabilities for motion blur. Low-light motion blur typically relies on event cameras (such as DAVIS346) or multispectral sensors (such as infrared + RGB), which are expensive hardware.

[0117] This method uses optical flow-guided dynamic feature decoupling: it fuses dense optical flow (RAFT algorithm) with feature pyramids and uses motion mask weighted channel attention (SE-Net) to separate moving objects from static backgrounds. The improved quantization performance significantly surpasses traditional FPN.

[0118] Hierarchical design of spatiotemporal convolution: 3D convolution (3×3×3 kernel) is embedded in each layer of the pyramid to capture dynamic information across frames, significantly improving the recall rate of dynamic features.

[0119] See also Figure 5 , in some embodiments, also includes hardware acceleration:

[0120] FP16 mixed precision: Utilizes the RTX 4090's Tensor Cores for half-precision matrix operations, halving the memory bandwidth requirements and increasing computational throughput to twice that of FP32.

[0121] INT8 quantization: Model weights are dynamically quantized to 8-bit integers, maintaining accuracy through the following process:

[0122] -Calibration phase: statistical activation layer distribution range;

[0123] -Quantitative Reasoning: , S is the scaling factor;

[0124] - Dequantization reconstruction: restore to FP16 when output.

[0125] Quantized deployment: After the model is INT8 quantized, the inference speed is doubled (from 4.5ms / frame to 2.1ms / frame), and the end-to-end latency is less than 20ms.

[0126] Sparse voxel rendering: based on NVIDIA Sparse Tensor Core (Sparse Tensor Core) implementation:

[0127] -Build dynamic airspace mask: ;

[0128] -Only for Ray sampling is performed on voxels, and invalid areas are skipped.

[0129] In some embodiments, it also includes automatic adjustment of NeRF sampling rate, dynamically reducing the number of NeRF sampling points to reduce the amount of calculation while keeping the joint error less than 5mm, which can reduce the amount of calculation by up to 15%. Figure 6 , the automatic adjustment of NeRF sampling rate is achieved as follows:

[0130] 1. Monitoring indicators: Real-time calculation of the average position error of the skeleton joints in the current frame :

[0131] (Unit: mm)

[0132] 2. Trigger conditions:

[0133] When the error is less than 5mm, the system considers that the current reconstruction accuracy is sufficient and can appropriately reduce the sampling rate to save computing resources;

[0134] when When the diameter is ≥5mm, the basic sampling rate (128 points / ray) is maintained to ensure accuracy;

[0135] 3. Formula for dynamic adjustment of sampling points:

[0136]

[0137] in,

[0138] is the base sampling rate (default 128 points / ray);

[0139] is the maximum downsampling ratio (value is 0.15), which means that the maximum number of sampling points can be reduced by 15%;

[0140] is the error sensitivity coefficient (value 0.2), which controls the sensitivity of the downsampling amplitude to the change of error;

[0141] is the average position error of the skeleton joints in the current frame.

[0142] 4. Formula behavior analysis:

[0143] When the error As it approaches 0, the exponential term approaches 1, at which point the sampling rate is reduced to its maximum: =128×(1−0.15)=108.8 (approximately 109 points);

[0144] When the error As it increases, the exponential term decays rapidly, resulting in Approaching 0, the sampling rate is close to the base sampling rate of 128;

[0145] when = 5, the formula no longer downsamples (because the trigger condition requires <5, downsampling is performed), so 128 is used directly at this time.

[0146] 5. Boundary Protection:

[0147] In actual implementation, a lower limit for the sampling rate (for example, not less than 100 points / ray) and rounding to the nearest integer are set.

[0148] 6. Effect:

[0149] In low-motion scenarios (small errors), the number of sampling points can be reduced by about 15%, and the amount of calculation is reduced accordingly.

[0150] Experimental data shows that when downsampling is 15%, the reconstruction error only increases by 0.3mm (from 6.8mm to 7.1mm), still meeting the application requirement of a 5mm error threshold.

[0151] Comparison between spatiotemporal feature pyramid network (ST-FPN) and traditional FPN:

[0152]

[0153] Comparison between Physically Constrained Neural Radiance Field (PC-NeRF) and Traditional NeRF:

[0154]

[0155] Based on hardware configuration:

[0156] 3 IMX585 low-light cameras (120dB wide dynamic range, F1.4 aperture), resolution 1920

[0157] ×1080@60fps.

[0158] RTX4090 GPU (using TensorCore to accelerate FP16 calculations).

[0159] Experimental data (RTX4090 platform, pure RGB configuration):

[0160]

[0161] In some embodiments, the multi-view 3D skeleton real-time reconstruction method proposes a single-modal lightweight architecture of "Spatiotemporal Feature Pyramid Network (ST-FPN) and Physical Constrained Neural Radiance Field (PC-NeRF)":

[0162] 1. Motion blur decoupling: Based on an optical flow-guided feature pyramid, it separates moving objects from static backgrounds in low-light videos and restores high-frequency details in blurred areas.

[0163] 2. Physically constrained rendering: Embed skeletal kinematics and dynamics constraints in the neural radiation field, and verify the biomechanical rationality of the reconstructed posture through a real-time physics engine.

[0164] 3. Hardware cost optimization: Using only three low-light RGB cameras or video cameras + a single RTX4090 GPU, the system cost is low, down 70% compared to traditional solutions.

[0165] Highlights of technological differentiation:

[0166]

[0167] Pure RGB motion blur decoupling: No event camera is required. Dynamic details are restored through the spatiotemporal feature pyramid guided by optical flow, breaking through the low-light motion blur problem and reducing costs by 70%.

[0168] Two-stage physical constraints: For the first time, real-time physical engine verification is embedded into the neural radiation field, achieving a qualitative change from "geometric reconstruction" to "physically credible reconstruction", reducing the abnormal posture rate by 89%.

[0169] Hardware cost-performance optimization: Through model quantization, sparse computing, and shared memory architecture, 60fps real-time reconstruction is achieved on low-cost hardware, increasing computing efficiency by 3 times.

[0170] End-to-end biomechanical modeling: Deeply integrate joint kinematic priors and dynamic constraints into the loss function to build an ergonomic reconstruction system, filling the gap in existing technology.

[0171] See also Figure 7 , a storage medium 710, wherein the storage medium 710 stores a computer program, wherein the computer program is executed by a processor 720 to perform the following steps:

[0172] The RGB camera array is used to collect skeleton data to obtain a video stream;

[0173] Preprocessing the video frames of the currently acquired video stream, wherein the preprocessing includes performing optical flow guided alignment on the current video frames;

[0174] The pre-processed current video frame is input into the spatiotemporal feature pyramid network for processing and then a multi-scale dynamic feature map is output;

[0175] The multi-scale dynamic feature map is input into the physically constrained neural radiation field for joint optimization to obtain a multi-view 3D skeleton model.

[0176] When performing 3D skeleton reconstruction, skeletal data is collected through an RGB camera array to generate a video stream. Optical flow-guided alignment is then performed on the video frames in the currently captured video stream to reduce motion blur and misalignment, and the video frames are enhanced in low-light conditions. This is then fed into a spatiotemporal feature pyramid network with preprocessed video frames to generate a multi-scale dynamic feature map. This multi-scale dynamic feature map is then fed into a physically constrained neural radiation field for joint optimization, resulting in a multi-view 3D skeleton model. Dynamic details are restored through optical flow-guided alignment combined with the spatiotemporal feature pyramid network, overcoming the challenge of low-light motion blur. This allows for low-cost, high-precision, and real-time 3D skeleton reconstruction using pure RGB camera or video camera hardware.

[0177] In some embodiments, performing optical flow guided alignment on the current video frame comprises the following steps:

[0178] Calculate the optical flow field of adjacent video frames of the video stream through the lightweight dense optical flow field algorithm;

[0179] The dynamic region features of the current video frame are aligned through bilinear interpolation Warp operation.

[0180] Use the lightweight RAFT optical flow algorithm (parameter compression to 12M) to calculate the optical flow field of adjacent frames , align dynamic region features through bilinear interpolation Warp operation. Recover low-light motion blur region details through dynamic feature alignment guided by optical flow, the formula is , reducing motion blur dislocation.

[0181] In some embodiments, performing low-light enhancement on the current video frame includes:

[0182] An end-to-end U-Net-based model injects Poisson noise into the current video frame during training to simulate low-light scenes. This model improves SSIM by 0.32 and PSNR by 9dB. SSIM (Structural Similarity) is a metric that measures the similarity between two images. PSNR (Peak Signal-to-Noise Ratio) is an engineering term that represents the ratio of the maximum possible signal power to the destructive noise power that affects its representation accuracy. Because many signals have a very wide dynamic range, the peak signal-to-noise ratio is often expressed in logarithmic decibels.

[0183] In some embodiments, the step of inputting the multi-scale dynamic feature map into the physically constrained neural radiation field for joint optimization to obtain a multi-view 3D skeleton model specifically includes the following steps:

[0184] The multi-scale dynamic feature map is input into the skeleton branch and the radiation field branch of the physical constraint neural radiation field respectively;

[0185] The skeleton branch uses the encoder to calculate multi-head self-attention to capture joint spatial associations, and outputs 3D skeleton coordinates through the decoder. The calculated 3D skeleton coordinates are input into the physical operation engine to calculate the joint torque error and verify the output 3D skeleton coordinates.

[0186] The radiance field branch uses micro-NeRF with depthwise separable convolution and sparse tensor core acceleration to perform volume density and color prediction;

[0187] The prediction results of skeletal branches and radiation field beams are jointly optimized by embedding loss functions based on joint angle soft constraints and torque balance constraints to obtain a multi-view 3D skeletal model.

[0188] As shown in the figure, the dual-branch architecture of the physical constraint neural radiation field has a 2-layer Transformer encoder and a 3-layer 256-layer MLP decoder in the skeleton branch. The encoder calculates multi-head self-attention (8 heads, dimension 64) to capture the spatial correlation of joints, and the decoder outputs 3D skeleton coordinates. , perform 3D skeleton prediction, and then convert the 3D skeleton coordinates Input into NVIDIAPhysX engine to calculate joint torque error , for 3D bone coordinates The radiation field branch uses micro NeRF (parameter count: 0.2M) combined with sparse tensor core acceleration to perform volume density and color prediction: , which uses position encoding (10-level sine) + direction encoding (4-level sine) to improve the capture of high-frequency details, and uses the RTX4090 sparse tensor core to dynamically crop voxels, reducing the amount of calculation by 60%. Combining the joint angle soft constraint and the torque balance constraint to embed the loss function, the loss function is obtained:

[0189] ;

[0190] : The set loss of 3D skeleton coordinate reconstruction is calculated as follows:

[0191] ;

[0192] in, Indicates the number of 3D skeleton key points (integer), such as =17, indicating that 17 key bone points of the human body are selected to calculate the geometric loss of 3D reconstruction bone coordinates; represents the joint weight factor (torso joint = 1.5, end joint = 0.8); Represents the L2 norm, that is, the Euclidean norm, where the Euclidean distance between the predicted 3D skeleton coordinates and the true coordinates is calculated; Represents the predicted 3D bone coordinates (i-th bone point); represents the real 3D joint coordinates (the i-th bone point, generated by multi-view triangulation);

[0193] : Soft constraint of joint angle (e.g. knee flexion 10°~160°, Huber loss);

[0194] : Moment balance constraint ( is the L2 norm).

[0195] represents the rendering loss, which is defined as: , which means measuring the pixel-level error between the RGB image rendered by the neural radiance field (PC-NeRF) and the real image of the input video frame, and calculating the pixel-by-pixel difference through the L2 norm; its function is to ensure that the projection of the reconstructed 3D skeleton model under multiple perspectives is consistent with the input video stream, and to assist in optimizing the volume density and color prediction of the neural radiance field, thereby improving the geometric accuracy and visual realism of 3D reconstruction.

[0196] In some embodiments, the step of inputting the pre-processed current video frame into the spatiotemporal feature pyramid network for processing and then outputting a multi-scale dynamic feature map specifically includes the following steps:

[0197] Output 3 layers of feature maps through the lightweight ResNet-34 backbone network;

[0198] Capture cross-frame dynamic information through the spatiotemporal convolution layer with 3D convolution embedded in each layer, and output a three-level spatiotemporal feature map;

[0199] Generate dynamic masks;

[0200] The SE-Net structure is applied to each level of spatiotemporal feature map, and the channel weights are adjusted through dynamic masks to generate multi-scale dynamic feature maps.

[0201] The structure of the spatiotemporal feature pyramid network is a three-scale pyramid:

[0202] Backbone network: Use lightweight ResNet-34 to output 3 layers of feature maps (resolution 1 / 4, 1 / 8, 1 / 16, number of channels [64, 128, 256]).

[0203] Spatiotemporal convolution layer: Each layer embeds 3D convolution (kernel size 3×3×3, step size 1×2×2), captures cross-frame dynamic information, and outputs spatiotemporal features .

[0204] Dynamic mask generation:

[0205] ;

[0206] in is the optical flow amplitude map, It is Sigmoid activation, and the area with mask value > 0.5 is considered as a moving target.

[0207] Channel attention enhancement: Apply the SE-Net structure in each level of feature map and adjust the channel weights through dynamic masks:

[0208] ;

[0209] Output: Multi-scale dynamic feature maps and 2D skeleton point heatmaps (an improved version of YOLOv8-Pose, using a spatiotemporal feature pyramid as input). Based on the motion target mask, a weighted summation is performed on the feature maps of each channel to enhance the representation of dynamic region features while suppressing static background interference. This effectively highlights key skeletal motion information and provides high-quality feature input for subsequent 2D skeleton point heatmap generation and 3D reconstruction.

[0210] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of this application, this does not limit the scope of patent protection of this application. All technical solutions generated by replacing or modifying equivalent structures or equivalent processes based on the essential concepts of this application using the contents recorded in the specification and drawings of this application, as well as directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, are included in the scope of patent protection of this application.

Claims

1. A multi-view 3D skeleton real-time reconstruction method, characterized in that: The following steps are involved: The video stream is obtained by collecting skeleton data through the RGB camera array; Preprocessing the video frames of the currently acquired video stream includes performing optical flow guided alignment on the current video frames and performing low illumination enhancement on the current video frames; The pre-processed current video frame is input into the spatiotemporal feature pyramid network for processing and then a multi-scale dynamic feature map is output; The multi-scale dynamic feature map is input into the physical constraint neural radiation field for joint optimization to obtain a multi-view 3D skeleton model; The step of inputting the multi-scale dynamic feature map into the physical constraint neural radiation field for joint optimization to obtain a multi-view 3D skeleton model specifically includes the following steps: The multi-scale dynamic feature map is input into the skeleton branch and the radiation field branch of the physical constraint neural radiation field respectively; The skeleton branch uses the encoder to calculate multi-head self-attention to capture joint spatial associations, and outputs 3D skeleton coordinates through the decoder. The calculated 3D skeleton coordinates are input into the physical operation engine to calculate the joint torque error and verify the output 3D skeleton coordinates. The radiance field branch uses micro-NeRF with depthwise separable convolution and sparse tensor core acceleration to perform volume density and color prediction; The prediction results of skeletal branches and radiation field beams are jointly optimized by embedding loss functions based on joint angle soft constraints and torque balance constraints to obtain a multi-view 3D skeletal model.

2. The multi-view 3D skeleton real-time reconstruction method according to claim 1, characterized in that: The optical flow guided alignment of the current video frame comprises the following steps: Calculate the optical flow field of adjacent video frames of the video stream through the lightweight dense optical flow field algorithm; The dynamic region features of the current video frame are aligned through bilinear interpolation Warp operation.

3. The multi-view 3D skeleton real-time reconstruction method according to claim 1, characterized in that: The low-light enhancement of the current video frame includes: Based on the end-to-end U-Net model, Poisson noise is injected into the current video frame during training to simulate dark light scenes.

4. The multi-view 3D skeleton real-time reconstruction method according to claim 1, characterized in that: The step of inputting the pre-processed current video frame into the spatiotemporal feature pyramid network for processing and then outputting a multi-scale dynamic feature map specifically includes the following steps: Output 3 layers of feature maps through the lightweight ResNet-34 backbone network; Capture cross-frame dynamic information through the spatiotemporal convolution layer with 3D convolution embedded in each layer, and output a three-level spatiotemporal feature map; Generate dynamic masks; The SE-Net structure is applied to each level of spatiotemporal feature map, and the channel weights are adjusted through dynamic masks to generate multi-scale dynamic feature maps.

5. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the following steps are performed: The video stream is obtained by collecting skeleton data through the RGB camera array; Preprocessing the video frames of the currently acquired video stream, wherein the preprocessing includes performing optical flow guided alignment on the current video frames; The pre-processed current video frame is input into the spatiotemporal feature pyramid network for processing and then a multi-scale dynamic feature map is output; The multi-scale dynamic feature map is input into the physical constraint neural radiation field for joint optimization to obtain a multi-view 3D skeleton model; The step of inputting the multi-scale dynamic feature map into the physical constraint neural radiation field for joint optimization to obtain a multi-view 3D skeleton model specifically includes the following steps: The multi-scale dynamic feature map is input into the skeleton branch and the radiation field branch of the physical constraint neural radiation field respectively; The skeleton branch uses the encoder to calculate multi-head self-attention to capture joint spatial associations, and outputs 3D skeleton coordinates through the decoder. The calculated 3D skeleton coordinates are input into the physical operation engine to calculate the joint torque error and verify the output 3D skeleton coordinates. The radiance field branch uses micro-NeRF with depthwise separable convolution and sparse tensor core acceleration to perform volume density and color prediction; The prediction results of skeletal branches and radiation field beams are jointly optimized by embedding loss functions based on joint angle soft constraints and torque balance constraints to obtain a multi-view 3D skeletal model.

6. The storage medium according to claim 5, wherein: The optical flow guided alignment of the current video frame comprises the following steps: Calculate the optical flow field of adjacent video frames of the video stream through the lightweight dense optical flow field algorithm; The dynamic region features of the current video frame are aligned through bilinear interpolation Warp operation.

7. The storage medium according to claim 5, wherein: The low-light enhancement of the current video frame includes: Based on the end-to-end U-Net model, Poisson noise is injected into the current video frame during training to simulate dark light scenes.

8. The storage medium according to claim 5, wherein: The step of inputting the pre-processed current video frame into the spatiotemporal feature pyramid network for processing and then outputting a multi-scale dynamic feature map specifically includes the following steps: Output 3 layers of feature maps through the lightweight ResNet-34 backbone network; Capture cross-frame dynamic information through the spatiotemporal convolution layer with 3D convolution embedded in each layer, and output a three-level spatiotemporal feature map; Generate dynamic masks; The SE-Net structure is applied to each level of spatiotemporal feature map, and the channel weights are adjusted through dynamic masks to generate multi-scale dynamic feature maps.

Images