Sparse view angle multi-human body joint reconstruction method based on dynamic graph convolutional network

Abstract

The invention discloses a sparse view angle multi-human body joint reconstruction method based on a dynamic graph convolutional network. The method comprises the following steps: firstly, acquiring indoor RGB video data by using an RGB camera; secondly, based on RGB video data, estimating a human body 3D posture by adopting a human body parameter model estimation algorithm, and performing generation and prediction by utilizing a completion algorithm to complete reconstruction of a human body model; and finally, real-time classification, positioning and tracking of multiple human body actions are realized through an action recognition algorithm, and behavior modes and interaction intentions of family members are analyzed based on the reconstructed human body model and human body 3D postures. According to the method, more flexible human body dynamic representation is realized, the problem of possible inaccuracy of initial posture estimation is solved, and the precision and stability of multi-human body reconstruction are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and 3D reconstruction, and particularly relates to a method for realizing multi-person joint reconstruction, pose estimation, action recognition and tracking by using a dynamic graph convolutional network based on data collected by sparse-view cameras. Background Art

[0002] In recent years, with the rapid development of deep learning and computer vision technologies, intelligent home monitoring systems have become increasingly popular. In such systems, the monitoring of people is a core component, including pose tracking, behavior understanding, and intention analysis. Although significant breakthroughs have been made in the fields of human body reconstruction, behavior recognition, etc. with the support of artificial intelligence technologies, the existing technologies still face many challenges.

[0003] Traditional human body reconstruction methods mainly rely on multi-camera surround capture systems, such as marker-based motion capture systems or multi-camera array systems. Although these methods have high reconstruction accuracy, they have strict requirements on the number of devices and the deployment environment, are costly, and are difficult to be actually deployed and applied in ordinary home environments. In contrast, single-camera reconstruction methods are simple to deploy and low in cost, but in the face of complex multi-person scenarios, occlusion situations, and human interaction scenarios, the reconstruction quality often drops significantly, making it difficult to meet the actual application requirements. Therefore, as a compromise solution, using a small number of cameras to achieve intelligent home monitoring has become a practical and feasible technical route.

[0004] In the field of human body reconstruction technology, parametric models such as SMPL (Skinned Multi-Person Linear Model) and its improved version SMPL-X have made significant progress. Such models represent the human body shape and pose through a predefined set of parameters and can infer a relatively accurate 3D human body model from a small number of images. However, such methods still face challenges in dealing with complex scenarios such as multi-person interaction and partial occlusion, especially in the home monitoring environment with limited camera numbers and sparse views, where their performance and stability are difficult to guarantee.

[0005] Recently, the 3D Gaussian Splatting technology has shown great potential in the fields of scene reconstruction and rendering. This technology uses three-dimensional Gaussian functions as the basic elements of scene representation, can efficiently represent and render complex scenes, and at the same time provides the ability to update in real time. Compared with traditional voxel or mesh representations, 3D Gaussian Splatting has obvious advantages in rendering speed and performance details. However, applying it to dynamic human body reconstruction, especially the joint reconstruction scenario of multiple human bodies, for human body mutual occlusion, lacking complete and effective supervision signals, and realizing the complete reconstruction of multiple human bodies in an interactive scenario, is still a technical challenge to be solved urgently. Summary of the Invention

[0006] To solve the above problems, the present invention provides a sparse-view multi-person joint reconstruction method for home intelligent monitoring. This method uses a small number of RGB cameras to collect indoor video data from different perspectives, transmits the data to a central control computer through a wireless or wired network, and uses the human parameter model SMPL-X estimation algorithm combined with the 3D Gaussian splash reconstruction technology to realize the recognition of multi-person postures, actions, and behaviors. And a completion algorithm based on a dynamic graph convolutional network is used to generate and supplement the missing data in the sparse view, thereby overcoming the problems of occlusion and incomplete view.

[0007] In the prior art, the following problems mainly exist in multi-person joint reconstruction under sparse-view conditions:

[0008] 1. The number of cameras is limited, and it is impossible to obtain complete multi-angle observation information, resulting in poor human body reconstruction quality;

[0009] 2. In a multi-person interaction scenario, there is a serious problem of human body mutual occlusion, and it is difficult to effectively distinguish and reconstruct each individual;

[0010] 3. Traditional methods lack effective modeling of human interaction relationships and cannot make full use of the spatial correlation information between human bodies;

[0011] 4. Existing 2D completion methods cannot effectively handle occlusion and view missing problems in 3D space;

[0012] 5. The initial pose estimation result may be inaccurate and needs to be optimized and adjusted during the reconstruction process.

[0013] To solve the above problems, the technical solution of the present invention is:

[0014] A sparse-view multi-person joint reconstruction method based on a dynamic graph convolutional network, the method comprising:

[0015] (1) A data acquisition step of using a small number of RGB cameras to collect indoor RGB video data;

[0016] (2) A multi-person joint reconstruction step of using a human parameter model estimation algorithm to estimate the 3D pose of the human body, using the 3D Gaussian feature sphere representation method, and performing three-dimensional dynamic reconstruction and pose estimation on multiple interacting human bodies in the room based on the RGB video data, and synchronously optimizing the initial pose data during the reconstruction process;

[0017] (3) A missing data generation step of using a completion algorithm based on a dynamic graph convolutional network to directly perform feature fusion on the connection graph of the Gaussian feature sphere, generate and predict the missing data in the sparse view, and realize the construction of a complete human body model;

[0018] (4) Action recognition and tracking step, which realizes real-time classification, positioning and tracking of multi-human actions through an action recognition algorithm; (5) Behavior analysis and intention understanding step, which analyzes the behavior patterns and interaction intentions of family members based on the reconstructed human model and human 3D postures.

[0019] Preferably, the multi-human joint reconstruction step specifically includes:

[0020] (2.1) Based on RGB video data, obtain the input image of each view, and use a deep convolutional neural network to extract the human 2D key point information and the human segmentation information of the input image; due to mutual occlusion in the interaction scene, the segmentation information of the human body may be an incomplete human body. "1" represents human pixels, and "0" represents non-human pixels.

[0021] (2.2) Based on the detected 2D key point information, use the SMPL-X parameter fitting algorithm to estimate the 3D posture parameters and shape parameters of the human body.

[0022] (2.3) Convert the estimated SMPL-X shape parameters into a human patch mesh representation, where each vertex of the human mesh representation is associated with a 3D Gaussian feature sphere, so as to convert the human mesh representation into a 3D Gaussian feature sphere representation, where each Gaussian feature sphere contains a learnable dynamic geometric feature, a learnable dynamic appearance feature and a learnable semantic feature. The geometric feature predicts its geometric attributes (density, scale, angle) through a geometric attribute decoder, the appearance feature predicts its color attribute through an appearance attribute decoder, and the semantic feature predicts its human ID attribute through a semantic decoder.

[0023] (2.4) Render the above geometric attributes and appearance attributes into a 2D image representation under the target view through the differentiable Gaussian sputtering rendering technology; render the above human ID attribute into the human segmentation information of the input image through the differentiable Gaussian sputtering rendering technology. Calculate the rendering errors between the rendered 2D image representation and the human segmentation information and the input image and the estimated human segmentation information of the input image respectively.

[0024] (2.5) According to the rendering errors calculated above, propose a Gaussian feature sphere adaptive densification algorithm to adaptively densify the Gaussian feature spheres in the corresponding regions guided by the loss magnitude for the regions with larger losses, so as to improve the reconstruction quality of the corresponding regions. The added Gaussian feature spheres are initialized with the features of the Gaussian feature spheres closest to them and are associated with the vertices in the human mesh representation closest to them.

[0025] The Gaussian feature sphere adaptive densification algorithm is implemented as follows:

[0026] First, initialize the Gaussian sphere set and set the error threshold ε and the upper limit r_max of the splitting ratio;

[0027] For each Gaussian sphere g in the Gaussian sphere set, calculate the rendering error e;

[0028] When the rendering error e is greater than the error threshold ε, calculate the splitting ratio r = min(e / ε, r_max);

[0029] Split the Gaussian sphere g into r Gaussian spheres and update the Gaussian sphere set.

[0030] (2.6) Through the joint iterative optimization in multiple perspectives in the above steps 2.4 and 2.5, learn the geometric features, color features, and semantic features in the Gaussian spheres and their corresponding attribute decoders, and at the same time optimize the 3D human pose parameters, correct the inaccuracies in the initial human pose estimation, and enhance the human body segmentation ability and reconstruction quality in the 3D space.

[0031] Preferably, the missing data generation step specifically includes:

[0032] (3.1) Based on the human mesh representation described in (2.3), use the topological structure of the mesh to construct a connection relationship graph between Gaussian feature spheres.

[0033] (3.2) Obtain the complete human body segmentation information by rendering the human mesh representation of each human body through mesh rendering technology, perform a pixel-by-pixel "AND" operation on this segmentation information and the incomplete segmentation information in (2.1) (i.e., the input image human body segmentation information) to obtain the human visible region mask; take the inverse of the incomplete segmentation information in (2.1) and perform an "AND" operation with this complete human body segmentation information to obtain the human invisible region mask.

[0034] (3.3) Through camera inverse transformation projection, project the above visible region mask and invisible region mask onto the 3D Gaussian feature sphere representation respectively to obtain the visible region Gaussian feature sphere and the invisible region Gaussian feature sphere.

[0035] (3.4) Construct a dynamic graph convolutional neural network (DGCN), perform dynamic graph convolutional calculation on the Gaussian feature sphere connection relationship graph described in (3.1), and use the geometric features, color features, and semantic features of the Gaussian feature spheres in the visible region to predict the corresponding features of the Gaussian feature spheres in the invisible region.

[0036] (3.5) Decode the geometric features and color features of the completed Gaussian feature sphere into complete geometric and color attributes through the geometric attribute decoder and color attribute decoder described in (2.3), so as to achieve high-fidelity complete reconstruction from sparse viewpoints.

[0037] Preferably, the missing data generation step further includes a temporal domain consistency constraint, which enables the geometric and texture attributes predicted by each decoder (MLP) to maintain spatio-temporal consistency under multiple viewpoints and multiple time points, increasing the constraints for completion and ensuring the rationality of the generated completion.

[0038] Preferably, the action recognition and tracking step adopts an algorithm combining spatio-temporal interest points and dense trajectories to achieve accurate recognition and real-time tracking of multi-human actions in complex scenarios.

[0039] Preferably, the behavior analysis and intention understanding step includes mining behavior patterns based on temporal data and modeling multi-person interaction relationships, and can identify the interaction behaviors and their intentions among family members.

[0040] The sparse-view multi-human joint reconstruction method, system and device based on the dynamic graph convolutional network provided by the present invention have the following beneficial effects:

[0041] Through the sparse-view data acquisition method, the hardware requirements and deployment costs of the home monitoring system are significantly reduced, and effective coverage of the home environment can be achieved by using 3-5 cameras.

[0042] Innovatively designed a Gaussian feature sphere representation method. Different from the original Gaussian sphere representation design, the Gaussian sphere representation in this method is replaced by a Gaussian feature sphere representation. The Gaussian feature sphere contains geometric features, color features and semantic features, and uses the corresponding geometric attribute decoder, color attribute decoder and semantic decoder to predict geometric attributes, color attributes and human ID attributes, realizing a more flexible representation of human dynamics.

[0043] By synchronously optimizing the SMPL-X pose parameters during the reconstruction process, the problem that the initial pose estimation may be inaccurate is overcome, and the accuracy and stability of multi-human reconstruction are significantly improved.

[0044] By using the incomplete segmentation information of each human body as a supervision signal, combining the Gaussian sputtering differentiable rendering technology, and the proposed reconstruction loss-guided adaptive Gaussian feature sphere densification technology, the human body segmentation accuracy in 3D space is realized, and the reconstruction quality of multi-person interaction scenarios is significantly improved.

[0045] Based on the dynamic graph convolutional network, convolution operations are directly performed on the connection relationship graph of Gaussian feature spheres. The features of Gaussian feature spheres in the invisible region are predicted using the features of Gaussian feature spheres in the visible region, without the intermediate step of UV mapping, simplifying the processing flow and improving efficiency, and effectively solving the problem of data loss under sparse viewpoints.

[0046] A behavior analysis system for multi-person interaction scenarios is designed, which can understand the interaction intentions among family members and provide technical support for applications such as home security monitoring and elderly health monitoring. Brief Description of the Drawings

[0047] Figure 1 It is the flowchart of multi-person joint reconstruction and missing data generation of the present invention;

[0048] Figure 2 It is the combined schematic diagram of the human body parameter model SMPL-X and the 3D Gaussian feature sphere of the present invention;

[0049] Figure 3 It is the architecture diagram of the data generation module based on the dynamic graph convolutional network of the present invention;

[0050] Figure 4 It is the flowchart of multi-person interaction behavior analysis of the present invention. Detailed Embodiments

[0051] The technical solutions of the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0052] As Figure 1 shown, a multi-person joint reconstruction method from sparse viewpoints based on the dynamic graph convolutional network proposed by the present invention is specifically implemented as follows:

[0053] Embodiment 1: Data Acquisition and Transmission

[0054] The data acquisition in this embodiment consists of 3-5 RGB cameras at fixed positions. The camera resolution is 1920×1080 pixels and the frame rate is 30fps. The arrangement of the cameras follows the principle of "less but better", mainly covering the main activity areas such as the living room and corridor of the home, and at the same time ensuring that at least two cameras have overlapping viewpoints in the key areas for subsequent multi-view data fusion processing.

[0055] Data transmission supports two transmission methods: wireless network (WiFi 6, 802.11ax standard) and wired network (gigabit Ethernet). After the original video captured by the camera is encoded and compressed by H.265, it is transmitted to the central control computer through the network. The transmission module uses the Real-Time Streaming Protocol (RTSP) to ensure low-latency transmission of video data, and the average transmission delay is controlled within 30ms.

[0056] Example 2: Multi-person Joint Reconstruction

[0057] As Figure 1 and Figure 2 shown, multi-person joint reconstruction is one of the core innovations of the present invention. Through the innovative 3D Gaussian feature sphere representation and reconstruction error-guided adaptive Gaussian feature sphere densification technology, high-precision real-time reconstruction of multiple people under sparse viewpoints is achieved.

[0058] In the first step, for the images of each viewpoint, a deep convolutional neural network is used to extract 2D human key points and the segmentation information of each human body. In this example, the human body segmentation information and the 2D human key point localization are obtained to identify the positions of 23 key nodes of each human body.

[0059] In the second step, based on the above-mentioned 2D human key points, the SMPL-X human parameter fitting algorithm is used to estimate the 3D pose parameters and shape parameters of the human body. The SMPL-X model is defined as follows:

[0060]

[0061] Among them, is the shape parameter, is the pose parameter, is the facial expression parameter, is the blending weight, is the 3D key point, T is the human mesh representation, is the linear blend skinning function. The parameters are estimated by optimizing the following objective function:

[0062] Among them, is the key point projection error, and are the prior constraints of the shape parameter and the pose parameter respectively, , and are the loss weight coefficients. It should be noted that the initial SMPL-X pose parameters may be inaccurate, especially in scenarios with occlusion and complex poses. Therefore, in the subsequent reconstruction process, these pose parameters will be synchronously optimized to obtain more accurate reconstruction results.

[0063] In the third step, the estimated SMPL-X human parameter model is converted into a 3D Gaussian feature sphere representation . Specifically, for each vertex in the human body patch mesh representation of the SMPL-X model corresponding to each human body, a 3D Gaussian feature sphere is initialized. Different from the 3D Gaussian spheres in the original Gaussian sputtering technology, the present invention abandons the representation method that directly represents geometric attributes and color attributes, and adds the representation of semantic attributes. Specifically, each Gaussian feature sphere uses geometric features , color features and semantic features , and their corresponding geometric attribute decoders , color attribute decoders and semantic attribute decoders to represent geometric attributes , color attributes and semantic attributes . is the position of the Gaussian feature sphere, is the opacity of the Gaussian feature sphere, is the variance of the Gaussian feature sphere.

[0064] Using a dynamically learnable feature representation to replace the direct attribute representation improves the representation ability and flexibility of the model, and through the added semantic attributes, through the Gaussian sputtering differentiable rendering technology, the effective fusion of 2D segmentation information to 3D segmentation information is realized, solving the problem of accurate separation of multiple human bodies in the interaction scenario.

[0065] Step 4: Using the Gaussian sputtering rendering technology, the Gaussian feature sphere representation can be rendered into a 2D image and human body segmentation information . Using the corresponding input image from the target perspective and the pre-estimated human body segmentation information , the rendering error can be calculated:

[0066]

[0067] Step 5: In order to better achieve the segmentation accuracy at the edges of interacting human bodies, we propose an error-guided Gaussian feature sphere adaptive densification algorithm. At the human body separation boundary, the threshold for splitting the Gaussian feature sphere is adaptively adjusted according to the magnitudes of the human body reconstruction error and the human body segmentation error, so as to achieve adaptive densification, and use a larger number of smaller Gaussian feature spheres to cover, improving the reconstruction quality and separation accuracy. The pseudo-code of this algorithm is as follows:

[0068] Input: Initial set of Gaussian spheres , error threshold ε, upper limit r_max of the splitting ratio;

[0069] Output: Optimized set of Gaussian spheres G'

[0070] 1. Initialize G' =

[0071] 2. For each Gaussian sphere g_i in G':

[0072] 2.1 Calculate the rendering error e_i

[0073] 2.2 If e_i > ε:

[0074] 2.2.1 Calculate the splitting ratio r = min(e_i / ε, r_max)

[0075] 2.2.2 Split g_i into r small Gaussian spheres

[0076] 2.2.3 Update G'

[0077] 3. Return G'

[0078] Through the above steps, the system can achieve simultaneous separation and reconstruction of up to 6 people. The reconstruction accuracy is improved by 30% compared with traditional methods, and the frame rate reaches 100 fps, meeting the requirements of real-time applications.

[0079] Example 3: Missing data generation

[0080] As Figure 3 shown, missing data generation is another key innovation point of the present invention. A generation algorithm based on a dynamic graph convolutional network is used to perform intelligent reasoning and completion on the 3D human body regions that are invisible or occluded under sparse viewpoints.

[0081] Due to insufficient observation from sparse viewpoints and mutual occlusion of multiple human bodies in an interactive scene, some regions of the Gaussian feature human body representation output by multi-human joint reconstruction cannot be effectively reconstructed due to lack of effective supervision, resulting in an incomplete human body. We use the pre-defined connection relationship graph of SMPL-X to construct the connection relationship graph of Gaussian feature spheres of the 3D human body and complete it through a dynamic graph convolutional network.

[0082] In the first step, based on the human mesh representation described above, which consists of a vertex set V and an interconnected relationship F, a connection relationship matrix A of all vertices can be constructed, thereby obtaining the connection relationship graph of Gaussian feature spheres associated with all vertices.

[0083] In the second step, the human mesh representation of the SMPL-X parameterized human body model can represent the complete human body geometry. Through mesh rendering technology, the 2D segmentation information of the complete human body can be obtained , which is subjected to an "AND" operation with the aforementioned estimated incomplete human body segmentation information to obtain the human visible region mask , and the invisible region mask can be calculated as .

[0084] In the third step, given the camera parameters C under the target perspective, through the inverse projection transformation , the Gaussian feature sphere set of the visible region and the Gaussian sphere set of the invisible region can be obtained.

[0085] In the fourth step, a dynamic graph convolutional neural network (DGCN) is constructed. On the connection relationship graph of Gaussian feature spheres, using the connection relationship matrix A, the Gaussian sphere features of the invisible region are predicted from the Gaussian feature sphere features of the visible region, including geometric features, appearance features, and semantic features. The core operation of DGCN is:[[]]

[0086]

[0087] represents the feature representation of node in the th layer, is the neighborhood connection weight calculated through the connection relationship matrix A, N(i) represents the set of neighbor nodes connected to node , (·) is the activation function, and are the convolution kernel weight and bias of the th layer respectively.

[0088] The innovation of the present invention lies in directly performing graph convolution operations on the connection relationship graph of Gaussian feature spheres without going through the intermediate step of UV mapping. By performing dynamic graph convolution calculations on the Gaussian feature spheres bound to the human mesh, using the Gaussian feature sphere features of the visible region, including geometric features, color features, and semantic features, the Gaussian sphere features of the invisible region are predicted. For each Gaussian feature sphere of the invisible region, the feature vector is completed, and its update formula is:[[]]

[0089]

[0090] where represents the feature transfer weight between the i-th node and the j-th node, satisfying the normalization constraint. F_j is the Gaussian feature sphere in the visible region that is the closest to f_i according to the connection relationship matrix A .

[0091] In the fifth step, by combining the Gaussian feature spheres of the visible region and the predicted Gaussian feature spheres of the invisible region, the complete human body reconstruction result can be obtained through the aforementioned Gaussian sputtering rendering technology.

[0092] Through the above process, the proposed solution can fuse spatial domain information and time series information in both the spatial domain and the time domain, generate missing human data consistent with the known part, and achieve the reconstruction of a complete human model. The average time consumption of the generation process is 350 ms, meeting the requirements of near real-time processing.

[0093] Example 4: Action Recognition and Tracking

[0094] The action recognition and tracking in this example are based on the spatio-temporal graph convolutional network (ST-GCN) and the two-stream network architecture to achieve action recognition and tracking of the reconstructed human body.

[0095] First, represent the human key point sequence as a spatio-temporal graph G=(V, E), where V is the set of nodes (corresponding to key points) and E is the set of edges (corresponding to bone connections). The spatio-temporal graph convolution operation is defined as:

[0096]

[0097] Among them, and are the input and output features respectively, is the convolution kernel, is the adjacency matrix, and K is the set of convolution kernels.

[0098] Secondly, introduce a feature extraction method based on spatio-temporal interest points and dense trajectories. By analyzing the movement trajectories of key points, capture the spatio-temporal features of actions. Spatio-temporal interest points (STIPs) locate the positions where actions occur by detecting significant region changes in the video, while dense trajectories describe the detailed changes of actions by tracking the movement paths of multiple feature points.

[0099] Finally, use the two-stream network architecture to process both spatial stream (single-frame image) and temporal stream (optical flow) information simultaneously. By weighted fusion of the prediction results of the two streams, obtain the final action classification result:

[0100]

[0101] Among them, and are the prediction scores of the spatial stream and the temporal stream respectively, and are the weight coefficients.

[0102] This example predefines 80 common categories of household activity actions, including daily actions such as walking, sitting down, standing up, lying down, bending over, falling, and household activities such as cooking, cleaning, watching TV, etc. The accuracy rate can reach over 92% in complex household scenarios, meeting the requirements of real-time monitoring.

[0103] Example 5: Behavior Analysis and Intention Understanding

[0104] As Figure 4 shown, behavior analysis and intention understanding are based on the mining of behavioral patterns in time-series data and the modeling of multi-person interaction relationships, analyzing the behavioral patterns and interaction intentions of family members.

[0105] First, construct a behavioral sequence representation model to represent the recognized action sequence as a time-series feature vector:

[0106] , where is the action feature at time t.

[0107] Secondly, design a multi-person interaction relationship graph , where is the set of human body nodes, is the set of interaction relationship edges. The relationships between nodes are modeled through the following attention mechanism:

[0108] ,

[0109] where, is the query vector, is the key vector, is the value vector, is the set of neighbors of node i.

[0110] Finally, through time-series pattern analysis and rule reasoning, identify the behavioral intentions of family members, such as:

[0111] Identify abnormal behaviors (such as the elderly falling, children's dangerous behaviors, etc.);

[0112] Understand daily activities (such as dining, resting, entertainment, etc.);

[0113] Analyze the interaction relationships of family members (such as chatting, assisting, etc.).

[0114] This embodiment can identify 10 basic intention categories and 30 complex interaction intentions, providing intelligent support for home security monitoring.

[0115] Embodiment Six: System Integration and Deployment

[0116] The system of the present invention is implemented using a centralized processing architecture, and a central control computer (PC host, equipped with a high-performance GPU) is responsible for executing all computationally intensive tasks. The minimum configuration requirements of the central control computer are: A

[0117] CPU: Intel Core i7-10700 or above AMD Ryzen 7 3700X;

[0118] GPU: NVIDIA RTX 3060 12GB or above, supporting CUDA 11.0+;

[0119] Memory: 32GB DDR4;

[0120] Storage: 1TB SSD;

[0121] Network: Support for gigabit wired network and WiFi 6 wireless network.

[0122] The software architecture adopts a modular design, and each functional module communicates and exchanges data through standard interfaces. The system running environment is Ubuntu 20.04 LTS or Windows 10 Professional Edition, and the core algorithm library is implemented using PyTorch version 1.10 or above.

[0123] The end-to-end processing delay of the system (from data acquisition to result output) is controlled within 100ms, meeting the requirements of real-time monitoring. The system supports continuous operation for 7×24 hours and has functions of self-recovery from faults and data backup to ensure the stability and reliability of the monitoring system.

[0124] Effect verification of the embodiment:

[0125] The present invention has been tested and verified in an actual home environment. The test environment is a typical three-bedroom residence with an area of about 120 square meters, and 5 cameras are deployed. The test scenarios include various complex scenarios such as daily life, multi-person interaction, and occlusion situations.

[0126] The test results are shown in Table 1:

[0127] Multi-person body reconstruction accuracy: It reaches 97.5% under unoccluded conditions, 92.3% under mild occlusion, and still reaches 85.6% under severe occlusion.

[0128] Quality of missing data generation: Through user research and quantitative evaluation, the structural similarity (SSIM) between the generated data and the real data reaches above 0.85, meeting the visual quality requirements.

[0129] Action recognition accuracy: The single-person action recognition accuracy is 94.2%, and in the multi-person interaction scenario, it is 91.5%.

[0130] System response time: The average end-to-end processing delay is 82ms, meeting the real-time requirements.

[0131] System stability: It runs continuously without failure for 72 hours, with an average CPU load of 42% and an average GPU load of 65%.

[0132] Compared with the prior art, the present invention has significant advantages in the following aspects:

[0133] The demand for the number of cameras is significantly reduced, from 10 - 20 in traditional systems to 3 - 5, greatly reducing the deployment cost.

[0134] The multi - person processing ability is enhanced, capable of handling complex interaction scenarios with up to 6 people simultaneously.

[0135] The reconstruction accuracy is improved. Especially under occlusion and sparse view conditions, the error is reduced by more than 30% compared with traditional methods.

[0136] The real - time performance is optimized, with a frame rate of 100fps, meeting the requirements of real - time monitoring.

[0137] The degree of intelligence is increased, capable of understanding the interaction intentions and behavior patterns of multiple people, providing more value for home security monitoring.

[0138] Table 1

[0139]

Claims

1. A sparse-view multi-human joint reconstruction method based on a dynamic graph convolutional network, characterized in that Including the following steps: S1. Use an RGB camera to collect indoor RGB video data; S2. Based on the RGB video data, adopt a human body parameter model estimation algorithm to estimate the 3D pose of the human body, and use a completion algorithm for generation and prediction to complete the reconstruction of the human body model; S3. Based on the reconstructed human body model, use an action recognition algorithm to achieve real-time classification, positioning, and tracking of multi-human actions; and analyze the behavior patterns and interaction intentions of family members based on the reconstructed human body model and the 3D pose of the human body.

2. The multi-person joint reconstruction method from a sparse perspective based on a dynamic graph convolutional network according to claim 1, wherein The specific implementation process of step 2 is as follows: Step 2.

1. Adopt a human body parameter model estimation algorithm to estimate the 3D pose of the human body. Use the 3D Gaussian feature sphere representation method to perform three-dimensional dynamic reconstruction and pose estimation on multiple interacting human bodies indoors based on the RGB video data, and synchronously optimize the initial pose data during the reconstruction process; Step 2.

2. Based on the results of the three-dimensional dynamic reconstruction and pose estimation, use a completion algorithm based on a dynamic graph convolutional network to directly perform feature fusion on the connection relationship graph of the Gaussian feature sphere, generate and predict the missing data under the sparse view, and realize the construction of a complete human body model.

3. The multi-person joint reconstruction method based on a dynamic graph convolutional network from a sparse perspective according to claim 2, wherein The specific implementation process of step 2.1 is as follows: Step 2.1.

1. Based on the RGB video data, obtain the input image of each view, and use a deep convolutional neural network to extract the 2D key point information of the human body and the human body segmentation information of the input image; Step 2.1.

2. Based on the detected 2D key point information, use the SMPL-X parameter fitting algorithm to estimate the 3D pose parameters and shape parameters of the human body; Step 2.1.

3. Convert the estimated shape parameters into a human body patch mesh representation. Each vertex of the human body mesh representation is associated with a 3D Gaussian feature sphere, and the human body mesh representation is converted into a 3D Gaussian feature sphere representation. Each Gaussian feature sphere contains a learnable dynamic geometric feature, a learnable dynamic appearance feature, and a learnable semantic feature. The geometric feature predicts its geometric attributes through a geometric attribute decoder, the appearance feature predicts its color attributes through an appearance attribute decoder, and the semantic feature predicts its human body ID attribute through a semantic decoder; Step 2.1.

4. The above geometric attributes and appearance attributes are rendered into a 2D image representation in the target view through the differentiable Gaussian sputtering rendering technology; the above human body ID attribute is rendered into the human body segmentation information of the input image through the differentiable Gaussian sputtering rendering technology; Calculate the rendering errors between the rendered 2D image representation and the human body segmentation information and the input image and the estimated human body segmentation information of the input image respectively; Step 2.1.

5. Based on the rendering errors calculated above, propose a Gaussian feature sphere adaptive densification algorithm to adaptively densify the Gaussian feature spheres in the corresponding areas guided by the error magnitude. The added Gaussian feature spheres are initialized with the features of the Gaussian feature sphere closest to them and are associated with the vertices in the human body mesh representation closest to them; Step 2.1.6: Through the joint iterative optimization from Step 2.1.4 and Step 2.1.5 under multiple perspectives, learn the geometric features, color features, and semantic features in the Gaussian sphere and their corresponding attribute decoders, and at the same time optimize the human 3D pose parameters to correct the inaccuracies in the initial human pose estimation.

4. The multi-person joint reconstruction method from a sparse perspective based on a dynamic graph convolutional network according to claim 3, characterized in that In the Gaussian feature sphere adaptive densification algorithm, on the human separation boundary, adaptively adjust the threshold for Gaussian feature sphere splitting according to the magnitudes of the human reconstruction error and the human segmentation error to achieve adaptive densification. The implementation process is as follows: First, initialize the Gaussian sphere set and set the error threshold ε and the upper limit r_max of the splitting ratio. For each Gaussian sphere g in the Gaussian sphere set, calculate the rendering error e. When the rendering error e is greater than the error threshold ε, calculate the splitting ratio r = min(e / ε, r_max). Split the Gaussian sphere g into r Gaussian spheres and update the Gaussian sphere set.

5. The multi-person joint reconstruction method based on a dynamic graph convolutional network from a sparse perspective according to claim 4, wherein The specific implementation process of Step 2.2 is as follows: Step 2.2.1: Based on the human mesh representation, use the topological structure of this mesh to construct a connection relationship graph between Gaussian feature spheres. Step 2.2.2: Obtain the complete human segmentation information for each human through the mesh rendering technique of the human mesh representation. Perform a pixel-by-pixel AND operation on this segmentation information and the human segmentation information of the input image in Step 2.1.1 to obtain the human visible region mask; take the inverse of the human segmentation information of the input image in Step 2.1.1 and perform an AND operation with this complete human segmentation information to obtain the human invisible region mask. Step 2.2.3: Through camera inverse transformation projection, project the visible region mask and the invisible region mask onto the 3D Gaussian feature sphere representation respectively to obtain the visible region Gaussian feature sphere and the invisible region Gaussian feature sphere. Step 2.2.4: Construct a dynamic graph convolutional neural network DGCN, perform dynamic graph convolutional calculations on the Gaussian sphere connection relationship graph, and use the geometric features, color features, and semantic features of the visible region Gaussian feature sphere to predict the corresponding features of the Gaussian feature sphere in the invisible region. Step 2.2.5: Through the geometric attribute decoder and color attribute decoder described in Step 2.1.3, decode the geometric features and color features of the complemented Gaussian feature sphere into complete geometric and color attributes to achieve complete reconstruction under sparse perspectives.

6. The multi-person joint reconstruction method based on a dynamic graph convolutional network from a sparse perspective according to claim 5, characterized in that The time-domain consistency constraint is also included in Step 2.2 to make the geometric and texture attributes predicted by the decoder maintain spatio-temporal consistency under multiple perspectives and multiple time points, increasing the constraints for complementation.

7. The sparse-view multi-human joint reconstruction method based on a dynamic graph convolutional network according to claim 1, characterized in that The action recognition algorithm adopts an algorithm that combines spatio-temporal interest points and dense trajectories to achieve multi-human action recognition and real-time tracking.

Images