Single-view large-scale crowd 3D reconstruction method based on hypergraph relational reasoning

Abstract

This invention discloses a method for single-view large-scale 3D reconstruction of crowds based on hypergraph relational reasoning. The method first constructs training data, which is composed of true values obtained by labeling and fitting a 3D human dataset, estimated values obtained using open-source methods, and original images from the 3D human dataset. A reconstruction network is then constructed, and the network is trained to reconstruct the 3D crowd. A loss function is constructed using the reconstruction results and the true values in the constructed training data to achieve supervision. Backpropagation is used to optimize network parameters until the network converges. The trained network is then used for subsequent reasoning. Finally, single-view large-scale crowd reconstruction is performed. This method can be applied to large-scale crowd reconstruction, with the number of people reconstructed in a single pass exceeding 50, significantly more than existing single-view reconstruction methods.

Description

Technical Field

[0001] The present invention relates to a single-view large-scale crowd three-dimensional reconstruction method based on hypergraph relationship reasoning, and belongs to the fields of computer vision and computer graphics. Background Art

[0002] Single-view 3D reconstruction of large-scale crowds holds enormous potential in areas such as crowd analysis, autonomous driving, human-computer interaction, and crowd safety management. With the rapid development of computer vision and artificial intelligence, conventional multi-person 3D reconstruction technology has found widespread application in fields such as sports analysis, autonomous driving, and virtual reality. Consequently, market demand for single-view 3D reconstruction of large-scale crowds is increasing. However, achieving single-view 3D reconstruction of large-scale crowds still faces numerous challenges. First, occlusions between people make it difficult to accurately recover human poses. Second, the large size disparity of human figures in large-scale scenes, coupled with the coupling of their depth in 3D space with the camera's focal length, makes it difficult to accurately reconstruct the spatial distribution of crowds in 3D space. Furthermore, reconstructing large crowds significantly increases the computational complexity. Therefore, by leveraging the collective nature of crowds to model group characteristics based on hypergraphs, we can achieve accurate pose and reasonable spatial distribution 3D reconstruction of large-scale crowds in single-view scenarios. This holds promise for the practical application of single-view 3D reconstruction of large-scale crowds. Summary of the Invention

[0003] Purpose of the Invention: This paper proposes a single-view large-scale 3D crowd reconstruction method based on hypergraph relational reasoning. This method first extracts human features from the original image, then uses hypergraph relational reasoning to model group relationships and obtain group features. This group feature is then used to update more reasonable human features. The updated human features and position information are then used to reconstruct a spatially distributed 3D crowd.

[0004] Technical solution: The present invention provides a method for single-view large-scale crowd 3D reconstruction based on hypergraph relationship reasoning, comprising the following steps:

[0005] S1. Construction of training data: Select a dataset containing 3D human body annotations. For datasets that only contain 3D joint point annotations, use a human mesh model with a unified topological structure frame by frame. Driven by skeleton skinning, fit the 3D joint points of each human body in the annotations and obtain the corresponding human mesh model parameters as the 3D human body annotations of the dataset. Store the 2D joint points, bounding boxes, 3D joint points, and human mesh model parameters in a unified format as the true value of the training data. Send the original images in the existing 3D human dataset into the existing open source target detection network to obtain the estimated human bounding box coordinates. Use the estimated human bounding box and the original image to obtain the estimated 2D human joint points through the existing open source 2D posture estimation method. Where n∈[1,…,N], N is the number of human bodies, N K is the number of joints in a single human body. The estimated human bounding box coordinates and the estimated two-dimensional joints are used as the estimated values in the training data. The true values obtained by annotation and fitting based on the 3D human dataset, the estimated values obtained based on open source methods, and the original images in the 3D human dataset together constitute the training data.

[0006] S2. Reconstruction Network Construction: The reconstruction network consists of three parts: a feature extraction module, a hypergraph relationship reasoning module, and a regression network. The human feature preprocessing module processes the human body bounding box into human body position information and processes the two-dimensional human body joints into tokens. The feature extraction module is composed of a Transformer network to implement the human body feature extraction process. The hypergraph relationship reasoning module models group relationships based on human body features and obtains group characteristics. The human body features are updated based on group characteristics, and the collectivity and interaction of groups are used to obtain more accurate human body features. The regression network is used to regress the updated human body features to obtain human body mesh model parameters, which are subsequently used to achieve three-dimensional human body reconstruction.

[0007] S3. Network training: Using the estimated human bounding box and estimated two-dimensional human joint points in the training data constructed in step S1 as the reconstruction network input, the human feature preprocessing module constructed in step S2 is used to obtain posture features and human position information. The human features are generated through the Transformer network constructed in step S2. A hypergraph is constructed based on the similarity of human postures and human features. The hypergraph is iteratively updated to update the human features. The human mesh model parameters are regressed based on the human features and position information to reconstruct the three-dimensional crowd. The loss function is constructed using the reconstruction results and the true values in the training data constructed in step S1 to achieve supervision. The network parameters are optimized by backpropagation until the network converges. The trained network is used for subsequent inference processes.

[0008] S4. Single-view large-scale crowd reconstruction: Collect images of large-scale crowd scenes from a single viewpoint, obtain the estimated human bounding box and estimated two-dimensional human joint points through existing open source methods as input to the reconstruction network constructed in step S2, obtain the human body parameter model through network forward reasoning, and reconstruct the three-dimensional crowd using the skeleton-skinned three-dimensional human mesh model.

[0009] Furthermore, the specific method of step S2 includes:

[0010] S21. Human Feature Extraction Module: The Transformer network consists of four identically structured Transformer blocks. Learnable spatial position embedding vectors are added to the inputs of the first two Transformer blocks, and their outputs are fed into the next two Transformer blocks. The Transformer network maps the inputs to a high-dimensional space, which serves as the output of the Human Feature Extraction Module.

[0011] S22. Hypergraph Relationship Reasoning Module: The output of step S21 is used to construct a hypergraph, which consists of nodes and hyperedges, defined as in is the set of nodes, ε is the set of hyperedges, and the hypergraphs of multiple scales are defined as The superscript indicates the scale number, and the hypergraph of the sth scale is defined as where ε (s) is the hyperedge of size s, defined as in Indicates the Mth scale under the sth scale s Hyperedges. Unless otherwise specified, hypergraphs in the following text refer to hypergraphs of multiple scales. Hypergraph updates are divided into two stages: updating hyperedges from nodes and updating nodes from hyperedges. Both update processes are implemented by multi-layer perceptrons. After multiple iterations, the updated nodes are obtained.

[0012] S23. Regression network: The regression network is composed of a multilayer perceptron, which maps the output obtained in step S22 back to a low-dimensional space as the output of the regression network;

[0013] Furthermore, the specific method of step S3 includes:

[0014] S31. Human feature preprocessing: Convert the human bounding box coordinates estimated in step S1 into human position information b n :

[0015]

[0016] Where n is the nth human body, f is the focal length of the original image, (c x ,c y ) is the coordinate of the center of the bounding box relative to the center of the original image, and d is the size of the bounding box;

[0017] The confidence of some of the human body's 2D joints estimated in step S1 is low. The 2D joints with confidence lower than T are replaced with learnable tokens, and the coordinates of the 2D joints are normalized and used as tokens.

[0018] S32. Human body feature extraction: Input the token obtained in step S31 into the Transformer network constructed in step S21 to obtain posture features Posture features and human body position information b n Constituent features of the human body n , as nodes of the hypergraph:

[0019]

[0020] Among them, v n is the human body feature and is also a node of the hypergraph, and m represents the dimension of the posture feature;

[0021] S33. Hypergraph construction: using adjacency matrix To define the connection relationship of the edge of the hypergraph under the s-th scale, that is Indicates that the i-th node is contained in the j-th hyperedge, otherwise The topological structure of the hypergraph is formed, and the hypergraph is defined as the form of step S22;

[0022] The hypergraph topology construction includes the following steps, and the hypergraph topology construction mentioned later is the same as this step:

[0023] S331. Calculate the correlation of postures based on posture features and construct a matching matrix

[0024]

[0025] in, is an element in the matching matrix, used to measure the posture similarity between the a-th person and the b-th person, q a is the posture feature of the ath person, q b is the posture feature of the bth person, T is the matrix transpose, ‖·‖2 is the two-norm, and the matching matrix is used to divide people with similar postures into the same group;

[0026] S332. Considering the existence of groups of different sizes in the crowd, the size of the sth group is recorded as K (s) , based on the matching matrix obtained in step S331 Get K (s) ×K (s) The high-density cofactor of (s) nodes as the same group, using hyperedges To represent this group, where t represents the t-th hyperedge, that is, the t-th group:

[0027]

[0028]

[0029] in, represents the tth hyperedge under the sth group size, v represents the set of all hypergraph nodes, N is the number of human bodies, that is, the number of hypergraph nodes, represents the pth node, Ω represents a subset of the hypergraph nodes, and the size of the set is K (s) , Represents the high-density cofactor of the matching matrix, Indicates the value of Ω when the subsequent expression is solved to obtain the maximum value, ‖·‖ 1,1 The sum of the absolute values of all elements in the expression is solved by the greedy algorithm;

[0030] S333. For each node in the hypergraph All proceed to step S332, where for the 0th group size, the connection relationship between each two nodes is constructed to obtain the adjacency matrix A hypergraph has N hyperedges and N nodes at any given group size, thus constructing the topological structure of the hypergraph

[0031] S34. Hypergraph relationship reasoning: Based on the hypergraph structure obtained in step S33, group information is transferred, and the human features on the nodes obtained in step S32 are aggregated into group features as the features of the hyperedge:

[0032]

[0033] Among them, e t is the t-th hyperedge, represents the p-th node, is a learnable multilayer perceptron, λ t is the contribution coefficient, which indicates the contribution of the p-th node to the t-th group, c t is the collective coefficient of the group, λ t Defined as:

[0034]

[0035] in, is a learnable multilayer perceptron, is the rth node, c t Defined as:

[0036]

[0037] Among them, σ(·) is the sigmoid function, is the average characteristic of the t-th group, It is a learnable multilayer perceptron;

[0038] Use the features on all hyperedges related to the current node to update the human features of the current node:

[0039]

[0040] in, is the pth human feature, e t is the t-th hyperedge, Represents The set of all hyperedges related to It is a learnable multi-layer perceptron that updates hyperedges and nodes multiple times on hypergraphs of all sizes to obtain updated human features.

[0041] After iterative updates of the hypergraph, the human features at different group sizes are connected with the human position information to form the final human representation Where n represents the nth person;

[0042] S35. Human body posture parameter regression: The human body representation obtained in step S34 is input into the regression network defined in step S23 to obtain human body posture parameters θ, human body shape parameters β and camera parameters [f c ,t x ,t y ], where f c is the estimated camera focal length, (t x ,t y ) is the estimated translation of the human mesh model relative to the center of the bounding box. The relative translation of the human mesh model is converted into an absolute translation based on the camera parameters and the human position information. x ,t y ,t z ):

[0043]

[0044] Where, f is the focal length of the original image, (c x ,c y ) is the coordinate of the center of the bounding box relative to the center of the original image, d is the size of the bounding box, and the estimated human body mesh model parameters of N human bodies are finally obtained.

[0045] S36. Loss function construction: The network is trained in an end-to-end manner. After the network structure constructed in step S2, the 157-dimensional estimated human mesh model parameters output in step S35 are obtained. Based on the estimated human mesh model parameters, an estimated 3D human mesh model and its corresponding estimated 3D joint points are obtained. The true values of the 2D joint points in the training data obtained in step S1 are used as supervision to construct a reprojection loss function:

[0046]

[0047] Among them, Π represents the projection of the estimated three-dimensional joint point to the two-dimensional image, is the estimated 3D joint point of the nth person, t n is the absolute translation of the nth person, Represents the true value of the two-dimensional joint point of the nth person, The square of the two norm of the expression is used, and the true value of the human mesh model parameters in the training data is used as supervision to construct the loss function

[0048]

[0049] in, is the true value of the body shape parameters of the nth person, is the true value of the human body posture parameter of the nth person, using the true value of the three-dimensional joint points in the training data as supervision to construct the loss function

[0050]

[0051] in, is the estimated 3D joint point of the nth person, is the true value of the three-dimensional joint point of the nth person. The above constraints are not enough to supervise the absolute position of the human body, so crowd constraints are introduced:

[0052]

[0053] Where std(·) is the standard deviation, To estimate the root joint in the 3D joints of the human body, l represents the crowd vector:

[0054]

[0055] in, Estimate the 3D joint points for the head of the nth person, The midpoint of the three-dimensional joint point of the nth person's ankle is estimated, and the overall loss function is:

[0056]

[0057] Among them, λ1, λ2, λ3, and λ4 are all weights of the loss function. The values of the above loss functions are back-propagated to realize the optimization process of network parameters until the network converges;

[0058] Furthermore, the specific method of step S4 includes:

[0059] S41. Large-Scale Crowd Image Acquisition: Acquire large-scale crowd images from publicly available online resources, either by capturing a single frame of a large-scale crowd image with a camera, or by simultaneously capturing different partial images with a camera array and then stitching the partial images together using an algorithm to construct a large-scale crowd image with billions of pixels.

[0060] S42. Inferring human mesh model parameters based on the network: The image obtained in step S41 is fed into an existing open source target detection network to obtain estimated human bounding box coordinates. The human bounding box and the image obtained in step S41 are fed into an existing open source two-dimensional pose estimation network to obtain estimated two-dimensional human joint points. The estimated human bounding box coordinates are converted into human position information b through step S31. n :

[0061]

[0062] Among them, f is the focal length of the image. For network images, the focal length is set to Where w is the image width and h is the image height. The confidence of some of the estimated two-dimensional joints of the human body is low. The two-dimensional joints with confidence lower than T are replaced with learnable tokens. The estimated two-dimensional joint coordinates of the human body are normalized and input as tokens into the network constructed and trained in steps S2 and S3. The posture feature q is obtained through the Transformer block. n , posture features and human position information constitute human features As nodes of the hypergraph, human body features are constructed into a hypergraph through step S33 After step S34, the hypergraph is iteratively updated to obtain updated human features. The updated human features at different scales are connected with the human position information to form the final human representation. After step S35, the estimated human body mesh model parameters of N human bodies are obtained through regression.

[0063] S43. 3D crowd reconstruction: The estimated human body mesh model parameters obtained in step S42 are An estimated three-dimensional human body mesh model of N human bodies is obtained through linear mixed skinning, thereby obtaining an estimated three-dimensional human body mesh model with accurate three-dimensional position distribution.

[0064] Compared with the existing technology, the beneficial effects of the present invention are: 1. Compared with the existing technology, the method proposed in the present invention can be applied to crowd reconstruction in large-scale scenes, and the number of human bodies reconstructed in a single time exceeds 50, which is significantly more than the existing single-view reconstruction method. 2. The present invention uses posture similarity to construct a hypergraph network, and realizes crowd relationship reasoning based on the hypergraph structure, thereby improving the reconstruction accuracy of occluded human bodies in single-view multi-person scenes. 3. The present invention proposes crowd constraints, and uses crowd characteristics to alleviate the depth ambiguity problem faced by single-view human body reconstruction, and improves the rationality of the spatial distribution of the reconstructed crowd. 4. The use process of the present invention only relies on a single RGB camera, which is low-cost and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of the present invention;

[0066] Figure 2 It is a schematic diagram of hypergraph relational reasoning;

[0067] Figure 3 It is a schematic diagram of the hypergraph structure;

[0068] Figure 4 It is a collective diagram of the crowd;

[0069] Figure 5 It is to reconstruct the top view of the crowd;

[0070] Figure 6 This is a schematic diagram of a three-dimensional human body mesh model, where (a) is the three-dimensional mesh model and (b) is the driving skeleton. DETAILED DESCRIPTION

[0071] The implementation process of the present invention is described in detail below with reference to the embodiments and accompanying drawings.

[0072] The method for single-view large-scale crowd 3D reconstruction based on hypergraph relational reasoning described in the present invention comprises the following steps:

[0073] Training data construction

[0074] (1) Figure 1 As shown, training data is constructed based on the existing dataset, and a dataset containing 3D human body annotations is selected. For datasets that only contain 3D joint point annotations in the 3D human body annotations, the 3D human body mesh model with skeleton skinning is used frame by frame to fit the 3D joint points of each human body in the annotations, and the corresponding human body mesh model parameters are obtained as the 3D human body annotations of the dataset. The 2D joint points, bounding boxes, 3D joint points, and human body mesh model parameters are stored in a unified format as the true value of the training data. The true value is used as a supervisory item in the subsequent network training to participate in the network parameter optimization. The training data is not used in the network inference process, as shown in FIG. Figure 1As shown in the figure, the original image in the existing 3D human dataset is used as a single-view large-scale crowd image and fed into the existing open source target detection network to obtain the estimated human bounding box coordinates. The estimated human bounding box and the original image are used to obtain the estimated human 2D joint points through the existing open source 2D pose estimation method. in is the estimated two-dimensional joint point of the nth human body, N K is the number of joint points in a single human body. The estimated human bounding box coordinates and the estimated two-dimensional joint points are used as the estimated values in the training data. The true values obtained by annotation and fitting of the three-dimensional human data set, the estimated values obtained based on the open source method, and the original images in the three-dimensional human data set together constitute the training data.

[0075] Network training

[0076] (2) Figure 1 As shown in the figure, an original image is extracted from the training data constructed in (1) as a single-view large-scale crowd image, and the estimated human bounding box and estimated human two-dimensional joint point input obtained by the open source method in (1) are Figure 1 The human feature preprocessing module in the human body converts the human body bounding box coordinates into human body position information b n :

[0077]

[0078] Where n is the nth human body, f is the focal length of the original image, (c x ,c y ) is the coordinate of the center of the bounding box relative to the center of the original image, and d is the size of the bounding box;

[0079] Due to the existence of occlusion and human interaction problems, the confidence of some of the two-dimensional joints of the human body estimated in (1) is low. The two-dimensional joints with confidence lower than T are replaced with learnable tokens. In this embodiment, T is set to 0.7. The coordinates of the two-dimensional joints of the human body are normalized and used as tokens to achieve Figure 1 Human feature preprocessing in .

[0080] (3) Input the token obtained in (2) Figure 1 The Transformer network in the figure obtains the posture features. Posture features and human body position information b n Constituent features of the human body n , as nodes of the hypergraph:

[0081]

[0082] Among them, v nis the human body feature of the nth person, which is also a node of the hypergraph, that is, Figure 2 In the input, m represents the dimension of the posture feature;

[0083] (4) Figure 2 As shown in the figure, the matching matrix is constructed, and the correlation of the posture is calculated based on the posture features obtained in (3), and the matching matrix is constructed.

[0084]

[0085] Elements in the matching matrix It is used to measure the posture similarity between the a-th human body and the b-th human body. The human bodies with similar postures are divided into the same group through the matching matrix. Considering the existence of groups of different sizes in the crowd, the size of the s-th group is recorded as K (s) , based on the matching matrix Get K (s) ×K (s) The high-density cofactor of (s) nodes as the same group, using hyperedges To represent this group, where t represents the t-th hyperedge, that is, the t-th group:

[0086]

[0087]

[0088] in, represents the tth hyperedge under the sth group size, ν represents the set of all hypergraph nodes, N is the total number of human bodies in the image selected in (2), that is, the number of hypergraph nodes, represents the pth node, Ω represents a subset of the hypergraph nodes, and the size of the set is K (s) , Represents the high-density cofactor of the matching matrix, Indicates the value of Ω when the subsequent expression is solved to obtain the maximum value, ‖·‖ 1,1 It represents the sum of the absolute values of all elements in the expression, which is solved by the greedy algorithm.

[0089] (5) For each node in the hypergraph (4) is performed for each node. Search for K that is closest to its posture (s) -1 other nodes, together forming a group, using the adjacency matrix To define the connection relationship of the edge of the hypergraph at the sth scale, so as to represent the connection relationship between groups, that is, Indicates that the i-th node is contained in the j-th hyperedge, otherwise The topological structure of the hypergraph is constructed, where for the 0th group size, the connection relationship between two nodes is constructed to obtain the adjacency matrix The hypergraph has N hyperedges and N nodes at any given group size, so we can construct Figure 2 The topology of the hypergraph shown The hypergraph structures of multiple scales are constructed, such as Figure 3 shown.

[0090] (6) Figure 2 As shown in the figure, since the connection between groups is constructed in the hypergraph structure, the group information can be obtained by using all the human features in the group. The group information is transmitted based on the hypergraph structure obtained in (5), and the human features obtained in (3) are aggregated into group features as the features of the hyperedge to achieve Figure 2 The hyperedge update process shown:

[0091]

[0092] Among them, e t is the t-th hyperedge, represents the p-th node, is a learnable multilayer perceptron, λ t for Figure 2 The contribution coefficient shown here represents the contribution of the p-th node to the t-th group, c t for Figure 2 The collective coefficient of the group shown, the intuitive meaning of the collective nature of the population is as follows Figure 4 As shown, λ t Defined as:

[0093]

[0094] in, is a learnable multilayer perceptron, is the rth node, c t Defined as:

[0095]

[0096] Among them, σ(·) is the sigmoid function, is the average characteristic of the t-th group, It is a learnable multilayer perceptron;

[0097] like Figure 2 The node update process shown in Figure 1 uses the features on all hyperedges related to the current node to update the human body features of the current node:

[0098]

[0099] in, is the pth human feature, e t is the t-th hyperedge, Represents The set of all hyperedges related to It is a learnable multi-layer perceptron that updates hyperedges and nodes multiple times on hypergraphs of all sizes to obtain updated human features and achieve Figure 1 The hypergraph relationship reasoning module in the hypergraph is updated iteratively, and the human features of different group sizes are connected with the human position information to form the final human representation of the nth human body. Since there are highly similar human postures in the group, these posture features can provide additional posture information and reconstruct a more accurate 3D human mesh model for the occluded human body.

[0100] (7) Input the final human body representation obtained in (6) Figure 1 The regression network in the human body pose parameters θ, human body shape parameters β and camera parameters [f c ,t x ,t y ], where f c is the estimated camera focal length, (t x ,t y ) is the estimated translation of the human mesh model relative to the center of the bounding box. The relative translation of the human mesh model is converted into an absolute translation based on the camera parameters and the human position information. x ,t y ,t z ):

[0101]

[0102] Where, f is the focal length of the original image, (c x ,c y ) is the coordinate of the center of the bounding box relative to the center of the original image, d is the size of the bounding box, and finally the human body mesh model parameters of all human bodies in the selected image in (2) are obtained

[0103] (8) The network is trained in an end-to-end manner, and the estimated 3D human mesh model and its corresponding estimated 3D joint points are obtained based on the estimated human mesh model parameters obtained in (6), such as Figure 1 As shown in Figure 2, the true value of the two-dimensional joint points in the training data is used as supervision to construct the reprojection loss function:

[0104]

[0105] Where N is the total number of human bodies in the image selected in (2), Π represents the projection of the estimated 3D joint points onto the 2D image, is the estimated 3D joint point of the nth person, t n is the absolute translation of the nth person, Represents the true value of the two-dimensional joint point of the nth person, The square of the two norm of the expression is used, and the true value of the human mesh model parameters in the training data is used as supervision to construct the loss function

[0106]

[0107] in, is the true value of the body shape parameters of the nth person, is the true value of the human body posture parameter of the nth person, using the true value of the three-dimensional joint points in the training data as supervision to construct the loss function

[0108]

[0109] in, is the estimated 3D joint point of the nth person, is the true value of the three-dimensional joint point of the nth person. The above constraints are not enough to supervise the absolute position of the human body, so crowd constraints are introduced:

[0110]

[0111] Where std(·) is the standard deviation, To estimate the root joint in the 3D joints of the human body, l represents the crowd vector:

[0112]

[0113] in, Estimate the 3D joint points for the head of the nth person, The ankle of the nth person estimates the midpoint of the 3D joint. The crowd constraint enables the network regression to obtain a more accurate human height, thereby obtaining a more accurate absolute translation of the human body. The overall loss function is:

[0114]

[0115] Among them, λ1, λ2, λ3, and λ4 are all weights of the loss function. In this embodiment, the values are λ1 = 5.0, λ2 = 5.0, λ3 = 5.0, and λ4 = 0.1. The values of the above loss functions are back-propagated to realize the optimization process of network parameters. The training data obtained by (1) is repeatedly extracted for training until the network converges. The trained network is used for subsequent reasoning process.

[0116] Network Inference

[0117] (9) Large-scale crowd image acquisition: Acquire large-scale crowd images based on public network resources, or use a camera to shoot a single-frame large-scale crowd image, or use a camera array to simultaneously shoot different local images, and use algorithms to stitch the local images to form a large-scale crowd image with billions of pixels, that is, Figure 1 Input single-view large-scale crowd image.

[0118] (10) The single single-view image obtained in (9) is fed into the existing open source target detection network to obtain the coordinates of the human body bounding box. The human body bounding box and the image obtained in (9) are fed into the existing open source two-dimensional posture estimation network to obtain the two-dimensional joint points of the human body. The bounding box coordinates of the nth human body are converted into human body position information b through (2) n :

[0119]

[0120] Among them, f is the focal length of the original image, for network images Where w is the image width, h is the image height, and the confidence of some of the estimated two-dimensional joints of the human body is low. The estimated two-dimensional joints with confidence lower than T are replaced with learnable tokens. In this embodiment, T is set to 0.7. The coordinates of the estimated two-dimensional joints of the human body are normalized and used as token input (3), as shown in Figure 1 As shown, the token of the nth person passes through the transformer block to obtain the posture feature q n , posture features and human position information constitute human features The human features of all human bodies in a single image obtained in (9) are used as nodes of a hypergraph and constructed through (4) and (5). The nodes and hyperedges of the hypergraph are iteratively updated through (6). For the nth human body, the human features at different scales are connected with the human position information to form the final human body representation. The final human representation of all human bodies in a single image obtained in (9) is passed through the regression network to obtain the estimated human mesh model parameters

[0121] (11) The estimated human body mesh model parameters obtained in (10) are subjected to linear mixed skinning to obtain an estimated three-dimensional human body mesh model of N human bodies, and an estimated three-dimensional human body mesh model with accurate three-dimensional position distribution is obtained. The top view result of the estimated three-dimensional human body mesh model is as follows: Figure 5 shown.

[0122] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely intended to further illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A single-view large-scale crowd 3D reconstruction method based on hypergraph relational reasoning, characterized by: The method comprises the following steps: S1. Training Data Construction: The training data consists of the ground truth values obtained through annotation and fitting of the 3D human body dataset, the estimated values obtained using open-source methods, and the original images in the 3D human body dataset. S2. Reconstruction Network Construction: The reconstruction network consists of three parts: a feature extraction module, a hypergraph relationship reasoning module, and a regression network. The human feature preprocessing module processes the human body bounding box into human body position information and processes the two-dimensional human body joints into tokens. The feature extraction module is composed of a Transformer network to implement the human body feature extraction process. The hypergraph relationship reasoning module models group relationships based on human body features and obtains group characteristics. The human body features are updated based on group characteristics, and the collectivity and interaction of groups are used to obtain more accurate human body features. The regression network is used to regress the updated human body features to obtain human body mesh model parameters, which are subsequently used to achieve three-dimensional human body reconstruction. S3. Network training: Using the estimated human bounding box and estimated two-dimensional human joint points in the training data constructed in step S1 as the reconstruction network input, the human feature preprocessing module constructed in step S2 is used to obtain posture features and human position information. The human features are generated through the Transformer network constructed in step S2. A hypergraph is constructed based on the similarity of human postures and human features. The hypergraph is iteratively updated to update the human features. The human mesh model parameters are regressed based on the human features and position information to reconstruct the three-dimensional crowd. The loss function is constructed using the reconstruction results and the true values in the training data constructed in step S1 to achieve supervision. The network parameters are optimized by backpropagation until the network converges. The trained network is used for subsequent inference processes. S4. Single-view large-scale crowd reconstruction: Collect images of large-scale crowd scenes from a single viewpoint, obtain the estimated human bounding box and estimated two-dimensional human joint points through existing open source methods as input to the reconstruction network constructed in step S2, obtain the human body parameter model through network forward reasoning, and reconstruct the three-dimensional crowd using the skeleton-skinned three-dimensional human mesh model.

2. The method for single-view large-scale crowd 3D reconstruction based on hypergraph relationship reasoning according to claim 1, characterized in that: The true value obtained based on the annotation and fitting of the three-dimensional human body dataset in step S1 is obtained by the following method: Select a dataset containing 3D human body annotations. For datasets containing only 3D joint annotations, use a human mesh model with a unified topological structure frame by frame. Driven by skeleton skinning, fit the 3D joint points of each human body in the annotations to obtain the corresponding human mesh model parameters as the 3D human body annotations of the dataset. The two-dimensional joint points, bounding boxes, three-dimensional joint points, and human mesh model parameters are stored in a unified format as the true values of the training data.

3. The method for single-view large-scale crowd 3D reconstruction based on hypergraph relationship reasoning according to claim 1, characterized in that: The estimated value obtained based on the open source method in step S1 refers to sending the original image in the existing 3D human data set into the existing open source object detection network to obtain the estimated human bounding box coordinates, and using the estimated human bounding box and the original image to obtain the estimated human 2D joint points through the existing open source 2D pose estimation method. Where n∈[1,…,N], N is the number of human bodies, N K is the number of joint points in a single human body, the estimated human bounding box coordinates and the estimated two-dimensional joint points are used as the estimated values in the training data.

4. The method for single-view large-scale crowd 3D reconstruction based on hypergraph relationship reasoning according to claim 1, characterized in that: The specific method of step S2 includes: S21. Human Feature Extraction Module: The Transformer network consists of four identically structured Transformer blocks. Learnable spatial position embedding vectors are added to the inputs of the first two Transformer blocks, and their outputs are fed into the next two Transformer blocks. The Transformer network maps the inputs to a high-dimensional space, which serves as the output of the Human Feature Extraction Module. S22. Hypergraph Relationship Reasoning Module: The output of step S21 is used to construct a hypergraph. The hypergraph consists of nodes and hyperedges. Each hypergraph is defined as in is the set of nodes, ε is the set of hyperedges, and the hypergraphs of multiple scales are defined as The superscript indicates the scale number, and the hypergraph of the sth scale is defined as where ε (s) is the hyperedge of size s, defined as in Indicates the Mth scale under the sth scale s Hyperedges, the hypergraph update is divided into two stages: updating hyperedges from nodes and updating nodes from hyperedges. The update process is implemented by multi-layer perceptrons. After multiple iterations, the updated nodes are obtained. S23. Regression network: The regression network is composed of a multi-layer perceptron, which maps the output obtained in step S22 back to a low-dimensional space as the output of the regression network.

5. The method for single-view large-scale crowd 3D reconstruction based on hypergraph relationship reasoning according to claim 3 is characterized in that: The specific method of step S3 includes: S31. Human feature preprocessing: Convert the estimated human bounding box coordinates obtained in step S1 into human position information b n : Where n is the nth human body, f is the focal length of the original image, (c x ,c y ) is the coordinate of the center of the bounding box relative to the center of the original image, and d is the size of the bounding box; The confidence of some of the 2D joints of the human body estimated in step S1 is replaced with learnable tokens by the 2D joints with confidence lower than T. The coordinates of the 2D joints of the human body are normalized and used as tokens. S32. Human body feature extraction: Input the token obtained in step S31 into the Transformer network constructed in step S21 to obtain posture features Posture features and human body position information b n Constituent features of the human body n , as nodes of the hypergraph: Among them, v n is the human body feature and is also a node of the hypergraph, and m represents the dimension of the posture feature; S33. Hypergraph construction: using adjacency matrix To define the connection relationship of the edge of the hypergraph under the s-th scale, that is Indicates that the i-th node is contained in the j-th hyperedge, otherwise The topological structure of the hypergraph is formed, and the hypergraph is defined as the form of step S22; The hypergraph topology construction includes the following steps, and the hypergraph topology construction mentioned later is the same as this step: S331. Calculate the correlation of postures based on posture features and construct a matching matrix in, is an element in the matching matrix, used to measure the posture similarity between the a-th person and the b-th person, q a is the posture feature of the ath person, q b is the posture feature of the bth person, T is the matrix transpose, ‖·‖2 is the two-norm, and the matching matrix is used to divide people with similar postures into the same group; S332. Considering the existence of groups of different sizes in the crowd, the size of the sth group is recorded as K (s) , based on the matching matrix obtained in step S331 Get K (s) ×K (s) The high-density cofactor of (s) nodes as the same group, using hyperedges To represent this group, where t represents the t-th hyperedge, that is, the t-th group: in, represents the tth hyperedge under the sth group size, represents the set of all hypergraph nodes, N is the number of human bodies, that is, the number of hypergraph nodes, represents the pth node, Ω represents a subset of the hypergraph nodes, and the size of the set is K (s) , Represents the high-density cofactor of the matching matrix, Indicates the value of Ω when the subsequent expression is solved to obtain the maximum value, ‖·‖ 1,1 The sum of the absolute values of all elements in the expression is solved by the greedy algorithm; S333. For each node in the hypergraph All proceed to step S332, where for the 0th group size, the connection relationship between each two nodes is constructed to obtain the adjacency matrix A hypergraph has N hyperedges and N nodes at any given group size, thus constructing the topological structure of the hypergraph S34. Hypergraph relationship reasoning: Based on the hypergraph structure obtained in step S33, group information is transferred, and the human features on the nodes obtained in step S32 are aggregated into group features as the features of the hyperedge: Among them, e t is the t-th hyperedge, represents the p-th node, is a learnable multilayer perceptron, λ t is the contribution coefficient, which indicates the contribution of the p-th node to the t-th group, c t is the collective coefficient of the group, λ t Defined as: in, is a learnable multilayer perceptron, is the rth node, c t Defined as: Among them, σ(·) is the sigmoid function, is the average characteristic of the t-th group, It is a learnable multilayer perceptron; Use the features on all hyperedges related to the current node to update the human features of the current node: in, is the pth human feature, e t is the t-th hyperedge, Represents The set of all hyperedges related to It is a learnable multi-layer perceptron that updates hyperedges and nodes multiple times on hypergraphs of all sizes to obtain updated human features. After iterative updates of the hypergraph, the human features at different group sizes are connected with the human position information to form the final human representation Where n represents the nth person; S35. Human body posture parameter regression: The human body representation obtained in step S34 is input into the regression network defined in step S23 to obtain human body posture parameters θ, human body shape parameters β and camera parameters [f c ,t x ,t y ], where f c is the estimated camera focal length, (t x ,t y ) is the estimated translation of the human mesh model relative to the center of the bounding box. The relative translation of the human mesh model is converted into an absolute translation based on the camera parameters and the human position information. x ,t y ,t z ): Where, f is the focal length of the original image, (c x ,c y ) is the coordinate of the center of the bounding box relative to the center of the original image, d is the size of the bounding box, and the estimated human body mesh model parameters of N human bodies are finally obtained. S36. Loss function construction: The network is trained in an end-to-end manner. After the network structure constructed in step S2, the 157-dimensional estimated human mesh model parameters output in step S35 are obtained. Based on the estimated human mesh model parameters, an estimated 3D human mesh model and its corresponding estimated 3D joint points are obtained. The true values of the 2D joint points in the training data obtained in step S1 are used as supervision to construct a reprojection loss function: Among them, Π represents the projection of the estimated three-dimensional joint point to the two-dimensional image, is the estimated 3D joint point of the nth person, t n is the absolute translation of the nth person, Represents the true value of the two-dimensional joint point of the nth person, The square of the two norm of the expression is used, and the true value of the human mesh model parameters in the training data is used as supervision to construct the loss function in, is the true value of the body shape parameters of the nth person, is the true value of the human body posture parameter of the nth person, using the true value of the three-dimensional joint points in the training data as supervision to construct the loss function in, is the estimated 3D joint point of the nth person, is the true value of the three-dimensional joint point of the nth person. The above constraints are not enough to supervise the absolute position of the human body, so crowd constraints are introduced: Among them, std(·) is the standard deviation, To estimate the root joint in the 3D joints of the human body, l represents the crowd vector: in, Estimate the 3D joint points for the head of the nth person, The midpoint of the three-dimensional joint point of the nth person's ankle is estimated, and the overall loss function is: Among them, λ1, λ2, λ3, and λ4 are all weights of the loss function. The values of the above loss functions are back-propagated to realize the optimization process of network parameters until the network converges.

6. The method for single-view large-scale crowd 3D reconstruction based on hypergraph relationship reasoning according to claim 5, characterized in that: The specific method of step S4 includes: S41. Large-Scale Crowd Image Acquisition: Acquire large-scale crowd images from publicly available online resources, either by capturing a single frame of a large-scale crowd image with a camera, or by simultaneously capturing different partial images with a camera array and then stitching the partial images together using an algorithm to construct a large-scale crowd image with billions of pixels. S42. Inferring human mesh model parameters based on the network: The image obtained in step S41 is fed into an existing open source target detection network to obtain estimated human bounding box coordinates. The human bounding box and the image obtained in step S41 are fed into an existing open source two-dimensional pose estimation network to obtain estimated two-dimensional human joint points. The estimated human bounding box coordinates are converted into human position information b through step S31. n : Among them, f is the focal length of the image. For network images, the focal length is set to Where w is the image width, h is the image height, and the confidence of some of the two-dimensional joints of the human body is estimated. The two-dimensional joints with confidence lower than T are replaced with learnable tokens. The estimated two-dimensional joint coordinates of the human body are normalized and input as tokens into the network constructed and trained in steps S2 and S3. The posture feature q is obtained through the Transformer block. n , posture features and human position information constitute human features As nodes of the hypergraph, human body features are constructed into a hypergraph through step S33 After step S34, the hypergraph is iteratively updated to obtain updated human features. The updated human features at different scales are connected with the human position information to form the final human representation. After step S35, the estimated human body mesh model parameters of N human bodies are obtained through regression. S43. 3D crowd reconstruction: The estimated human body mesh model parameters obtained in step S42 are An estimated three-dimensional human body mesh model of N human bodies is obtained through linear mixed skinning, thereby obtaining an estimated three-dimensional human body mesh model with accurate three-dimensional position distribution.

Images