Three-dimensional human skeleton fitting method and system based on multi-phase association graph and combination search

Abstract

The application discloses a three-dimensional human skeleton fitting method and system based on a multi-camera joint association graph and combined search, and relates to the three-dimensional human skeleton fitting method, which comprises the following steps: S1, acquiring images collected by multiple cameras, detecting human two-dimensional joint nodes by using a two-dimensional pose estimation algorithm, and projecting the two-dimensional joint nodes into three-dimensional rays; S2, taking the joint nodes as nodes and the geometric consistency between three-dimensional rays of different cameras as edge weights, constructing a multi-camera joint node association graph; S3, performing group clustering on the cross-camera joint nodes with high geometric consistency on the association graph by using a breadth-first search, and forming a candidate skeleton cluster; S4, reserving the first K joint nodes with the highest geometric consistency with the candidate skeleton cluster in each camera, and obtaining candidate joint node pairs; S5, using a depth-first search (DFS) to enumerate the cross-camera joint node pairs, and screening optimal joint node pairs in combination with a pruning strategy; and S6, performing triangulation processing on the screened optimal joint node pairs, obtaining three-dimensional skeleton points, and outputting a three-dimensional human skeleton structure.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for fitting three-dimensional human skeletons based on multi-camera correlation graphs and combined search. Background Technology

[0002] In the field of computer vision, 3D human skeleton fitting technology is of great significance for motion capture, virtual reality, and human-computer interaction. Currently, the main multi-camera 3D skeleton fitting methods in this field are as follows:

[0003] Firstly, direct triangulation methods based on binocular or depth cameras can directly obtain 3D key points through depth information or binocular matching, but their dependence on special hardware leads to high costs and limitations in applicability.

[0004] Secondly, methods based on human body model fitting such as SMPL can generate human skeletons and shapes with reasonable structures. However, their sensitivity to initial values ​​and computational resources makes it difficult to guarantee real-time performance.

[0005] Thirdly, there are multi-view joint fitting methods based on graph optimization, such as 4D Association, VNect, and OpenPose's multi-view extension. These methods exhibit good robustness in handling multi-person scenes and occlusion situations; however, the complex global optimization process makes them computationally expensive, making it difficult to meet the needs of real-time interaction.

[0006] For example, patent CN111797714A discloses a multi-viewpoint human motion capture method based on keypoint clustering, including: detecting human keypoints in multi-viewpoint images; establishing a relationship graph of human keypoints based on image information and geometric relationships; associating human keypoints from different viewpoints based on the keypoint relationship graph and clustering algorithm to find keypoints belonging to the same person; and reconstructing the 3D position of each human keypoint based on the association results. While the aforementioned patent can establish associations between human keypoints in images from different viewpoints through keypoint clustering, it cannot guarantee that all 2D skeletons are used.

[0007] In summary, existing technologies suffer from the following technical problems: 1. Insufficient robustness in cross-view keypoint matching: Existing methods largely rely on global optimization, making them susceptible to noise interference. 2. Excessive combination space: When multiple cameras and multiple candidate keypoints exist, the matching combinations increase exponentially, resulting in excessively high computational costs. 3. Insufficient real-time performance: Complex graph optimization is difficult to apply in real-time interactive scenarios.

[0008] Therefore, in view of the above-mentioned prior art, the present invention provides a method and system for fitting a three-dimensional human skeleton based on multi-camera correlation graphs and combined search. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and system for fitting three-dimensional human skeletons based on multi-camera correlation graphs and combined search. This method can achieve robust three-dimensional human skeleton fitting in multi-person or occluded scenes, significantly reducing the combination space and improving real-time performance and accuracy. It is applicable to fields such as motion capture, virtual reality, and human-computer interaction.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A 3D human skeleton fitting method based on multi-camera correlation graphs and combined search includes:

[0012] S1. Acquire images from multiple cameras and use a two-dimensional pose estimation algorithm to detect two-dimensional joints of the human body. Then, back-project the detected two-dimensional joints into three-dimensional rays.

[0013] S2. Using key points as nodes and the geometric consistency between 3D rays from different cameras as edge weights, construct a multiphase mechanism node association graph;

[0014] S3. Cluster cross-phase mechanism nodes with high geometric consistency on the association graph using breadth-first search (BFS) to form candidate skeleton clusters;

[0015] S4. Within each camera, retain the top K key points with the highest geometric consistency to the candidate skeleton cluster to obtain candidate key point pairs;

[0016] S5. Depth-first search (DFS) is used to enumerate cross-phase key node pairs, and the optimal key pair is selected by combining it with a pruning strategy.

[0017] S6. Triangulate the selected optimal joint point pairs to obtain three-dimensional skeleton points and output the three-dimensional human skeleton structure.

[0018] Furthermore, the edge weight calculation in step S2 is based on the shortest distance between three-dimensional rays and the confidence level, and is expressed as:

[0019]

[0020] Among them, w ij The edge weights are represented by i and j, respectively, which represent camera i and camera j. This represents the average score of all matched keypoint pairs; g ij Indicates the geometric consistency of the matched keypoint pairs; r ij S represents the skeleton of camera i i And the frame of camera j S j The proportion of matched joints; α and β both represent weight values.

[0021] Furthermore, the geometric consistency g of the matched keypoint pairs ij Represented as:

[0022]

[0023] Where, m ij Indicates the skeleton S of camera i i And the frame of camera j S j The number of key points matched; k represents the index of the matched key point pair; a i,k This indicates that the k-th matching joint pair is located in the skeleton S of camera i. i The key points; a j,k Indicates with a i,k Matching skeleton S in camera j j The two are a pair of anchor points of the same type from different cameras.

[0024] Furthermore, the average score of all matched keypoint pairs Represented as:

[0025]

[0026]

[0027] Among them, s k represents the score of the k-th matched key pair; s represents the score of each matched key pair; d represents the distance between two key pairs in the pre-computed distance matrix; T d This represents the maximum tolerance threshold between two joints.

[0028] Furthermore, the ratio r of the matching joints ij Represented as:

[0029]

[0030] Among them, |A i | Represents skeleton S i Total number of joints; |A j | Represents skeleton S j The total number of joints.

[0031] Furthermore, in step S4, the top K keypoints with the highest geometric consistency with the candidate skeleton cluster are retained within each camera, as follows:

[0032]

[0033] Among them, K i w represents the set of the top K key nodes of the high affinity graph selected within camera i;ij The affinity weight between node u and node v is derived from the previously calculated affinity graph; v represents a neighboring key point that may match u in other cameras; N(u) represents the set of neighboring nodes connected to node u in the association graph, which are the 2D skeleton key points that match u with higher scores in different cameras; u represents the reference key key point in the current skeleton.

[0034] Furthermore, in step S5, the depth-first search (DFS) is used to enumerate cross-phase control node pairs, which is represented as follows:

[0035]

[0036] Where S(C) represents the overall consistency score of the current combination; C represents the cross-camera keypoint combination selected in the current DFS search path; v represents the neighboring keypoints; u represents the current keypoint;

[0037] The optimal key pair is selected by combining pruning strategies, and is represented as follows:

[0038]

[0039] Among them, T S This represents the maximum similarity threshold between two skeletons; Prune represents the pruning strategy.

[0040] Furthermore, after obtaining the three-dimensional skeleton points in step S6, the method further includes: performing proportional and angular constraint processing on the three-dimensional skeleton points.

[0041] Furthermore, after obtaining the three-dimensional skeleton points in step S6, the method further includes: performing nonlinear optimization on the three-dimensional skeleton points based on the human statistical model SMPL.

[0042] Correspondingly, a 3D human skeleton fitting system based on multi-camera correlation maps and combined search is also provided, including:

[0043] The acquisition module is used to acquire images from multiple cameras and use a two-dimensional pose estimation algorithm to detect two-dimensional joints of the human body, and then back-project the detected two-dimensional joints into three-dimensional rays.

[0044] The module is used to construct a multiphase mechanism node association graph with key points as nodes and geometric consistency between 3D rays from different cameras as edge weights.

[0045] The clustering module is used to cluster cross-phase mechanism nodes with high geometric consistency on the association graph through breadth-first search (BFS) to form candidate skeleton clusters.

[0046] The first screening module is used to cluster cross-phase mechanism nodes with high geometric consistency on the association graph through breadth-first search (BFS) to form candidate skeleton clusters.

[0047] The second filtering module is used to enumerate cross-phase key node pairs using depth-first search (DFS) and combine it with a pruning strategy to filter out the optimal key node pairs.

[0048] The processing module is used to triangulate the selected optimal joint point pairs to obtain three-dimensional skeleton points and output the three-dimensional human skeleton structure.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. High efficiency and robustness: By using BFS clustering and Top-K screening, cross-view key points with high geometric consistency are aggregated to reduce mismatches and improve the robustness of skeleton fitting; combined with pruning strategies, low-scoring candidate matches are eliminated to further reduce the risk of error propagation.

[0051] 2. Strong real-time performance: By reducing the combinatorial space, the computational complexity is reduced from exponential to polynomial level; DFS enumeration and pruning strategies are adopted to ensure real-time operation in multi-person, multi-view scenarios, meeting the needs of real-time interaction.

[0052] 3. High ease of use: It only requires camera intrinsic parameters to run, without the need for complex extrinsic parameter optimization, which simplifies the system deployment and usage process and lowers the threshold for use.

[0053] 4. Wide applicability: It is suitable for multi-person scenarios and complex occlusion scenarios, and has significant advantages in motion capture, virtual reality, human-computer interaction and other fields, which can effectively improve the interactive experience and application effect in these fields. Attached Figure Description

[0054] Figure 1 This is a flowchart of the three-dimensional human skeleton fitting method based on multi-camera correlation graph and combined search provided in Embodiment 1;

[0055] Figure 2 This is the overall system framework diagram provided in Implementation Example 1;

[0056] Figure 3 This is a schematic diagram of multi-camera keypoint association based on affinity graph provided in Implementation Example 1;

[0057] Figure 4 This is a schematic diagram of candidate skeleton clusters obtained by BFS clustering provided in Example 1;

[0058] Figure 5 This is the flowchart of Top-K filtering and DFS enumeration provided in Example 1. Detailed Implementation

[0059] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0060] The purpose of this invention is to address the shortcomings of existing technologies by providing a method and system for fitting a three-dimensional human skeleton based on multi-camera correlation maps and combined search.

[0061] Example 1

[0062] This embodiment provides a 3D human skeleton fitting method based on multi-camera correlation maps and combined search, such as... Figures 1-2 As shown, it includes:

[0063] S1. Acquire images from multiple cameras and use a two-dimensional pose estimation algorithm to detect two-dimensional joints of the human body. Then, back-project the detected two-dimensional joints into three-dimensional rays.

[0064] S2. Using key points as nodes and the geometric consistency between 3D rays from different cameras as edge weights, construct a multiphase mechanism node association graph;

[0065] S3. Cluster cross-phase mechanism nodes with high geometric consistency on the association graph using breadth-first search (BFS) to form candidate skeleton clusters;

[0066] S4. Within each camera, retain the top K key points with the highest geometric consistency to the candidate skeleton cluster to obtain candidate key point pairs;

[0067] S5. Depth-first search (DFS) is used to enumerate cross-phase key node pairs, and the optimal key pair is selected by combining it with a pruning strategy.

[0068] S6. Triangulate the selected optimal joint point pairs to obtain three-dimensional skeleton points and output the three-dimensional human skeleton structure.

[0069] In step S1, images acquired by multiple cameras are obtained, and two-dimensional pose estimation algorithms are used to detect two-dimensional joints of the human body. The detected two-dimensional joints are then back-projected into three-dimensional rays.

[0070] Each camera backprojects 2D keypoints detected into 3D rays, forming a cross-camera candidate ray set. The specific representation is as follows:

[0071] For each camera C i Using a two-dimensional pose estimation network to obtain the input image I iHuman body key points are detected and pixel coordinates and confidence scores are obtained. i,k ,v i,k ,c i,k ).

[0072] Given the intrinsic parameter matrix K of the camera i With external parameters [R] i |t i Each two-dimensional key point can be back-projected into a three-dimensional spatial ray, represented as:

[0073]

[0074]

[0075] Transforming it to the world coordinate system, we obtain the ray direction, represented as:

[0076]

[0077] The center position of the camera is represented as: ;

[0078] The ray equation is then expressed as:

[0079]

[0080] This ultimately forms the cross-camera candidate ray set:

[0081] This ray set serves as the foundational input for subsequently constructing the multi-camera affinity graph, used to compute geometric consistency across camera keypoints.

[0082] Where i represents the camera index; k represents the key point index; This represents the confidence threshold.

[0083] In step S2, a multiphase mechanism node association graph is constructed using key points as nodes and the geometric consistency between 3D rays from different cameras as edge weights; the edge weights are calculated based on the shortest distance between 3D rays and the confidence level.

[0084] like Figure 3 As shown, multiple (e.g., 4) cameras can acquire the corresponding 2D skeleton joints in the human image. The 2D joints are then used to calculate their 3D rays in space using camera intrinsic parameters. The distance of each ray is calculated to all 3D rays outside the view of that camera. The specific calculation formula is as follows:

[0085] Let the k-th ray from camera i be represented as:

[0086]

[0087] The l-th ray from camera j is represented as:

[0088]

[0089] Among them, o i o j Indicates the positions of the two cameras in the world coordinate system; d i,k d j,l Let λ represent the normalized ray direction vector; λ and μ represent the scale parameters on the ray, and λ > 0, μ > 0.

[0090] The shortest distance between two space rays is expressed as:

[0091]

[0092] The formula represents the minimum Euclidean distance between two infinitely extended straight lines. Geometrically, the numerator is the projection length of the vector connecting the origins of the two rays onto the direction of the normal vector, and the denominator is the sine of the angle between the directions of the two rays.

[0093] The confidence score is calculated using the confidence score formula, and all confidence scores are stored in the affinity graph Affinity Gaphy. The closer the distance, the higher the confidence score, and the more likely it corresponds to the same 3D skeleton point.

[0094] In this embodiment, the edge weights calculated in the affinity graph weight formula mainly consist of three parts: the proportion of matching keypoint pairs, the average similarity score (based on anchor distance), and geometric consistency.

[0095] 1. The proportion of matching joints is expressed as:

[0096]

[0097] Where, m ij Indicates the skeleton S of camera i i And the frame of camera j S j The number of anchor points (i.e., key points) matched in the middle; |A i | Represents skeleton S i Total number of joints; |A j | Represents skeleton S j The total number of joints; r ij S represents the skeleton of camera i i And the frame of camera j S j The matched joint proportions, i.e., the skeleton S of camera i. i And the frame of camera j S jThe ratio of the smaller of the number of matched joints to the total number of joints in the skeleton is used to measure the similarity between two skeletons.

[0098] 2. The matching score (based on anchor distance) is as follows:

[0099] The score for each matched keypoint pair is represented as follows:

[0100]

[0101] Where d represents the distance in physical space between the 3D rays corresponding to the two joints, which has been calculated using the formula above and stored in the correlation matrix; T d represents the maximum tolerance threshold between two joints; s represents the score of each matched joint pair. A joint pair refers to the matching relationship between two two-dimensional joints of the same human body part detected by two different cameras in a multi-camera system.

[0102] The average score of all matched keypoint pairs is expressed as:

[0103]

[0104] Among them, s k This represents the score of the k-th matched keypoint pair; This represents the average score of all matched keypoint pairs.

[0105] 3. Compute the geometric consistency of the matched keypoint pairs, represented as:

[0106]

[0107] Where k represents the index of the matched keypoint pair; a i,k This indicates that the k-th matching joint pair is located in the skeleton S of camera i. i The key points; a j,k Indicates with a i,k Matching skeleton S in camera j j The two are a pair of anchor points of the same type from different cameras; g ij This indicates the geometric consistency of the matched keypoint pairs.

[0108] 4. The edge weight (Affinity) is calculated as follows:

[0109]

[0110] Among them, w ij The edge weights are represented by α and β, respectively; i and j represent camera i and camera j, respectively; α and β both represent weight values, α = 0.7 and β = 1.5.

[0111] In step S3, breadth-first search (BFS) is used to cluster cross-phase mechanism nodes with high geometric consistency on the association graph to form candidate skeleton clusters.

[0112] Breadth-first search (BFS) is used to cluster cross-view joints with strong geometric consistency on the association graph, forming candidate skeleton clusters. Specifically:

[0113] On the Affinity Graph, perform BFS clustering based on ray distance / weight, and process "single node type" class by class. This allows a 2D node to belong to multiple candidate clusters (soft / overlap), thereby reducing missed detections and controlling combinatorial complexity.

[0114] Objective: In (G) k Find a candidate cluster {C} = {C1, ..., C} on the array. M}, each cluster (C m ⊆V k () represents a set of 2D detections that may correspond to the same 3D joint across cameras.

[0115] Basic approach (allowing overlap / multiple inclusions):

[0116] 1. Select a node v that has not yet been "fully explored" or is used as a cluster seed as the starting point for BFS.

[0117] 2. Start from v and perform BF (or breadth-first expansion), expanding only along edges that satisfy the adjoining condition (close distance or close edge weight) until the queue is exhausted or the cluster size limit is reached (e.g., each cluster takes at most from the top-R nodes of each camera to avoid explosion).

[0118] 3. Denote the set of nodes found by BFS as a candidate cluster C. Record the distance and weight of each node in this cluster from the cluster center / seed for subsequent Top-K selection.

[0119] 4. Do not delete directly from contained nodes; allow nodes to participate in multiple clusters (soft overlap). However, to avoid useless duplication, maintain a "coverage count" or a low-probability suppression.

[0120] like Figure 4 As shown, the system has three cameras, each capable of detecting multiple two-dimensional human skeletons. Figure 4The left side of the middle section shows the two-dimensional detection results of each camera: different colors (red, green, blue) represent the detection results from camera 1, camera 2, and camera 3, respectively; each shape (circle, triangle, rhombus, etc.) represents a different two-dimensional skeleton individual (Skeleton1, Skeleton2, Skeleton3); each skeleton contains multiple numbered joints (e.g., 0, 1, 2 represent different parts, such as head, shoulder, hip, etc.).

[0121] Figure 4 The right side shows the three-dimensional ray interaction relationships formed by the back projection of these two-dimensional key points from different cameras in space. The system performs breadth-first search (BFS) clustering on the multi-camera affinity graph, aggregating cross-phase key nodes that are geometrically close in spatial location and have high ray consistency into candidate skeleton clusters.

[0122] exist Figure 4 In the diagram, the yellow dashed boxes represent two candidate skeleton clusters (candidate skeleton cluster 1 and candidate skeleton cluster 2). It can be seen that candidate skeleton cluster 1 simultaneously contains a red "circle" node from camera 1, a green "triangle" node from camera 2, and a blue "circle" node from camera 3. These three nodes are geometrically close, suggesting they may correspond to the same real 3D joint. Candidate skeleton cluster 2 contains a red "diamond" node from camera 1, a green "diamond" node from camera 2, and a blue "diamond" node from camera 3, forming another set of candidate matches.

[0123] It should be noted that the same two-dimensional joint point can be contained by multiple candidate skeleton clusters at the same time (as shown in the figure, some nodes are located in the overlapping area of ​​the two yellow dashed boxes).

[0124] This design allows the system to further filter the optimal combination in subsequent steps by matching confidence and geometric consistency, avoiding missed detections or mismatches caused by 2D detection errors or occlusion. At the same time, since clustering only matches at the keypoint level (rather than the entire skeleton), computational complexity is significantly reduced, providing an efficient candidate space for subsequent Top-K screening and DFS combinatorial search.

[0125] In step S4, the top K joints with the highest geometric consistency with the candidate skeleton cluster are retained in each camera to obtain candidate joint pairs, which significantly reduces the subsequent combination space.

[0126] In each candidate skeleton cluster, select the top-K keypoint pairs, such as... Figure 2In the process, the 0th joint of the skeleton 1 of camera 1 may belong to the same three-dimensional joint as the 0th joint of the skeleton 1 or skeleton 2 of camera 2. Similarly, the 0th joint of the skeleton 1 or skeleton 2 of camera 3 may be the same three-dimensional space point as the 0th joint of the skeleton 1 of camera 1. Therefore, under the constraint of a certain distance, K joints are selected as candidate joint pairs.

[0127] The formula for candidate keypoint pairs obtained from the first K selected keypoints within each camera is expressed as follows:

[0128]

[0129] Among them, K i w represents the set of the top K key nodes of the high affinity graph selected within camera i; ij The affinity weight between node u and node v is derived from the previously calculated affinity graph; v represents a neighboring key point that may match u in other cameras; N(u) represents the set of neighboring nodes connected to node u in the association graph, which are the 2D skeleton key points that match u with higher scores in different cameras; u represents the reference key key point in the current skeleton.

[0130] In step S5, depth-first search (DFS) is used to enumerate cross-phase key node pairs and calculate the overall consistency score. Combined with a pruning strategy, low-scoring key node pairs are eliminated in advance, and the optimal key node pairs are selected to improve efficiency.

[0131] like Figure 5 As shown, in the candidate skeleton cluster, the corresponding cross-view skeleton is found based on the Top-K keypoint pairs. A depth-first search (DFS) strategy is adopted, and the skeleton is then processed according to... Figure 3 The IDs of the human skeleton on the left side of the center are used to calculate the score of the entire skeleton. During the calculation, if the scores of multiple joint point pairs are far below the similarity threshold, the skeleton across the view is directly excluded. Figure 3 On the right side, Skeleton 1 in camera 1 has a higher overall confidence level than Skeleton 1 in both camera 2 and camera 3. However, Skeleton 2 in camera 2 and camera 3 only has a higher confidence level at keypoints. During calculation, if multiple confidence levels are low, it is considered a mismatch. The depth search is then interrupted to find other cross-view matching skeletons.

[0132] Using Depth-First Search (DFS) to enumerate cross-phase node pairs, the representation is as follows:

[0133]

[0134] Where S(C) represents the overall consistency score of the current combination; C represents the cross-camera key combination selected in the current DFS search path; v represents the neighboring key; and u represents the current key.

[0135] The optimal key pair is selected by combining pruning strategies, and is represented as follows:

[0136]

[0137] Among them, T S This represents the maximum similarity threshold between two skeletons; Prune represents the pruning strategy.

[0138] In step S6, the selected optimal joint point pairs are triangulated to obtain three-dimensional skeleton points, and the three-dimensional human skeleton structure is output.

[0139] Triangulation is performed on the final selected cross-camera combination to obtain 3D skeleton points.

[0140] Table 1 below shows a schematic diagram of the comparison between this solution and existing technologies.

[0141] Table 1 shows the results of comparing this solution with existing technologies.

[0142] Comparison Dimensions This solution (BFS + Top-K + DFS + pruning) 4D Association (Hungarian Matching) Cross-camera matching method Local clustering + Top-K screening, performing combined search only within candidate clusters. Global one-to-one matching requires constructing a complete cost matrix. Core calculation steps BFS clustering, Top-K screening, DFS enumeration + pruning Construct an N×N cost matrix + Hungarian matching Algorithm complexity Approximately O(M×N×K) (approximately polynomial order) O(N³) (cubic growth) Multiplayer scene expansion By using cluster partitioning for parallel processing, the complexity increases linearly. The Hungarian matrix becomes ineffective as its dimension increases quadratically with the number of people. Real-time performance It can run in real time at 30-60 FPS (6 cameras, 20 people). As the number of participants increases, the matching phase time increases significantly (>1 s / frame). robustness Supports local redundancy and multiple candidate clusters, with strong resistance to occlusion. Relying on a single global match makes it susceptible to false positives. Overall Results Highly efficient, scalable, and suitable for real-time multi-person scenarios. High computational cost, insufficient real-time performance, and difficult to use in multi-user scenarios.

[0143] As shown in Table 1, this embodiment decomposes the originally global matching problem into several local sub-problems by introducing BFS clustering and Top-K filtering on the multi-camera association graph. This significantly reduces the algorithm complexity from O(N³) of traditional Hungarian matching to the order of O(M×N×K). This localization and pruning strategy enables the system to maintain real-time operation in multi-person, multi-view scenarios, while the traditional 4D Association algorithm is difficult to execute in real-time under multi-person scenarios due to the explosion of the cost matrix dimension.

[0144] Example 2

[0145] This embodiment also provides a 3D human skeleton fitting system based on multi-camera correlation maps and combined search, including:

[0146] The acquisition module is used to acquire images from multiple cameras and use a two-dimensional pose estimation algorithm to detect two-dimensional joints of the human body, and then back-project the detected two-dimensional joints into three-dimensional rays.

[0147] The module is used to construct a multiphase mechanism node association graph with key points as nodes and geometric consistency between 3D rays from different cameras as edge weights.

[0148] The clustering module is used to cluster cross-phase mechanism nodes with high geometric consistency on the association graph through breadth-first search (BFS) to form candidate skeleton clusters.

[0149] The first screening module is used to cluster cross-phase mechanism nodes with high geometric consistency on the association graph through breadth-first search (BFS) to form candidate skeleton clusters.

[0150] The second filtering module is used to enumerate cross-phase key node pairs using depth-first search (DFS) and combine it with a pruning strategy to filter out the optimal key node pairs.

[0151] The processing module is used to triangulate the selected optimal joint point pairs to obtain three-dimensional skeleton points and output the three-dimensional human skeleton structure.

[0152] It should be noted that the three-dimensional human skeleton fitting system based on multi-camera correlation graph and combined search provided in this embodiment is similar to that in Embodiment 1, and will not be described in detail here.

[0153] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for fitting a 3D human skeleton based on multi-camera correlation graphs and combined search, characterized in that, include: S1. Acquire images from multiple cameras and use a two-dimensional pose estimation algorithm to detect two-dimensional joints of the human body. Back-project the detected two-dimensional joints into three-dimensional rays to form a cross-camera candidate ray set. S2. Using key points as nodes, the geometric consistency of key point pairs matched between different camera 3D rays, the proportion of matched key point pairs, and the average score of matched key point pairs are used as edge weights to construct a multiphase mechanism node association graph. The calculation of the edge weights in step S2 is expressed as follows: ； in, The edge weights are represented by i and j, respectively, which represent camera i and camera j. S represents the skeleton of camera i i And the frame of camera j S j The average score of the matched keypoint pairs; g ij S represents the skeleton of camera i i And the frame of camera j S j Geometric consistency of matched keypoint pairs; r ij S represents the skeleton of camera i i And the frame of camera j S j The proportion of matched keypoint pairs; α and β both represent weight values; The geometric consistency g of the matched keypoint pairs ij Represented as: ； Where, m ij Indicates the skeleton S of camera i i And the frame of camera j S j The number of key points matched; k represents the index of the matched key point pairs; a i,k This indicates that the k-th matching joint pair is located in the skeleton S of camera i. i Key points; a j,k Indicates and a i,k Matching skeleton S in camera j j The two joints are a pair of joints of the same type from different cameras; The average score of the matched keypoint pairs Represented as: ； ； Where s represents the score of each matched keypoint pair; k The score represents the score of the k-th matched key pair; d represents the distance in physical space between the 3D rays corresponding to the two key pairs; T d This represents the maximum tolerance threshold between two joints; The proportion r of the matched keypoint pairs ij Represented as: ； Among them, |A i | Represents skeleton S i Total number of joints; |A j | Represents skeleton S j The total number of joints; S3. Cluster cross-phase mechanism nodes with high geometric consistency on the association graph using breadth-first search (BFS) to form candidate skeleton clusters; S4. Within each camera, retain the top K key points with the highest geometric consistency to the candidate skeleton cluster to obtain candidate key point pairs; S5. Depth-first search (DFS) is used to enumerate cross-phase key node pairs, and the optimal key pair is selected by combining it with a pruning strategy. S6. Triangulate the selected optimal joint point pairs to obtain three-dimensional skeleton points and output the three-dimensional human skeleton structure.

2. The three-dimensional human skeleton fitting method based on multi-camera correlation graph and combined search according to claim 1, characterized in that, In step S4, the top K joints with the highest geometric consistency with the candidate skeleton cluster are retained within each camera, denoted as: ； Among them, K i This represents the set of key nodes in the top K high-association graphs selected within camera i; The edge weight between node u and node v is derived from the calculated association graph; v represents a neighboring node; u represents the current node; N(u) represents the set of neighboring nodes connected to node u in the association graph.

3. The three-dimensional human skeleton fitting method based on multi-camera correlation graph and combined search according to claim 1, characterized in that, In step S5, the depth-first search (DFS) is used to enumerate cross-phase control node pairs, which are represented as follows: ； in, This indicates the overall consistency score of the current portfolio; This represents the selected cross-camera key combination in the current DFS search path; v represents neighboring key points; u represents the current key point; The optimal key pair is selected by combining pruning strategies, and is represented as follows: ； Among them, T S This represents the maximum similarity threshold between two skeletons; Prune represents the pruning strategy.

4. The three-dimensional human skeleton fitting method based on multi-camera correlation graph and combined search according to claim 1, characterized in that, After obtaining the three-dimensional skeleton points in step S6, the method further includes: performing proportional and angular constraint processing on the three-dimensional skeleton points.

5. The three-dimensional human skeleton fitting method based on multi-camera correlation graph and combined search according to claim 1, characterized in that, After obtaining the three-dimensional skeleton points in step S6, the method further includes: performing nonlinear optimization on the three-dimensional skeleton points based on the human statistical model SMPL.

6. A fitting system based on the three-dimensional human skeleton fitting method based on multi-camera correlation graph and combined search as described in any one of claims 1-5, characterized in that, include: The acquisition module is used to acquire images from multiple cameras and use a two-dimensional pose estimation algorithm to detect two-dimensional joints of the human body. The detected two-dimensional joints are back-projected into three-dimensional rays to form a cross-camera candidate ray set. The module is used to construct a multiphase mechanism node association graph, with key points as nodes and the geometric consistency of key point pairs matched between 3D rays from different cameras, the proportion of matched key point pairs, and the average score of matched key point pairs as edge weights. The clustering module is used to cluster cross-phase mechanism nodes with high geometric consistency on the association graph through breadth-first search (BFS) to form candidate skeleton clusters. The first screening module is used to cluster cross-phase mechanism nodes with high geometric consistency on the association graph through breadth-first search (BFS) to form candidate skeleton clusters. The second filtering module is used to enumerate cross-phase key node pairs using depth-first search (DFS) and combine it with a pruning strategy to filter out the optimal key node pairs. The processing module is used to triangulate the selected optimal joint point pairs to obtain three-dimensional skeleton points and output the three-dimensional human skeleton structure.

Images