Single-view three-dimensional human skeleton key point detection method, device, equipment and medium

Abstract

The present application provides a single-view three-dimensional human skeleton key point detection method, device, equipment and medium, comprising: acquiring a single-view human image sequence, the single-view human image sequence comprising a target image and a related image; acquiring a first skeleton key point of the target image and a second skeleton key point of the related image respectively, extracting spatial feature of the first skeleton key point to obtain spatial semantic feature, and extracting time sequence feature of the first skeleton key point and the second skeleton key point; fusing the supervision feature with the spatial semantic feature and the time sequence feature to obtain three-dimensional human skeleton key point feature information. In the present application, the spatial semantic feature and the time sequence feature are extracted to obtain the plane information and the depth information of the skeleton key point; the supervision feature is fused with the spatial semantic feature and the time sequence feature, which can effectively reduce the depth ambiguity and the ill-posedness of the two-dimensional human skeleton key point mapping three-dimensional human skeleton key point of the single-view image.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method, apparatus, device, and medium for detecting key points of a three-dimensional human skeleton from a single perspective. Background Technology

[0002] 3D human skeleton key point detection is one of the fundamental problems and research hotspots in the field of computer vision. Its task is to obtain the position and connection information of human skeleton key points in a target video frame in three-dimensional space. It is a basic technology for many vision tasks such as scene understanding, behavior recognition, and pedestrian re-identification, and is widely used in video surveillance, behavior recognition, online teaching, motion capture, virtual reality, medical assistance and other fields.

[0003] In single-view image 3D human skeleton keypoint detection, due to the single viewpoint, depth information is lost when the image is projected from 3D space to a 2D plane, which may cause multiple different 3D human skeleton keypoints to be projected onto the same 2D keypoint. Therefore, in single-view images, the mapping of human skeleton information from 2D space to 3D space has depth ambiguity and ill-posedness, and even small positioning errors of 2D keypoints may lead to large pose distortions in 3D space.

[0004] Existing single-view 3D human skeleton key point detection methods mainly include direct estimation methods and 2D-to-3D enhancement methods.

[0005] Direct estimation methods typically involve designing an end-to-end network to infer 3D human skeleton keypoints directly from the input 2D image, without requiring intermediate estimation of the 2D human information representation. While this method can extract rich information from the image, it lacks the intermediate supervision process for converting 2D human skeleton information into 3D human skeleton information, requiring a large amount of 3D labeled data to train a high-performance model.

[0006] The 2D-to-3D upscaling method first uses a 2D human skeleton keypoint detection model to estimate 2D skeleton keypoint information. Then, it uses 2D-to-3D upscaling methods such as fusion of 2D skeleton information and 3D image features, and 3D spatial reprojection to obtain 3D human skeleton keypoints. This method is generally superior to direct estimation methods, reducing the learning pressure on the model on 2D skeleton keypoints. However, since it directly estimates 3D information based on existing 2D information, it is greatly affected by the performance of the 2D human skeleton keypoint detector and lacks supervision from original image features. Summary of the Invention

[0007] In view of the problems existing in the prior art, the present invention proposes a method, device, equipment and medium for detecting key points of a three-dimensional human skeleton from a single perspective, which mainly solves the problems of lack of intermediate supervision and lack of supervision of original image features in the detection of key points of a three-dimensional human skeleton from a single perspective image in the prior art.

[0008] To achieve the above and other objectives, the technical solution adopted by the present invention is as follows.

[0009] Optionally, a single-view 3D human skeleton key point detection method is provided, including:

[0010] A single-view human image sequence is obtained, which includes multiple frames of images of the same target object in different poses within a preset time period, wherein one frame is used as the target image and the remaining images are used as related images of the target image.

[0011] The first skeletal key points of the target image and the second skeletal key points of the related image are obtained respectively. Spatial features are extracted from the first skeletal key points to obtain spatial semantic features, and temporal features are extracted from the first skeletal key points and the second skeletal key points to obtain temporal features. The spatial semantic features include global semantic features and local semantic features.

[0012] The supervised features are fused with the spatial semantic features and temporal features to obtain the key point feature information of the three-dimensional human skeleton. The supervised features are obtained by extracting information from the target image through a feature extraction network. The supervised features include the deep semantic information, texture information and edge information of the target image.

[0013] Optionally, spatial semantic features are extracted from the first skeletal key points, including:

[0014] Construct a global spatial graph and a global adjacency matrix based on the first skeletal key points;

[0015] The weight matrix of the global spatial graph is mined by a multi-head attention mechanism, and multidimensional features of multiple receptive fields in the global spatial graph are obtained by using a dilated convolutional network.

[0016] The global adjacency matrix is ​​updated using a graph convolutional network to obtain the first matrix;

[0017] The multi-head attention global space graph is constructed by combining the weight matrix, multi-dimensional features, and the first matrix;

[0018] The global semantic features are obtained by performing feature representation on the multi-head attention global spatial graph.

[0019] Optionally, spatial semantic features are extracted from the first skeletal key points, including:

[0020] Multiple local adjacency matrices are constructed based on the local connectivity relationships of the first skeletal key points. The multiple local adjacency matrices are then updated using a graph convolutional network to obtain multiple second matrices.

[0021] Multiple local spatial graphs are constructed based on the multiple second matrices, and the local semantic features are obtained by performing feature representation on the multiple local spatial graphs.

[0022] Optionally, temporal features are extracted from the first and second skeletal keypoints to obtain temporal features, including:

[0023] Construct the temporal adjacency matrix of the first and second skeletal key points, and update the temporal adjacency matrix through a graph convolutional network to obtain the third matrix;

[0024] A time series graph is constructed using the third matrix, and the time series graph is then used to represent its features to obtain the time series features.

[0025] Optionally, the supervised features are fused with the spatial semantic features and temporal features to obtain three-dimensional human skeleton key point feature information, including:

[0026] The global semantic features and local semantic features are used as attention factors of the supervision features. The global semantic features and local semantic features are linked to form the first spatial structure features of the target image. The first spatial structure features are adjusted using a spatial feature adjuster to obtain the second spatial structure features.

[0027] The temporal features and the supervisory features are fused to obtain the first temporal structure feature after fusion. The first temporal structure feature is then adjusted using a temporal feature adjuster to obtain the second temporal feature.

[0028] By using the second temporal feature as the attention factor for the second spatial structural feature, the key point feature information of the three-dimensional human skeleton is obtained.

[0029] Optionally, before acquiring a single-view human image sequence, the following steps are included:

[0030] Construct an initial network model and obtain single-view human image training samples;

[0031] The training samples of the single-view human images are input into the initial network model, and the parameters of the initial network model are optimized according to the preset objective function to obtain the trained network model. The sequence of single-view human images is then input into the trained network model.

[0032] Optionally, the first skeletal key points of the target image and the second skeletal key points of the related image are obtained respectively, including:

[0033] A two-dimensional skeletal keypoint detector is used to detect the target object in the target image and the related image respectively, so as to obtain the first skeletal keypoint of the target image and the second skeletal keypoint of the related image.

[0034] Optionally, a single-view 3D human skeleton key point detection device is provided, comprising:

[0035] The image acquisition module is used to acquire a single-view human image sequence, which includes multiple frames of images of the same target object in different poses within a preset time period, wherein one frame is used as the target image and the remaining images are used as related images of the target image.

[0036] The feature acquisition module is used to acquire the first skeletal key points of the target image and the second skeletal key points of the related image respectively, extract spatial features from the first skeletal key points to obtain spatial semantic features, and extract temporal features from the first skeletal key points and the second skeletal key points to obtain temporal features. The spatial semantic features include global semantic features and local semantic features.

[0037] The feature fusion module is used to fuse the supervised features with the spatial semantic features and temporal features to obtain three-dimensional human skeleton key point feature information. The supervised features are obtained by extracting information from the target image through a feature extraction network. The supervised features include deep semantic information, texture information and edge information of the target image.

[0038] Optionally, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the single-view three-dimensional human skeleton key point detection method as described above.

[0039] Optionally, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a single-view 3D human skeleton key point detection method:

[0040] A single-view human image sequence is obtained, which includes multiple frames of images of the same target object in different poses within a preset time period, wherein one frame is used as the target image and the remaining images are used as related images of the target image.

[0041] The first skeletal key points of the target image and the second skeletal key points of the related image are obtained respectively. Spatial features are extracted from the first skeletal key points to obtain spatial semantic features, and temporal features are extracted from the first skeletal key points and the second skeletal key points to obtain temporal features. The spatial semantic features include global semantic features and local semantic features.

[0042] The supervised features are fused with the spatial semantic features and temporal features to obtain the key point feature information of the three-dimensional human skeleton. The supervised features are obtained by extracting information from the target image through a feature extraction network. The supervised features include the deep semantic information, texture information and edge information of the target image.

[0043] In the aforementioned single-view 3D human skeleton keypoint detection method, the single-view human image sequence is divided into target image and related images, and skeletal keypoints are obtained from the target image and related images. Global and local semantic features of the target image are extracted, and temporal features of the single-view human image sequence are extracted. Supervised features are fused with spatial semantic features and temporal features to obtain 3D human skeleton keypoint feature information. The planar information of 3D human skeleton keypoints is mapped spatially using the global and local semantic features of the target image; the depth information of 3D human skeleton keypoints is mapped using the temporal features of multiple consecutive images from the same viewpoint. Furthermore, fusing supervised features with spatial semantic features and temporal features during feature fusion effectively reduces the depth ambiguity and ill-posedness of mapping 2D human skeleton keypoints from single-view images to 3D human skeleton keypoints, thus improving detection accuracy. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a single-view three-dimensional human skeleton key point detection method in one embodiment of the present invention.

[0046] Figure 2 for Figure 1 A flowchart illustrating a specific implementation method of step S1;

[0047] Figure 3 for Figure 1 A flowchart illustrating a specific implementation method of step S2;

[0048] Figure 4 for Figure 1 A flowchart illustrating a specific implementation method of step S2;

[0049] Figure 5 for Figure 1 A flowchart illustrating a specific implementation method of step S2;

[0050] Figure 6 This is a schematic diagram of a single-view three-dimensional human skeleton key point detection device according to an embodiment of the present invention. Detailed Implementation

[0051] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0052] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0053] The single-view 3D human skeleton key point detection method provided by this invention, for human image sequences acquired from a single viewpoint, utilizes graph convolutional networks and attention mechanisms to effectively acquire and fuse spatial semantic features and temporal features of 3D human skeleton key points in single-view images, mining feature representations with spatiotemporal consistency and strong representational properties. In addition, supervised features are fused with spatial semantic features and temporal features to reduce the depth ambiguity and ill-posedness of mapping from 2D to 3D human skeleton key points in single-view images, thereby improving the accuracy of skeleton key point detection.

[0054] See Figure 1 , Figure 1 A flowchart illustrating a single-view 3D human skeleton key point detection method provided in an embodiment of the present invention includes the following steps:

[0055] S1. Obtain a single-view human image sequence, and detect and acquire the first skeletal key points of the target image and the second skeletal key points of the related images.

[0056] When detecting key points of the human skeleton from a single viewpoint, a sequence of human images from a single viewpoint is first acquired. This sequence includes multiple frames of the same target object in different poses within a preset time period. One frame is designated as the target image, and the remaining frames are designated as related images. Then, two-dimensional skeletal key points are acquired from the target image and the related images. In this embodiment, the two-dimensional skeletal key points of the target image are the first skeletal key points, and the two-dimensional skeletal key points of the related images are the second skeletal key points.

[0057] The single-view human image sequence in this embodiment can be obtained by editing consecutive video frames in a video, or it can be obtained by a camera capturing images multiple times from a single view within a preset time.

[0058] In this embodiment, the single-view human body image sequence is denoted as I1, I2, ..., I t-2 I t-1 I t I t+1 I t+2 , ..., I N-1 I N , where I t Let N be the target image and N be the total number of image samples. Multiple frames of the same target object in different poses within a preset time period are selected as related images, denoted as I1, I2, ..., I... t-2 I t-1 I t+1 I t+2 , ..., I n-1 I n Where n is the number of relevant image samples selected, and n is less than or equal to the total number of image samples N. The specific number of relevant image samples can be determined according to the actual situation. For example, when n = 2, the relevant image is I. t-1 I t+1 .

[0059] Among them, such as Figure 2 As shown, in step S1, obtaining the two-dimensional skeletal key points of the target image and related images in a single-view human body image includes the following steps:

[0060] S11. Obtain a single-view human body image sequence and divide the single-view human body image sequence into target images and related images;

[0061] S12. Use a two-dimensional skeleton keypoint detector to detect the target object in the target image and related images respectively, and obtain the first skeleton keypoint and the second skeleton keypoint.

[0062] In this embodiment, after acquiring the target image and related images, the target image and related images are input into a two-dimensional skeletal keypoint detector for detection to obtain the first skeletal keypoint of the target image and the second skeletal keypoint of the related image. The two-dimensional skeletal keypoint detector is not limited to CPN or HRNet.

[0063] S2. Spatial semantic features are extracted from the key points of the first skeleton, and temporal features are extracted from the key points of the first and second skeletons.

[0064] We establish corresponding spatial graph extractors and temporal graph extractors using graph convolutional networks and attention mechanisms. The spatial graph extractor extracts global and local semantic features of the target image, while the temporal graph extractor extracts temporal features of a single-view human image sequence.

[0065] Among them, such as Figure 3 As shown, step S2 involves extracting spatial features from the key points of the first skeleton to obtain global semantic features, including the following steps:

[0066] S211. Construct a global spatial graph and a global adjacency matrix based on the key points of the first skeleton;

[0067] S212. The weight matrix of the global spatial graph is mined through a multi-head attention mechanism, and multi-dimensional features of multiple receptive fields in the global spatial graph are obtained by using a dilated convolutional network.

[0068] S213. Update the global adjacency matrix using a graph convolutional network to obtain the first matrix;

[0069] S214. Construct a global space graph for multi-head attention by combining the weight matrix, multi-dimensional features, and the first matrix;

[0070] S215. Perform feature representation on the multi-head attention global spatial graph to obtain the global semantic features.

[0071] Among them, using target image I t The first skeletal keypoints construct a global spatial map, target image I t The global spatial graph structure is represented as g = (V, ε), where V = {v ti |i=1,…,M} is the set of vertices in the graph, and M is the number of key points in the human skeleton; ε=}e ij |i=1,2,......,M;j=1,2,......,M} is the set of graph edges, representing the skeletal structure formed by connecting key points in the human body. ε is 1 when the connected skeletal structure conforms to human skeletal logic, and ε is 0 otherwise. The eigenvectors of the global spatial graph are X={x1,x2,……,x M |x1∈R1×C, where C is the number of feature channels.

[0072] According to SemGCN, after the input feature vector passes through a graph convolutional network, the features transform as follows: in for The normalized diagonal matrix; A∈R M×M W is an adjacency matrix representing the connections between adjacent skeletal keypoints. A first-order matrix represents the first-order connections between keypoints, while second- and third-order matrices represent the second- and third-order connections between keypoints; W = {ω} ij} is a learnable weight matrix representing the interaction between key points of the skeleton; σ is a non-linear activation function.

[0073] To focus on the global interactions between key points in the human skeleton, different trainable weights are assigned to the connections between any two joints, and an adjacency matrix is ​​constructed to represent the connections between all key points in the human skeleton. A multi-head attention mechanism is used to randomly mine the weight matrix of the global spatial graph multiple times to uncover the interactions between key points in the human skeleton and obtain global spatial features. A dilated convolutional network is used to extract multidimensional features from different receptive fields. A graph convolutional network is used to update multiple elements of the global adjacency matrix, and the updated global adjacency matrix is ​​defined as the first matrix. Combining the weight matrix, multi-dimensional features, and the first matrix, the multi-head attention global spatial graph is constructed. Global semantic features are obtained by representing the multi-head attention global spatial graph.

[0074] In one embodiment, the global spatial semantic features based on the multi-head attention global spatial graph are represented as follows:

[0075] Where K is the number of attention heads, multi-head attention is used to achieve attention to the global spatial semantic features of skeletal key points in all directions, and to reduce the influence of certain local interference factors; For adjustment matrix; This is an adaptively learnable global adjacency matrix, representing the connectivity and strength between all skeletal keypoints, whose elements b ij Instead of using simple 1s and 0s to represent whether there is a connection between two skeletal keypoints, we use the interaction of features between skeletal keypoints to describe the connection, specifically expressed as: Where θ and Γ is a convolutional layer with a kernel of 1, used to adjust the feature dimension; || represents feature concatenation; Γ is a mapping function used to map high-dimensional features to low-dimensional or real numbers; ρ is an activation function, which can be LeakyReLU.

[0076] See Figure 4In step S2, spatial feature extraction is performed on the key points of the first skeleton to obtain local semantic features, including the following steps:

[0077] S221. Construct multiple local adjacency matrices based on the local connectivity relationships of the first skeleton key points, and update the multiple local adjacency matrices through a graph convolutional network to obtain multiple second matrices;

[0078] S222. Construct multiple local spatial graphs based on multiple second matrices, and perform feature representation on the multiple local spatial graphs to obtain the local semantic features.

[0079] Besides focusing on the overall connectivity between key points in the human skeleton, the specific local relationships between these key points also affect the final detection accuracy. Local features such as symmetry, first-order connectivity, and second-order connectivity can be selected. Symmetry helps distinguish between limb and trunk joints, defining the main area of ​​the target key point. Based on known points, the approximate location of corresponding symmetrical points can be inferred. First-order connectivity points are points directly connected to the target key point; changes in their positions generally have the most direct impact on the target key point's location. Second-order connectivity points may form a body part together with first-order connectivity points and the target key point, serving as auxiliary information for determining the target key point's location. For each type of local feature, such as symmetry, first-order connectivity, and second-order connectivity, multiple local adjacency matrices are first constructed using the local relationships of the first skeletal keypoints. Then, the elements in these local adjacency matrices are updated through learning by a graph convolutional network, resulting in multiple updated local adjacency matrices. These updated local adjacency matrices can be defined as second matrices. Multiple local spatial graphs are constructed based on these second matrices, and feature representations are performed on these local spatial graphs to obtain local semantic features, thereby describing the local influence relationships between human skeletal joints.

[0080] In one embodiment, the local spatial semantic features based on multiple local spatial graphs are represented as follows: Where S represents the category of the local feature; The local adjacency matrix generated for the S-th type of local features; M S is a learnable mask matrix that can be used to shield the influence of non-S-class local features and reduce model parameters; ⊙ represents the dot product operation. The elements d of the local adjacency matrix... ij This can be expressed as: when the joints i and j are symmetrical, element d ij =d ji =1, and all other elements are 0. The elements of the first-order and second-order connection adjacency matrices are represented as follows: when there is a first-order or second-order connection between joints i and j, d ij Using b in global semantic feature representation ij The representation describes the local connection strength between joints i and j.

[0081] Through the above operations, the global and local semantic features of the skeletal key points of the target image in a single, single-view human body image can be obtained.

[0082] Among them, such as Figure 5 As shown, step S2 involves extracting temporal features from the first and second skeletal keypoints, including the following steps:

[0083] S231. Construct the temporal adjacency matrix of the first and second skeletal key points, and update the temporal adjacency matrix through a graph convolutional network to obtain the third matrix;

[0084] S232. Construct a time series graph using the third matrix, perform feature representation on the time series graph, and obtain the time series features.

[0085] A temporal adjacency matrix is ​​constructed for each keypoint using the first and second skeletal keypoints. Then, the elements in the temporal adjacency matrix are updated through learning by a graph convolutional network to obtain the updated temporal adjacency matrix, which can be defined as the third matrix. A temporal graph is constructed using the third matrix, and the temporal graph is used to represent its features to obtain temporal features, thereby depicting the positional changes of each keypoint in the temporal receptive field and its temporal contextual relevance.

[0086] In one embodiment, the timing features based on the timing graph are represented as follows:

[0087] in Let be the adjacency matrix formed by keypoint i in the time domain T, whose elements are represented by the cosine similarity distance between related images and the target image. The elements of the temporal adjacency matrix are represented by the similarity between each skeletal keypoint corresponding to the target frame and adjacent frames. In this embodiment, normalized cosine similarity is used, expressed as:

[0088] λ t =σ(ψ) t [L(P1,P t ), ..., L(P T-1 ,P t )] T )

[0089] Where P t Let t be the coordinates of the two-dimensional joint point, t be the index of the target frame image, and L be the cosine distance calculation function.

[0090] S3. The supervised features are fused with spatial semantic features and temporal features to obtain the key point feature information of the three-dimensional human skeleton.

[0091] To generate human skeletal keypoint features with spatiotemporal consistency and strong representation, a feature fusion processor is constructed to fuse global semantic features, local semantic features, and temporal features. During feature fusion, supervised features from the target image are used as supplementary and intermediate supervision to the human background information. The supervised features are obtained by extracting information from the target image through a feature extraction network. These supervised features include deep and shallow information from the target image. Deep information refers to deep semantic information, which semantically transforms image features into abstract features that humans can reason about and understand. These are high-level features possessed by humans; for example, one can imagine the shape of an "arm" by looking at a few skeletal keypoints, and imagine an "eye" by looking at eye socket keypoints. Because neural networks are actually constructed by mimicking the learning and reasoning processes of the human brain, the features they learn are as close as possible to features that humans can understand. Shallow features are generally surface information, such as directly visible surface textures, shapes, and colors. Deep features are abstract information at the level of human learning and understanding. For example, shallow information: blue corresponds to deep information: melancholy; shallow information: green light corresponds to deep information: passable. In this invention, the deep information mainly consists of deep semantic information, while the shallow information mainly consists of texture information and edge information.

[0092] Supervised features refer to the raw target image information obtained using multi-scale convolutional layers, including shallow texture information, edge information, and deep semantic information of the human target and background in the target image. Since spatial semantic features and temporal features are both obtained based on graph convolutional networks, and the graph features involved in graph convolutional networks are mainly obtained based on the position information of key skeletal joints of the human body, lacking raw image information, the supervised features of the target image here can serve as a supplement to spatial and temporal features.

[0093] Global and local semantic features are used as attention factors for supervised features. These global and local semantic features are then combined to form the first spatial structure feature of the target image. A spatial feature regulator is used to adjust this spatial structure feature, resulting in the second spatial structure feature. Temporal features are fused with supervised features to obtain the first temporal structure feature. This temporal feature regulator is then used to adjust this temporal structure feature, resulting in the second temporal structure feature. The second temporal structure feature is used as the attention factor for the second spatial structure feature, yielding the 3D human skeleton keypoint feature information. The spatial and temporal feature regulators are convolutional networks or fully connected layer networks, capable of upscaling or downscaling the spatial and temporal features to a consistent dimension.

[0094] In one embodiment, let F be the supervised feature of the target image in the original single-view human body image. S , to use global spatial semantic features F G and local spatial semantic features FL As a supervisory feature F S The attention factor represents the degree of attention paid to the global and local spatial information of skeletal key points. Then, the global and local semantic features are linked to form the spatial structural features of the image; the temporal features F... T Then, with the supervision feature F S The feature dimensions are adjusted by a temporal feature adjuster, and then the adjusted temporal structural features are used as attention factors for the adjusted spatial structural features. This focuses on how related images supplement the depth information of the target image in the temporal sequence, resulting in the fusion feature F of the 3D human skeleton keypoints. The fusion feature F is expressed as: F = α(F T ||F S )⊙β(F S ||F G ⊙(F S ||F L ⊙F S ), where α and β are adjustment functions; || is the feature connection; ⊙ represents the dot product; F S =G(I t ), representing the supervised features of the target image extracted by the feature extraction network. The feature extraction network G can be a common convolutional neural network such as ResNet.

[0095] S4. Refine the feature information of the key points of the 3D human skeleton to obtain the coordinate information of the key points of the 3D skeleton.

[0096] The feature refiner is used to optimize and adjust the feature information of the fused 3D human skeleton key points to obtain the coordinate information of the 3D human skeleton key points.

[0097] In this invention, before acquiring the single-view human image sequence, the following steps are also included:

[0098] Construct an initial network model and obtain single-view human image training samples;

[0099] Training samples of single-view human images are input into the initial network model. The parameters of the initial network model are optimized according to the preset objective function to obtain the trained network model. The sequence of single-view human images is then input into the trained network model.

[0100] The preset objective function is: in, φ t,iThese are the estimated three-dimensional coordinates of the i-th skeletal keypoint in the t-th frame of the target image, and the reference three-dimensional coordinates, respectively. The estimated three-dimensional coordinates are the three-dimensional skeletal keypoint coordinate information in this invention, and the reference coordinates are the keypoint coordinate information in the training data. When training the model based on a preset objective function, the smaller the value of the objective function, the higher the accuracy of keypoint detection.

[0101] As can be seen, in the above scheme, graph convolutional networks are used to obtain spatial semantic features and temporal features. These features mainly include the connection and positional information of human skeletal key points, and their fusion gives the features spatiotemporal relevance. Supervised features, containing original image information, supplement the spatial semantic features and temporal features, further improving the representativeness of the features acquired by the network. The global semantic features and local semantic structure features of two-dimensional skeletal key points in a single-view, single-image human body can map the planar information of three-dimensional human skeletal key points. The temporal features of multiple consecutive images from the same viewpoint jointly map the depth information of three-dimensional human skeletal key points. Using supervised features as intermediate supervision in the fusion process of spatial semantic features and temporal features can effectively reduce the depth ambiguity and ill-posedness when mapping two-dimensional human skeletal key points to three-dimensional human skeletal key points, thus improving the detection accuracy of three-dimensional human skeletal key points.

[0102] In one embodiment, a single-view 3D human skeleton key point detection device is provided, which corresponds one-to-one with the detection method in the above embodiment. Specifically, the detection device includes: an image acquisition module, used to acquire a single-view human image sequence, the single-view human image sequence including multiple frames of the same target object in different poses within a preset time period, wherein one frame is used as the target image and the remaining images are used as related images of the target image; a feature acquisition module, used to acquire the first skeletal key points of the target image and the second skeletal key points of the related images respectively, extract spatial features from the first skeletal key points to obtain spatial semantic features, and extract temporal features from the first skeletal key points and the second skeletal key points to obtain temporal features, the spatial semantic features including global semantic features and local semantic features; and a feature fusion module, used to fuse the supervised features with the spatial semantic features and temporal features to obtain 3D human skeleton key point feature information, the supervised features including deep semantic information, shallow texture information and edge information of the target image.

[0103] Specifically, a two-dimensional skeletal keypoint detector can be used to obtain the first skeletal keypoints of the target image and the second skeletal keypoints of related images; the feature acquisition modules can be a temporal map extractor and a spatial map extractor, respectively; the feature fusion module can be a feature fusion module. See also Figure 6 This describes a single-view 3D human skeleton key point detection device and the 3D human skeleton key point detection steps based on the device.

[0104] The two-dimensional skeleton keypoint detector acquires the target image and related images, and then inputs the target image and related images into the two-dimensional skeleton keypoint detector for detection to obtain the first skeleton keypoint of the target image and the second skeleton keypoint of the related images. The two-dimensional skeleton keypoint detector is not limited to CPN and HRNet.

[0105] A spatial graph extractor is constructed using a graph convolutional network and an attention mechanism to obtain global and local semantic features of the first skeletal keypoints in a target image. A global spatial graph and a global adjacency matrix are constructed based on the first skeletal keypoints. A weight matrix of the global spatial graph is mined using a multi-head attention mechanism, and multi-dimensional features of multiple receptive fields in the global spatial graph are obtained using a dilated convolutional network. The global adjacency matrix is ​​updated using a graph convolutional network to obtain a first matrix. A multi-head attention global spatial graph is constructed by combining the weight matrix, multi-dimensional features, and the first matrix. Feature representation is performed on the multi-head attention global spatial graph to obtain global semantic features. Multiple local adjacency matrices are constructed based on the local connectivity relationships of the first skeletal keypoints. Multiple local adjacency matrices are updated using a graph convolutional network to obtain multiple second matrices. Multiple local spatial graphs are constructed based on the multiple second matrices, and feature representation is performed on the multiple local spatial graphs to obtain the local semantic features.

[0106] A temporal graph extractor is constructed using a graph convolutional network to obtain temporal features of a single-view human image sequence. A temporal adjacency matrix is ​​constructed for the first and second skeletal keypoints. This matrix is ​​then updated using the graph convolutional network to obtain a third matrix. A temporal graph is constructed using this third matrix, and feature representation is performed on the temporal graph to obtain the temporal features.

[0107] A feature fusion processor integrates supervised features with spatial semantic features and temporal features to obtain key point feature information of the 3D human skeleton, acquiring feature representations with spatiotemporal consistency and strong representational power. Global and local semantic features are used as attention factors for the supervised features, connecting them to form the first spatial structure feature of the target image. A spatial feature adjuster is used to adjust the first spatial structure feature to obtain the second spatial structure feature. Temporal features are then integrated with the supervised features to obtain the fused first temporal structure feature. A temporal feature adjuster is used to adjust the first temporal structure feature to obtain the adjusted second temporal structure feature. The second temporal structure feature is used as the attention factor for the second spatial structure feature to obtain the key point feature information of the 3D human skeleton. In this embodiment, the spatial feature adjuster and temporal feature adjuster are convolutional networks or fully connected layer networks, responsible for dimensionality reduction or expansion of the spatial semantic features and temporal features, primarily adjusting the two types of features to a consistent dimension.

[0108] In addition, the device also includes a feature refiner, which uses the network layer to optimize and adjust the fusion features of the key points of the three-dimensional human skeleton to obtain the coordinate information of the key points of the three-dimensional human skeleton.

[0109] This invention provides a single-view 3D human skeleton key point detection device. The device detects and obtains first and second skeleton key points through a 2D skeleton key point detector, extracts global and local semantic features of the target image through a spatial graph extractor, obtains temporal features of the single-view human image sequence through a temporal graph extractor, and fuses supervised features with spatial semantic features and temporal features through a feature fusion unit. This achieves effective extraction and fusion of spatial semantic features and temporal features of 3D human skeleton key points in single-view images, mines feature representations with spatiotemporal consistency and strong representational properties, reduces the depth ambiguity and ill-posedness of the mapping from 2D to 3D human skeleton key points in single-view images, and improves the accuracy of 3D human skeleton key point detection.

[0110] Specific limitations regarding the single-view 3D human skeleton key point detection device can be found in the limitations of the single-view 3D human skeleton key point detection method described above, and will not be repeated here. Each module or part of the aforementioned single-view 3D human skeleton key point detection device can be implemented entirely through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0111] In one embodiment, a computer device is provided, which may be a single-view human image processing terminal. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used to communicate with an external single-view human image acquisition terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a single-view three-dimensional human skeleton key point detection method on the server side. The processor executes the following steps when executing the computer program:

[0112] A single-view human image sequence is obtained. The single-view human image sequence includes multiple frames of images of the same target object in different poses within a preset time period. One frame is used as the target image, and the remaining images are used as related images of the target image.

[0113] The first skeletal key points of the target image and the second skeletal key points of the related image are obtained respectively. Spatial features are extracted from the first skeletal key points to obtain spatial semantic features, and temporal features are extracted from the first skeletal key points and the second skeletal key points to obtain temporal features. Spatial semantic features include global semantic features and local semantic features.

[0114] By fusing supervised features with spatial semantic features and temporal features, key point feature information of the three-dimensional human skeleton is obtained. The supervised features include deep semantic information, texture information and edge information of the target image.

[0115] The feature refiner is used to refine the feature information of the key points of the three-dimensional human skeleton, and the coordinate information of the key points of the three-dimensional skeleton is obtained.

[0116] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0117] A single-view human image sequence is obtained. The single-view human image sequence includes multiple frames of images of the same target object in different poses within a preset time period. One frame is used as the target image, and the remaining images are used as related images of the target image.

[0118] The first skeletal key points of the target image and the second skeletal key points of the related image are obtained respectively. Spatial features are extracted from the first skeletal key points to obtain spatial semantic features, and temporal features are extracted from the first skeletal key points and the second skeletal key points to obtain temporal features. Spatial semantic features include global semantic features and local semantic features.

[0119] By fusing supervised features with spatial semantic features and temporal features, key point feature information of the three-dimensional human skeleton is obtained. The supervised features include deep semantic information, texture information and edge information of the target image.

[0120] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0121] In summary, the single-view image 3D human skeleton key point detection method of the present invention can acquire features with strong representation and spatiotemporal consistency. It utilizes the global and local semantic features of the target image to map the planar information of the 3D human skeleton key points, and utilizes the temporal features of multiple consecutive images from the same viewpoint to map the depth information of the 3D human skeleton key points. Furthermore, it uses supervised features as intermediate supervision, fusing these supervised features with the aforementioned spatial semantic features and temporal features. This effectively reduces the depth ambiguity and ill-posedness of mapping 2D human skeleton key points from a single-view image to 3D human skeleton key points, thus improving detection accuracy. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0122] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for detecting key points of a three-dimensional human skeleton from a single perspective, characterized in that, include: A single-view human image sequence is obtained, which includes multiple frames of images of the same target object in different poses within a preset time period, wherein one frame is used as the target image and the remaining images are used as related images of the target image. The first skeletal key points of the target image and the second skeletal key points of the related image are obtained respectively. Spatial features are extracted from the first skeletal key points to obtain spatial semantic features, which include global semantic features and local semantic features. Specifically, a global spatial graph and a global adjacency matrix are constructed based on the first skeletal key points; a weight matrix of the global spatial graph is mined using a multi-head attention mechanism, and multi-dimensional features of multiple receptive fields in the global spatial graph are obtained using a dilated convolutional network; the global adjacency matrix is ​​updated using a graph convolutional network to obtain a first matrix; a multi-head attention global spatial graph is constructed by combining the weight matrix, multi-dimensional features, and the first matrix; and the global semantic features are obtained by performing feature representation on the multi-head attention global spatial graph. Multiple local adjacency matrices are constructed based on the local connectivity relationships of the first skeletal keypoints. The multiple local adjacency matrices are updated through a graph convolutional network to obtain multiple second matrices. Multiple local spatial graphs are constructed based on the multiple second matrices, and the local semantic features are obtained by performing feature representation on the multiple local spatial graphs. Construct temporal adjacency matrices for the first and second skeletal keypoints, update the temporal adjacency matrices using a graph convolutional network to obtain a third matrix; construct a temporal graph using the third matrix, and perform feature representation on the temporal graph to obtain temporal features; The supervised features are fused with the spatial semantic features and temporal features to obtain the key point feature information of the three-dimensional human skeleton. The supervised features are obtained by extracting information from the target image through a feature extraction network. The supervised features include the deep semantic information, texture information and edge information of the target image.

2. The single-view three-dimensional human skeleton key point detection method according to claim 1, characterized in that, The supervised features are fused with the spatial semantic features and temporal features to obtain the key point feature information of the three-dimensional human skeleton, including: The global semantic features and local semantic features are used as attention factors of the supervision features. The global semantic features and local semantic features are linked to form the first spatial structure features of the target image. The first spatial structure features are adjusted using a spatial feature adjuster to obtain the second spatial structure features. The temporal features and the supervisory features are fused to obtain the first temporal structure feature after fusion. The first temporal structure feature is then adjusted using a temporal feature adjuster to obtain the second temporal feature. By using the second temporal feature as the attention factor for the second spatial structural feature, the key point feature information of the three-dimensional human skeleton is obtained.

3. The single-view three-dimensional human skeleton key point detection method according to claim 1, characterized in that, Before acquiring a single-view human image sequence, the following steps are included: Construct an initial network model and obtain single-view human image training samples; The training samples of the single-view human images are input into the initial network model, and the parameters of the initial network model are optimized according to the preset objective function to obtain the trained network model. The sequence of single-view human images is then input into the trained network model.

4. The single-view three-dimensional human skeleton key point detection method according to claim 1, characterized in that, Obtaining the first skeletal key points of the target image and the second skeletal key points of the related image, respectively, includes: A two-dimensional skeletal keypoint detector is used to detect the target object in the target image and the related image respectively, so as to obtain the first skeletal keypoint of the target image and the second skeletal keypoint of the related image.

5. A single-view three-dimensional human skeleton key point detection device, characterized in that, include: The image acquisition module is used to acquire a single-view human image sequence, which includes multiple frames of images of the same target object in different poses within a preset time period, wherein one frame is used as the target image and the remaining images are used as related images of the target image. The feature acquisition module is used to acquire the first skeletal key points of the target image and the second skeletal key points of the related image, respectively. Spatial feature extraction is performed on the first skeletal key points to obtain spatial semantic features, which include global semantic features and local semantic features. Specifically, a global spatial graph and a global adjacency matrix are constructed based on the first skeletal key points. A weight matrix of the global spatial graph is mined using a multi-head attention mechanism, and multi-dimensional features of multiple receptive fields in the global spatial graph are obtained using a dilated convolutional network. The global adjacency matrix is ​​updated using a graph convolutional network to obtain a first matrix. A multi-head attention mechanism is constructed by combining the weight matrix, multi-dimensional features, and the first matrix. A global spatial graph is constructed; feature representation is performed on the multi-head attention global spatial graph to obtain the global semantic features; multiple local adjacency matrices are constructed based on the local connectivity relationships of the first skeletal keypoints, and the multiple local adjacency matrices are updated through a graph convolutional network to obtain multiple second matrices; multiple local spatial graphs are constructed based on the multiple second matrices, and feature representation is performed on the multiple local spatial graphs to obtain the local semantic features; temporal adjacency matrices of the first skeletal keypoints and the second skeletal keypoints are constructed, and the temporal adjacency matrices are updated through a graph convolutional network to obtain a third matrix; a temporal graph is constructed using the third matrix, and feature representation is performed on the temporal graph to obtain temporal features; The feature fusion module is used to fuse the supervised features with the spatial semantic features and temporal features to obtain three-dimensional human skeleton key point feature information. The supervised features are obtained by extracting information from the target image through a feature extraction network. The supervised features include deep semantic information, texture information and edge information of the target image.

6. A computer apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the steps of the single-view three-dimensional human skeleton key point detection method as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the single-view three-dimensional human skeleton key point detection method as described in any one of claims 1-4.

Images