A skeleton detection and tracking method, device and electronic equipment

Abstract

The present invention discloses a skeleton detection and tracking method and device, including the steps of: S10, obtaining an aligned two-dimensional image and a depth image, inputting the two-dimensional image into a preset 2D skeleton network to obtain the coordinates of key points of the human body and a first feature map respectively; S20, inputting the depth image into a preset depth tracking feature extraction network to obtain a depth feature map; S30, according to the coordinates of key points of the human body, obtaining a 2D skeleton feature vector sequence and a depth skeleton feature vector sequence from the first feature map and the depth feature map respectively; S40, using the 2D skeleton feature vector sequence, the depth skeleton feature vector sequence and the preset corresponding skeleton sequence samples for matching calculation to obtain a target 2D skeleton sequence. The present invention comprehensively utilizes the two-dimensional image and the depth information, matches the outputs of the two, removes redundant skeletons and false detections, improves efficiency, and enhances the robustness of the tracking system.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a skeleton detection and tracking method, device and electronic equipment. Background Art

[0002] The multi-target tracking task (MOT) is one of the basic tasks of computer vision. Its core lies in associating the same object between different frames in a video. In the field of multi-target tracking, tracking-by-detection is a commonly used tracking method. This method first obtains a detection frame on each frame, and then matches the detection frames between frames to complete the tracking task. The MOT algorithm based on the Tracking-by-detaction framework first detects the target in each frame of the video sequence, crops the target according to the bounding box, and obtains all the targets in the image. Then, it is transformed into a target association problem between the previous and next frames, and a similarity matrix is ​​constructed through IoU, appearance, etc., and solved by methods such as the Hungarian algorithm and the greedy algorithm.

[0003] At present, the mainstream MOT methods are basically based on the idea of ​​tracking-by-detection, that is, first perform 2D key point detection (object detection), and then associate the detection results between frames (object association). For example, the SORT algorithm or DeepSORT algorithm commonly used in the industry are both tracking-by-detection methods.

[0004] However, the input of the detection network of the tracking-by-detection method is the grayscale image and the key point coordinates. When detecting key points, the key point coordinates are not directly regressed, but a heatmap is generated for each key point. The heatmap of the key point detection network is not effective for SORT tracking; DeepSORT needs to design a separate small network to extract the features required for tracking, which reduces efficiency. In addition, the current solution based on RGB information will have false detection in scenes with severe occlusion and interlacing, and the tracking solution based solely on deep matching is not ideal due to the lack of color information.

[0005] The disclosure of the above-mentioned background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above-mentioned content has been disclosed before the filing date of this patent application, the above-mentioned background technology should not be used to evaluate the novelty and creativity of this application. Summary of the invention

[0006] The object of the present invention is to provide a skeleton detection and tracking method, device and electronic device to solve at least one of the above-mentioned background technical problems.

[0007] To achieve the above object, the present invention comprises the following steps:

[0008] S10, obtaining an aligned two-dimensional image and a depth image, and inputting the two-dimensional image into a preset 2D skeleton network to obtain coordinates of key points of a human body and a first feature map respectively;

[0009] S20, inputting the depth image into a preset depth tracking feature extraction network to obtain a depth feature map;

[0010] S30, obtaining a 2D skeleton feature vector sequence and a deep skeleton feature vector sequence from the first feature map and the deep feature map respectively according to the coordinates of the human body key points;

[0011] S40, performing matching calculation using the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and preset corresponding skeleton sequence samples to obtain a target 2D skeleton sequence.

[0012] Another embodiment of the present invention is as follows:

[0013] A skeleton detection and tracking device, comprising:

[0014] An image acquisition module, used for acquiring aligned two-dimensional images and depth images;

[0015] A 2D skeleton network module, used for performing key point detection and feature extraction according to the input two-dimensional image, and obtaining the coordinates of the key points of the human body and the first feature map respectively;

[0016] A depth tracking feature network module is used to extract depth features according to the input depth image to obtain a depth feature map;

[0017] A feature vector sequence acquisition module, used to obtain a 2D skeleton feature vector sequence and a deep skeleton feature vector sequence from the first feature map and the deep feature map respectively according to the coordinates of the human body key points;

[0018] The target skeleton acquisition module is used to perform matching calculation using the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and each corresponding skeleton sequence sample to acquire a target 2D skeleton sequence.

[0019] Another embodiment of the present invention is as follows:

[0020] An electronic device comprises a processor, a memory, a network interface and a display screen; wherein the processor is used to provide computing and control capabilities; the memory stores an operating system and a computer program, and when the computer program is executed by the processor, the skeleton detection and tracking method described in the technical solution of any of the aforementioned embodiments is implemented.

[0021] Another embodiment of the present invention is as follows:

[0022] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the skeleton detection and tracking method described in any of the technical solutions of the aforementioned embodiments is implemented.

[0023] The beneficial effects of the technical solution of the present invention are:

[0024] Compared with the prior art, the present invention comprehensively utilizes two-dimensional images and depth information in the 2D skeleton key point tracking task, matches the outputs of the two and removes redundant skeletons and false detections, thereby improving efficiency and enhancing the robustness of the tracking system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 is a flowchart illustrating a skeleton detection and tracking method according to an embodiment of the present invention;

[0027] Figure 2 is a schematic diagram of human skeleton joints according to a skeleton detection and tracking method according to an embodiment of the present invention;

[0028] Figure 3 is a diagram of extracting a skeleton feature vector sequence according to a skeleton detection and tracking method according to an embodiment of the present invention;

[0029] Figure 4 is a schematic diagram of a skeleton detection and tracking device according to an embodiment of the present invention;

[0030] Figure 5 is a schematic diagram of the working principle of a skeleton detection and tracking device according to an embodiment of the present invention;

[0031] Figure 6 is a schematic diagram of an electronic device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the technical problems, technical solutions and beneficial effects to be solved by the embodiments of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0034] Figure 1 A schematic diagram of a skeleton detection and tracking method according to an embodiment of the present invention is provided. The method comprises the following steps:

[0035] S10, obtaining the aligned two-dimensional image and depth image, and inputting the two-dimensional image into a 2D skeleton network to obtain the coordinates of key points of the human body and the first feature map respectively.

[0036] In one embodiment, when acquiring a two-dimensional image and a depth image, it is necessary to align the two-dimensional image with the depth image, that is, to obtain a mapping relationship between each pixel on the two-dimensional image and each pixel on the depth image. It should be noted that the two-dimensional image can be any one of two-dimensional images such as an RGB image, a YUV image, or an infrared image, and is not limited here.

[0037] In one embodiment, the 2D skeleton network includes a backbone network, a key point detection branch network and a 2D tracking feature branch network, wherein the backbone network is used to detect the human frame in the two-dimensional image; the key point detection branch network is used to detect the key points of the human body to obtain the coordinates of the key points of the human body; Figure 2 As shown in the figure, the key points of the human body are represented as 18 joints. The key point detection of the human body is to detect the human body and the corresponding key point positions from the input image; the features are extracted through the 2D tracking feature branch network to obtain the first feature map featuremap A. It should be noted that the size of the two-dimensional image can be designed according to the 2D skeleton network used so that the size of the two-dimensional image conforms to the input size of the 2D skeleton network used. Specifically, the preset size can be obtained through normalization and other processing methods, which is not limited here.

[0038] It should be understood that in this step, by designing the key point detection branch network and the 2D tracking feature branch network to share a backbone network, the computational complexity of the algorithm can be reduced and the efficiency can be improved.

[0039] In one embodiment, inputting the two-dimensional image into the 2D skeleton network to obtain the coordinates of the key points of the human body and the first feature map respectively more specifically includes:

[0040] S101, inputting a two-dimensional image into a backbone network to obtain a human body detection frame in the two-dimensional image.

[0041] In some embodiments, a human body detection frame (bounding box) is first detected by a backbone network, and a human body detection frame is output. The backbone network may be YOLO, SSD, RCNN, RFCN, etc., which are not particularly limited in the embodiments of the present invention.

[0042] S102: According to the human body detection frame, the key point detection branch network is used to detect the key points of the human body skeleton to obtain the coordinates of the key points of the human body.

[0043] In some embodiments, all key points of the human body detection frame are detected by the key point detection branch network, and then all the detected key points are clustered to obtain the coordinates of the key points of the human body. Clustering all key points can be performed by modeling different limb structures of the human body, using vector fields to simulate different limb structures to cluster key points; of course, key point clustering can also be completed by segmenting different parts of the human body and modeling the relationship between key points through part segmentation.

[0044] It should be noted that the key point detection branch network can be G-RMI, CFN, RMPE, Mask R-CNN, CPN, etc., and is not limited here.

[0045] S103: Extract features through a 2D tracking feature branch network according to the human body detection frame to obtain a first feature map.

[0046] In some instances, a two-dimensional image is input into a 2D tracking feature branch network, and features are extracted through the 2D tracking feature branch network to obtain a first feature map featuremap A; specifically, a human body detection frame is convolved through the 2D tracking feature branch network to obtain a first feature map featuremap A of the human body detection frame. Preferably, the two-dimensional image is a 3-channel RGB image, and the human body detection frame obtained based on the two-dimensional image detection is also a 3-channel RGB image. At this time, the first layer convolution kernel of the 2D tracking feature branch network needs to use an n×n×3 convolution kernel so that the convolution kernel is consistent with the number of channels of the input feature map. Specifically, the input is a 32×32×3 image, 3 represents RGB three channels, and the convolution kernel of the first layer of the 2D tracking feature branch network is 5×5×3, and one convolution kernel produces a feature map feature map. It should be noted that, except for the first layer of convolution, the remaining convolution kernels of the 2D tracking feature branch network can also be 3×3×1 or other sizes, and can be designed according to the semantic information of the required feature map; in addition, in addition to the convolution layer, the 2D tracking feature branch network can also include BN layers, pooling and other network layers. The BN layer and pooling layer can specifically adopt existing technologies and will not be described in detail in the embodiments of the present invention.

[0047] S20, input the depth image into the depth tracking feature extraction network to obtain a depth feature map.

[0048] In some embodiments, a depth tracking feature extraction network is used to extract features from a depth image to obtain a depth feature map; specifically, the depth tracking feature extraction network uses a single-channel convolution kernel, and obtains a depth feature map featuremap B through convolution operations. It should be noted that the input of the depth image is a single-channel depth map, and the convolution kernel is n×n×1. It should be understood that the depth tracking feature extraction network also includes network layers such as a pooling layer and a BN layer, which will not be described in detail this time.

[0049] It should be noted that the size of the depth image can be designed according to the depth tracking feature extraction network used so that the size of the depth image conforms to the input size of the depth tracking feature extraction network used. Specifically, the preset size can be obtained through normalization and other processing methods, which is not limited here.

[0050] In addition, the size of the depth map input in step S20 may be equal to or different from the size of the two-dimensional image input in step S10, which is not particularly limited in the embodiments of the present invention. Regardless of the method adopted, it shall fall within the protection scope of this application.

[0051] S30. Obtain a 2D skeleton feature vector sequence and a deep skeleton feature vector sequence from the first feature map and the deep feature map respectively according to the coordinates of the key points of the human body.

[0052] In one embodiment, the coordinates of the key points of the human body are obtained according to step S20 to obtain the bounding box (i.e., bounding box, abbreviated as Bbox) of the key points of the human body, and the first feature map and the coordinate information of the bounding box are input into a preset region of interest pooling (ROIpooling) layer. More specifically, the ROI pooling layer maps the coordinate information of the bounding box to the first feature map, obtains the feature value of the bounding box corresponding to the key points of the human body on the first feature map, and then extracts the feature value of each key point in the bounding box based on the bounding box of the first feature map, and further transforms it into a feature of a fixed size length, so that the fixed-size features corresponding to each key point are composed of a 2D image skeleton feature vector sequence, such as Figure 3 shown.

[0053] Similarly, according to the alignment relationship between the two-dimensional image and the depth image in step S10, the depth feature map and the coordinate information of the bounding box of the key points of the human body are input into the ROI pooling layer. The ROI pooling layer maps the bounding box coordinates obtained above to the depth feature map, obtains the feature value of the bounding box of the key points of the human body on the corresponding depth feature map, and then extracts the feature value of each key point in the depth feature map based on the bounding box of the depth feature map, and further transforms it into a feature of a fixed size length, so as to form a depth skeleton feature vector sequence using the fixed size features corresponding to each key point.

[0054] S40, performing matching calculation using the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and the corresponding skeleton sequence samples to obtain a target 2D skeleton sequence.

[0055] Specifically, the confidence calculation is performed using the 2D skeleton feature vector, the deep skeleton feature vector and the preset corresponding skeleton sequence samples to obtain the target 2D skeleton sequence corresponding to the key point matching. The confidence calculation formula is:

[0056]

[0057] Among them, x, y are the extracted 2D skeleton feature vector and deep skeleton feature vector, fi, f i ′ is the corresponding skeleton sequence sample feature vector, To match the weighted coefficient, the default value is 0.5, which can be adjusted according to the experimental results.

[0058] Furthermore, the above confidence and skeleton key point combing are compared with a preset threshold. If the preset threshold is met, the target 2D skeleton sequence is output. Preferably, a sequence with a confidence greater than 0.5 and more than 6 skeleton key points is judged to be the target 2D skeleton sequence.

[0059] In some embodiments, before the method is implemented, the method further includes the steps of:

[0060] The skeleton sequence samples are obtained by detecting the two-dimensional image and the depth image through the skeleton key point network; wherein the skeleton key point network can be open pose or lightweight open pose. It should be noted that the network model used to obtain the skeleton sequence samples is a highly complex super-large network model, which can extract rich features to ensure the accuracy of the skeleton sequence samples, but the network model requires high-performance computing and has a slow calculation speed.

[0061] In some embodiments, in step S40, when the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and the key point coordinates are used as input, the SORT algorithm can be used for matching, and the target 2D skeleton sequence that meets the preset threshold is output. Specifically, the Mahalanobis distance is calculated through the key point coordinates to obtain the motion information of the target, and the Euclidean distance or the angle cosine distance is calculated through the 2D skeleton feature vector and the deep skeleton feature vector to obtain the appearance information of the target. Finally, the motion information and appearance information of the target are fused and weighted, and the weighted value is compared with the preset threshold to eliminate redundant skeletons and false detections, and the target 2D skeleton sequence is obtained.

[0062] The skeleton detection and tracking method of the embodiment of the present invention realizes human body key point detection and feature extraction by sharing the same backbone network through the key point detection branch and the 2D tracking feature branch network, and then extracts the depth features through the depth tracking feature extraction network. Finally, the outputs of the two are combined to calculate and obtain the target 2D skeleton sequence, thereby reducing the computational complexity of the algorithm, improving efficiency, and enhancing the robustness of the tracking system.

[0063] Figure 4 This is a schematic diagram of a skeleton detection and tracking device provided according to another embodiment of the present invention, the device includes: an image acquisition module 400, used to acquire aligned two-dimensional images and depth images; a 2D skeleton network module 410, used to perform key point detection and feature extraction based on the input two-dimensional image, and obtain human body key point coordinates and a first feature map respectively; a depth tracking feature network module 420, used to extract depth features based on the input depth map, and obtain a depth feature map; a feature vector sequence acquisition module 430, used to obtain a 2D skeleton feature vector sequence and a deep skeleton feature vector sequence from the first feature map and the depth feature map respectively according to the human body key point coordinates; a target skeleton acquisition module 440, used to perform matching calculations using the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and the corresponding skeleton sequence samples, and obtain a target 2D skeleton sequence.

[0064] In one embodiment, referring to Figure 5As shown, the 2D skeleton network module 410 includes a backbone network, a key point detection branch network and a 2D tracking feature branch network, wherein the backbone network is used to detect the human body frame in the two-dimensional image; the key point detection branch network is used to perform human body key point detection and obtain the coordinates of human body key points; the 2D tracking feature branch network is used to perform feature extraction on the two-dimensional image to obtain the first feature map, and obtain the first feature map; wherein the key point detection branch network and the 2D tracking feature branch network share a backbone network.

[0065] In one embodiment, the depth tracking feature network module 420 includes a depth tracking feature extraction network for performing feature extraction on an input depth map to obtain a depth feature map; specifically, the depth tracking feature extraction network is a convolutional neural network, which uses a single-channel convolution kernel to obtain a depth feature map through a convolution operation.

[0066] In one embodiment, the feature vector sequence acquisition module 430 includes a ROI pooling processing network, wherein the ROI pooling processing network includes a ROI pooling layer for receiving input human key point coordinates and a first feature map, processing the feature value of each key point in the first feature map into a feature of a fixed size and length, and obtaining a two-dimensional image skeleton feature vector sequence; the ROI pooling layer is also used to receive input human key point coordinates and a depth feature map, processing the feature value of each key point in the depth feature map into a feature of a fixed size and length, and obtaining a depth skeleton feature vector sequence.

[0067] In one embodiment, the target skeleton acquisition module 440 performs confidence calculation using the 2D skeleton feature vector, the deep skeleton feature vector and each corresponding skeleton sequence sample to obtain the target 2D skeleton sequence corresponding to the key point match. The confidence calculation formula is:

[0068]

[0069] Among them, x, y are the extracted 2D skeleton feature vector and deep skeleton feature vector, fi, f i ′ is the corresponding skeleton sequence sample feature vector, To match the weighted coefficient, the default value is 0.5, which can be adjusted according to the experimental results.

[0070] It should be noted that the skeleton detection and tracking device of the embodiment of the present invention is used to execute the skeleton detection and tracking method of the aforementioned embodiment. The detailed description of the specific functions of each module can be found in the description of the skeleton detection and tracking method embodiment, which will not be repeated here.

[0071] The present application also provides an electronic device, such as Figure 6As shown, the electronic device may include one or more processors 600 ( Figure 6 Only one is shown), a memory 610 and a computer program 620 stored in the memory 610 and executable on one or more processors 600, for example, a program of a skeleton detection and tracking method. When one or more processors 600 execute the computer program 620, the various steps in the above-mentioned skeleton detection and tracking method embodiment can be obtained. Alternatively, when one or more processors 600 execute the computer program 620, the functions of the various modules / units in the above-mentioned skeleton detection and tracking device embodiment can be implemented, which is not limited here.

[0072] In one embodiment, the electronic device further includes an acquisition device, which may include a depth camera and a two-dimensional camera. The two-dimensional camera may be any one of an infrared camera, a color camera, etc., for respectively acquiring depth images and two-dimensional images, and the processor uses the acquired depth images and two-dimensional images to achieve the various steps in the above-mentioned skeleton detection and tracking method embodiment. It should be noted that the depth camera and the two-dimensional camera may be integrated or independently provided, and the acquisition device may be provided in the electronic device, or may transmit data with the electronic device through wireless communication or wired communication, which is not limited here.

[0073] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0074] The present invention also proposes a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the skeleton detection and tracking method of the above embodiment scheme is implemented. The storage medium can be implemented by any type of volatile or non-volatile storage device, or a combination thereof.

[0075] Embodiments of the present invention may include or utilize a special or general-purpose computer including computer hardware, as discussed in more detail below. Embodiments within the scope of the present invention also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any available media that can be accessed by a general or special-purpose computer system. The computer-readable medium that stores computer-executable instructions is a physical storage medium. The computer-readable medium that carries computer-executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present invention may include at least two distinct computer-readable media: a physical computer-readable storage medium and a transmission computer-readable medium.

[0076] It is to be understood that the above content is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be deemed to belong to the protection scope of the present invention. In the description of this specification, the description of reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention.

[0077] In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradicting each other. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope defined by the appended claims.

[0078] Furthermore, the scope of the present invention is not limited to the specific embodiments of the processes, machines, manufactures, compositions of matter, means, methods, and steps described in the specification. One of ordinary skill in the art will readily appreciate that the above disclosures, processes, machines, manufactures, compositions of matter, means, methods, or steps, currently existing or to be developed, may be utilized to perform substantially the same functions as the corresponding embodiments described herein or to obtain substantially the same results as the embodiments described herein. Therefore, the appended claims are intended to include within their scope such processes, machines, manufactures, compositions of matter, means, methods, or steps.

Claims

1. A skeleton detection and tracking method, characterized in that: The steps include: S10, obtaining an aligned two-dimensional image and a depth image, and inputting the two-dimensional image into a preset 2D skeleton network to obtain the coordinates of key points of a human body and a first feature map respectively; the preset 2D skeleton network includes a backbone network, a key point detection branch network and a 2D tracking feature branch network; the backbone network is used to detect the human body frame in the two-dimensional image; Performing human key point detection through the key point detection branch network to obtain the coordinates of the human key points; extracting features through the 2D tracking feature branch network to obtain the first feature map; The key point detection branch network and the 2D tracking feature branch network share the backbone network; S20, inputting the depth image into a preset depth tracking feature extraction network to obtain a depth feature map; S30, obtaining a 2D skeleton feature vector sequence and a deep skeleton feature vector sequence from the first feature map and the deep feature map respectively according to the coordinates of the human body key points; S40, performing matching calculation using the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and preset corresponding skeleton sequence samples to obtain a target 2D skeleton sequence, including: The confidence calculation is performed using the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and the preset corresponding skeleton sequence samples to obtain the target 2D skeleton sequence corresponding to the key point matching; wherein the calculation formula is: Among them, x and y are the extracted 2D skeleton feature vector sequence and deep skeleton feature vector sequence respectively. is the corresponding skeleton sequence sample feature vector sequence, is the matching weighting factor.

2. The skeleton detection and tracking method according to claim 1, characterized in that: Step S10 includes: Inputting the two-dimensional image into the backbone network to obtain a human body detection frame in the two-dimensional image; According to the human body detection frame, the key point detection branch network is used to detect the key points of the human body skeleton to obtain the coordinates of the key points of the human body; According to the human body detection frame, features are extracted through the 2D tracking feature branch network to obtain a first feature map.

3. The skeleton detection and tracking method according to claim 1, characterized in that: In step S20, the depth tracking feature extraction network uses a single-channel convolution kernel to extract features from the depth image to obtain a depth feature map, wherein the size of the single-channel convolution kernel is .

4. The skeleton detection and tracking method according to claim 1, characterized in that: The step S30 comprises: Acquire the bounding box coordinate information of the human body key points according to the coordinates of the human body key points; Inputting the first feature map, the depth feature map and the bounding box coordinate information into a preset region of interest pooling layer; The preset region of interest pooling layer maps the bounding box coordinate information to the first feature map and the depth feature map, and obtains feature values ​​of the human body key point bounding box on the first feature map and the depth feature map; Extracting feature values ​​of each key point in the bounding box based on the first feature map and the bounding box of the depth feature map, and transforming them into features of fixed size and length; The fixed-size features corresponding to each key point of the first feature map and the depth feature map are respectively used to form the 2D skeleton feature vector sequence and the depth skeleton feature vector sequence.

5. A skeleton detection and tracking device, characterized in that: include: An image acquisition module, used for acquiring aligned two-dimensional images and depth images; A 2D skeleton network module is used to perform key point detection and feature extraction according to the input two-dimensional image, and obtain the coordinates of the key points of the human body and the first feature map respectively; the preset 2D skeleton network includes a trunk network, a key point detection branch network and a 2D tracking feature branch network; the trunk network is used to detect the human body frame in the two-dimensional image; Performing human key point detection through the key point detection branch network to obtain the coordinates of the human key points; extracting features through the 2D tracking feature branch network to obtain the first feature map; the key point detection branch network and the 2D tracking feature branch network share the backbone network; A depth tracking feature network module is used to extract depth features according to the input depth image to obtain a depth feature map; A feature vector sequence acquisition module, used to obtain a 2D skeleton feature vector sequence and a deep skeleton feature vector sequence from the first feature map and the deep feature map respectively according to the coordinates of the human body key points; A target skeleton acquisition module is used to perform matching calculation using the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and each corresponding skeleton sequence sample to acquire a target 2D skeleton sequence; comprising: The confidence calculation is performed using the 2D skeleton feature vector sequence, the deep skeleton feature vector sequence and the preset corresponding skeleton sequence samples to obtain the target 2D skeleton sequence corresponding to the key point matching; wherein the calculation formula is: Among them, x and y are the extracted 2D skeleton feature vector sequence and deep skeleton feature vector sequence respectively. is the corresponding skeleton sequence sample feature vector sequence, is the matching weighting factor.

6. An electronic device, characterized in that: It comprises a processor, a memory, a network interface and a display screen; wherein the processor is used to provide computing and control capabilities; the memory stores an operating system and a computer program, and when the computer program is executed by the processor, the skeleton detection and tracking method according to any one of claims 1 to 4 is implemented.

7. The electronic device according to claim 6, characterized in that: It also includes an acquisition device, which can acquire two-dimensional images and depth images, and the processor uses the two-dimensional images and the depth images to implement the skeleton detection and tracking method described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the skeleton detection and tracking method according to any one of claims 1 to 4 is implemented.

Images