Skeleton detection method, system, device and medium

Abstract

The present invention belongs to the field of computer vision and discloses a skeleton detection method, system, device and medium, comprising: inputting an image to be detected into a dual-branch convergence network model, and outputting a skeleton detection result of the image to be detected; wherein the dual-branch convergence network model includes a backbone network; the backbone network is used to obtain feature maps of several levels of different resolutions; a main branch network is used to aggregate and perform pixel regression processing on the feature maps of several levels of different resolutions to obtain a main branch prediction result; an auxiliary branch network is used to perform cascade convergence processing and pixel regression processing on the feature maps of several levels of different resolutions to obtain an auxiliary branch prediction result; an output network is used to perform feature fusion on the main branch and auxiliary branch prediction results; the present invention models multi-scale features through a dual-branch structure, enhances the expression capability of the network feature space, and meets the skeleton detection requirements of large-scale samples or objects in which the target object occupies a large proportion of the image.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a skeleton detection method, system, device and medium. Background Art

[0002] Skeleton representation describes the complex and changeable object contours, connectivity, symmetry and other properties in the form of simple lines. As an important representation method in the development of computer vision, it has important application value in computer vision, computer graphics, image processing and other fields.

[0003] At present, most of the existing skeleton detection methods are skeleton detection methods based on deep learning; however, there are two types of problems with skeleton detection methods based on deep learning. First, due to the influence of factors such as lighting, contrast, and background noise in natural images, the object skeletons of complex samples cannot be effectively detected; second, the scales of various parts of objects in natural images vary greatly, making it impossible to accurately capture large-scale features.

[0004] With the application of Transformer-based methods (a deep learning model based on the self-attention mechanism) in skeleton detection, it can effectively detect the skeleton of objects in natural image samples. From a qualitative perspective, it is found that the existing Transformer-based methods can indeed effectively improve long-distance modeling capabilities, and to some extent solve the problem of incorrect skeleton area detection caused by traditional CNN models due to background clutter and similar foreground and background.

[0005] However, when faced with large-scale image samples in natural images, although the existing Transformer-based methods can achieve approximate global feature extraction, their ability to model multi-scale features is still insufficient. Figure 1 As shown; attached Figure 1 The problem that the existing Transformer-based method predicts broken or inaccurate skeletons on some difficult samples is given in the paper. Figure 1 It can be seen that for large-scale samples or objects where the target object occupies a large proportion of the image, neither the traditional CNN method nor the existing Transformer-based method can accurately extract the skeleton of the target object. There are two main reasons for this: 1) The target object in the image occupies most of the space of the entire image, and the network needs to have a receptive field of the entire image; 2) Detecting large parts of the target object requires the network to have a high multi-scale feature extraction capability. Summary of the Invention

[0006] In response to the technical problems existing in the prior art, the present invention provides a skeleton detection method, system, device and medium to solve the technical problem that the existing skeleton detection method cannot accurately extract the skeleton of the target object for large-scale samples or objects with a large proportion of the target object in the image.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] The present invention provides a skeleton detection method, comprising:

[0009] Obtain an image to be detected I;

[0010] Inputting the image to be detected I into a dual-branch convergence network model, and outputting a skeleton detection result of the image to be detected I; wherein the dual-branch convergence network model includes a backbone network, a main branch network, an auxiliary branch network and an output network;

[0011] The backbone network is used to extract features from the image to be detected I and obtain feature maps X at several levels of different resolutions. i ;

[0012] The main branch network is used to analyze several levels of feature maps X with different resolutions. i Perform aggregation to obtain the aggregated feature map F X ; and the aggregate feature map F X Perform pixel regression processing to obtain the main branch prediction result Y X ;

[0013] The auxiliary branch network is used to analyze several levels of feature maps X with different resolutions. i Perform cascade aggregation processing to obtain the aggregated feature map F X '; and the aggregate feature map F X 'Perform pixel regression processing to obtain the auxiliary branch prediction result Y X ';

[0014] The output network is used to predict the result Y of the main branch X and the auxiliary branch prediction result Y X ' Perform feature fusion to obtain the skeleton detection result of the image to be detected I.

[0015] Furthermore, the backbone network is a Swin-Transformer network structure.

[0016] Furthermore, the main branch network includes a Sem-FPN module, a ShiftASPP module and a main branch classifier module;

[0017] The Sem-FPN module is used to generate several levels of feature maps X with different resolutions.i Perform channel-aligned convolution processing and perform feature map size recovery operations to obtain several channel-aligned feature maps; sum up several channel-aligned feature maps to obtain an aggregated result;

[0018] The ShiftASPP module is used to perform shift convolution processing on the aggregation result and then perform hole convolution calculation to obtain the aggregation feature map F X ;

[0019] The main branch classifier module is used to classify the aggregate feature map F X Perform convolution processing to obtain the main branch prediction result Y X .

[0020] Furthermore, the ShiftASPP module includes four parallel branch modules; each branch module includes a shift convolution module and a dilated convolution pooling pyramid module;

[0021] The shift convolution module is configured to perform a first convolution operation on the aggregation result to obtain a first convolution feature map; perform a shift operation on the first convolution feature map to obtain a shifted feature map; perform a second convolution operation on the shifted feature map to obtain a second convolution feature map; and perform an inverse shift operation on the second convolution feature map to obtain a shifted convolution feature map;

[0022] The dilated convolution pooling pyramid module is used to perform dilated convolution calculation on the shifted convolution feature map to obtain an aggregated feature map F X .

[0023] Furthermore, the auxiliary branch network includes a first-level convergence layer, a feature fusion module, a second-level convergence layer and an auxiliary branch classifier module;

[0024] The first-level convergence layer is used to input the feature maps of three adjacent levels with different resolutions in parallel and perform window self-attention calculation to obtain several sets of self-attention results A i ;

[0025] The feature fusion module is used to combine several sets of self-attention results A i Perform an additive fusion operation to obtain a fusion result A with several levels of different scale features;

[0026] The second-level pooling layer is used to perform self-attention calculation on the fusion result A with several levels of different scale features to obtain the pooling result; the feature maps X with several levels of different resolutions are i The residual of is summed with the aggregation result and mapped in the feature space to obtain the aggregate feature map F X ';

[0027] The auxiliary branch classifier is used to classify the aggregated feature map F X 'Convolution processing is performed to obtain the auxiliary branch prediction result Y X ′.

[0028] Furthermore, the loss function of the dual-branch convergence network model is:

[0029] L=L A +αL B

[0030]

[0031] Among them, L is the total loss of the network; L A is the main branch loss; L B is the auxiliary branch loss; α is a hyperparameter used to balance the main branch and the auxiliary branch; W i is the pixel weight; X i are feature maps of different resolutions; is the proxy label after the skeleton line neighbor gradient representation processing; i is each pixel.

[0032] Furthermore, the image I to be detected is a natural image that has undergone data enhancement.

[0033] The present invention also provides a skeleton detection system, comprising:

[0034] An input module is used to obtain an image I to be detected;

[0035] A detection output module is used to input the image to be detected I into a dual-branch convergence network model and output a skeleton detection result of the image to be detected I; wherein the dual-branch convergence network model includes a backbone network, a main branch network, an auxiliary branch network and an output network;

[0036] The backbone network is used to extract features from the image to be detected I and obtain feature maps X at several levels of different resolutions. i ;

[0037] The main branch network is used to analyze several levels of feature maps X with different resolutions. i Perform aggregation to obtain the aggregated feature map F X ; and the aggregate feature map F X Perform pixel regression processing to obtain the main branch prediction result Y X ;

[0038] The auxiliary branch network is used to analyze several levels of feature maps X with different resolutions. i Perform cascade aggregation processing to obtain the aggregated feature map F X '; and the aggregate feature map F X'Perform pixel regression processing to obtain the auxiliary branch prediction result Y X ';

[0039] The output network is used to predict the result Y of the main branch X and the auxiliary branch prediction result Y X ' Perform feature fusion to obtain the skeleton detection result of the image to be detected I.

[0040] The present invention provides a skeleton detection device, comprising:

[0041] a processor suitable for executing a computer program;

[0042] A computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the skeleton detection method is performed.

[0043] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the skeleton detection method is implemented.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The skeleton detection method provided by the present invention performs feature extraction on the image to be detected through a backbone network to obtain feature maps of several levels of different resolutions, uses a main branch network to perform multi-scale extraction and decoding on the feature maps of several levels of different resolutions, uses an auxiliary branch network to model the relationship between the feature maps of several levels of different resolutions, and performs feature fusion on the results of the main branch network and the auxiliary branch network to realize skeleton extraction of the image to be detected; in the present invention, multi-scale features are modeled through a dual-branch structure to enrich the information of the overall semantic space of the network, enhance the expression ability of the network feature space, and improve the multi-scale detection capability, thereby meeting the skeleton detection needs of large-scale samples or objects in which the target object occupies a large proportion of the image.

[0046] Furthermore, a shift operation is introduced in the convolution operation of the ShiftASPP module. By changing the relative positions between the pixels of the convolution feature map, the long-distance interaction ability of the convolution feature is improved, so as to enhance the information exchange ability of the feature map during the convolution process. By shifting a small number of pixels, performing convolution operations and activating them, the effect of enhancing the nonlinear ability of the network is achieved. Secondly, the shift convolution operation is combined with the void convolution pooling pyramid. The multi-scale feature extraction capability is enhanced through the shift operation between feature maps.

[0047] Furthermore, a cascaded convergence network is proposed in the auxiliary branch to perform cascaded self-attention calculations on feature maps of several levels with different resolutions. This can further extract the similarities between features of different scales and model the degree of correlation between the multi-level and different-scale feature maps of the Transformer, so as to supplement the feature expression of the main branch, enrich the information in the overall feature space of the network, and enhance the expressive ability of the network feature space.

[0048] The skeleton detection system, skeleton detection device and computer-readable storage medium provided by the present invention have all the advantages of the above-mentioned skeleton detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0050] Figure 1 Skeleton result map predicted by existing Transformer-based methods on difficult samples;

[0051] Figure 2 This is a schematic diagram of the structure of the dual-branch convergence network model in Example 1;

[0052] Figure 3 Schematic diagram of the structure of the shifted dilated convolution pooling pyramid module in Example 1;

[0053] Figure 4 Schematic diagram of the structure of the auxiliary branch network in Example 1;

[0054] Figure 5 This is a visualization result diagram of the two-level output feature map of the auxiliary branch network in Example 1;

[0055] Figure 6 This is a qualitative analysis diagram of the prediction results of the dual-branch convergence network model described in Example 1 and other existing models;

[0056] Figure 7 A structural block diagram of the skeleton detection system provided in Example 2;

[0057] Figure 8 This is a structural block diagram of the skeleton detection device provided in Example 3. DETAILED DESCRIPTION

[0058] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention more clearly understood, the present invention is further described in detail in the following specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] Example 1

[0060] As attached Figure 2 As shown, this embodiment 1 provides a skeleton detection method, including the following steps:

[0061] Step 1: Obtain an image to be detected I. The image to be detected I is a natural image that has undergone data enhancement. It should be noted that the data enhancement method for natural images uses existing image processing methods, such as pixel-level transformation or spatial-level transformation, which is not an improvement point of the present application and will not be described in detail here.

[0062] Step 2: Input the image to be detected I into a dual-branch cascaded aggregation transformer (DCFormer) model, and output the skeleton detection result of the image to be detected I.

[0063] In this embodiment 1, the dual-branch convergence network model uses two parallel branches to complement each other and enrich the semantic space features; wherein, the auxiliary branch network performs cascade sub-attention calculations on the four-level side outputs of the backbone network at different scales, which can further extract the similarity between features of different scales; specifically, the dual-branch convergence network model includes a backbone network, a main branch network, an auxiliary branch network and an output network, as shown in the attached figure. Figure 2 shown.

[0064] The backbone network is used to extract features from the image to be detected I and obtain feature maps X at several levels of different resolutions. i The main branch network and the auxiliary branch network serve as network decoders of the dual-branch convergence network model to model multi-scale features and enhance the expression ability of the network feature space; the main branch network is used to perform multi-scale feature map X on several levels of different resolutions. i Perform aggregation to obtain the aggregated feature map F X ; and the aggregate feature map F X Perform pixel regression processing to obtain the main branch prediction result Y X ; The auxiliary branch network is used to analyze several levels of feature maps X with different resolutions i Perform cascade aggregation processing to obtain the aggregated feature map F X '; and the aggregate feature map F X 'Perform pixel regression processing to obtain the auxiliary branch prediction result YX '; The output network is used to predict the result Y of the main branch X and the auxiliary branch prediction result Y X ' Perform feature fusion to obtain the skeleton detection result of the image to be detected I.

[0065] In this embodiment 1, the backbone network is a Swin-Transformer network structure. Specifically, after the image to be detected I is input into the Swin-Transformer network structure, it passes through the four-level network of the Swin-Transformer network structure to obtain four-level side outputs, namely, four-level feature maps X with different resolutions. i , 1≤i≤4; It should be noted that the Swin-Transformer network structure is a Transformer-based visual model, which introduces a window attention mechanism, a hierarchical structure, and a patch merging layer to solve the problems of computational complexity and insufficient local feature extraction.

[0066] The window attention mechanism is as follows: the input image is divided into non-overlapping windows, and self-attention is calculated within each window. It limits the scope of attention calculation to effectively capture the local features of the image while maintaining computational efficiency; the hierarchical structure is as follows: Swin Transformer adopts a hierarchical structure, and each layer calculates features through self-attention within the window; as the number of layers increases, the size of the window gradually increases, the resolution of the feature map gradually decreases, and the number of channels gradually increases, so that the model can capture features at different levels and is suitable for multi-scale visual tasks; the patch merging layer is used to reduce the size of the feature map and increase the number of channels. It merges adjacent patches to reduce the resolution while increasing the number of channels, thereby extracting higher-level features; the shift window mechanism is as follows: Swin Transformer shifts the window by half its size every other layer, so that information across windows can be fused through multi-layer superposition to achieve a larger receptive field, thereby enhancing the model's global context capture capability while maintaining computational efficiency.

[0067] In this embodiment 1, the main branch network includes a Sem-FPN module, a ShiftASPP module and a main branch classifier module; the Sem-FPN module is used to classify the feature maps X at several levels with different resolutions. i Perform channel-aligned convolution processing and perform feature map size recovery operation to obtain several channel-aligned feature maps; sum the several channel-aligned feature maps to obtain an aggregated result; the ShiftASPP module is used to perform shift convolution processing on the aggregated result and then perform hole convolution calculation to obtain an aggregated feature map F X ; The main branch classifier module is used to classify the aggregate feature map FX Perform convolution processing to obtain the main branch prediction result Y X .

[0068] In the main branch network, the Sem-FPN module is used as a multi-scale feature extractor to aggregate the feature maps of different resolutions output by the 1-4 level side of the backbone network; the specific operation is: the feature maps of different resolutions at each level X i Perform channel-consistent convolution and feature map size recovery operation, and finally integrate the obtained results by summing them up, and then connect them to the ShiftASPP module and the main branch classifier module; specifically, the feature map X i Input the Sem-FPN module for aggregation to obtain the aggregation result; the aggregation result is sent to the ShiftASPP module to obtain the aggregation feature map F X ; Then, aggregate feature map F X Perform 1×1 convolution layer for pixel regression to obtain the main branch prediction result Y X .

[0069] As attached Figure 3 As shown, the ShiftASPP module includes four parallel branch modules; each branch module includes a shift convolution module and a dilated convolution pooling pyramid module; the shift convolution module is used to perform a first convolution operation on the aggregation result to obtain a first convolution feature map; the first convolution feature map is shifted to obtain a shifted feature map; the shift feature map is second convolution operation to obtain a second convolution feature map; the second convolution feature map is reverse shifted to obtain a shifted convolution feature map; the dilated convolution pooling pyramid module is used to perform dilated convolution calculation on the shifted convolution feature map to obtain an aggregated feature map F X .

[0070] Specifically, in each branch module, the input feature map undergoes a 3×3 convolution operation, and then the obtained feature map is shifted. The shifted feature map is convolved again with 3×3, and the result is shifted in reverse to restore the feature to its original position. The shift operation improves the element interaction during the convolution process; finally, the ASPP module is used to perform a dilated convolution calculation to extract semantic information of different scales from feature maps of the same scale. In this way, the calculation can be completed through four parallel branches.

[0071] It should be noted that Atrous Spatial Pyramid Pooling (ASPP) is an important means of multi-scale feature extraction and aggregation, which has made an important contribution to the improvement of skeleton detection performance. At the same time, inspired by the shift window self-attention in Swin Transformer, the shift operation can strengthen the information interaction within the feature map when the window size is determined, so that the features that were originally not in the same window are moved to the same window through the shift operation, completing the self-attention calculation and extracting richer semantic information. Therefore, the shift operation is applied to the convolution operation in order to improve the information interaction of the convolution feature map, so that the convolution operation can indirectly increase the receptive field size. In the ShiftASPP module, the shift convolution operation is combined with the atrous convolution pooling pyramid. Through the shift operation between feature maps, the multi-scale feature extraction capability is enhanced, which can effectively improve the performance of the network.

[0072] It should also be noted that the purpose of the shift convolution operation is to improve the information exchange capability of the feature map during the convolution process. By performing a shift operation on a small number of pixels, and then performing convolution operations and activation, the effect of enhancing the nonlinear capability of the network is achieved. Specifically, in the shift convolution module, the shift operation is introduced into the convolution operation, and the feature map after the convolution is completed is subjected to an inverse transformation operation.

[0073] The following describes the working principle of the ShiftASPP module in detail by taking a set of feature maps containing a preset number as an example:

[0074] First, each group is set to contain 12 feature maps; the feature maps are convolved to increase the number of channels to the maximum number of groups, and then the first four groups are cyclically shifted left, right, up, and down to obtain pixel-shifted feature maps; then, convolution operations are performed and activated, and finally, the shifted feature maps are reverse-shifted back to the original pixel position, and subsequent dilated convolution operations are performed.

[0075] Specifically, the feature map input to the ShiftASPP module is a feature map that has been extracted by the Sem-FPN module at multiple scales, and contains multi-scale semantic information. Next, it is further decoded by ShiftASPP, that is, the single-scale feature map is sampled at different scales using dilated convolution, with the aim of further mining multi-scale feature information. The feature map will first pass through the first convolution layer in ShiftConvBlock, with the aim of increasing the number of feature map channels to meet the subsequent requirements for grouping feature maps. At the same time, it can also further reproject features of different scales in the feature space to improve the differentiation of features of each branch and enhance the network's expression capabilities. Each parallel branch performs a partial channel shift operation. The purpose of the shift operation is to change the pixel order between the original feature maps by performing a circular shift operation on some groups. By disrupting the pixel positions of the feature maps, elements that were originally impossible to be filtered at the same time will be allowed to undergo the next linear mapping. The resulting feature map will contain more feature information than the unshifted feature map, which improves the interaction ability between elements of the feature maps to a certain extent. Since the skeleton detection task is a pixel-level task, the ultimate goal is to classify each pixel in the feature map. Therefore, after completing the second convolution operation, the feature map needs to be reverse shifted to restore the initial position of each element and continue with the subsequent multi-scale feature extraction.

[0076] In this embodiment 1, a cascaded convergence layer and a classifier are set in the auxiliary branch network; wherein, the cascaded convergence layer is output by the four-level side of the associated Transformer. On the basis of the Transformer focusing on the self-attention of each level, the cascaded convergence network pays attention to the self-attention of feature maps at different levels, from which more semantic information is sought, thereby enriching the semantic space representation; specifically, the auxiliary branch network includes a first-level convergence layer, a feature fusion module, a second-level convergence layer and an auxiliary branch classifier module, as shown in the attached figure. Figure 4 shown.

[0077] The first-level convergence layer is used to input the feature maps of three adjacent levels with different resolutions in parallel and perform window self-attention calculation to obtain several sets of self-attention results A i ; The feature fusion module is used to combine several sets of self-attention results A i Perform an addition fusion operation to obtain a fusion result A with several levels of different scale features; the second-level pooling layer is used to perform self-attention calculation on the fusion result A with several levels of different scale features to obtain a pooling result; the feature maps X with several levels of different resolutions are combined i The residual of is summed with the aggregation result and mapped in the feature space to obtain the aggregated feature map F X '; The auxiliary branch classifier is used to classify the aggregate feature map FX 'Convolution processing is performed to obtain the auxiliary branch prediction result Y X ′.

[0078] In the auxiliary branch network, by designing a cascaded pooling layer, two-level self-attention calculation is performed on the output features of the backbone network at different scales to refine the relationship between feature maps of different scales and improve the feature representation ability of the overall model in the semantic feature space; among them, the cascaded pooling layer is divided into two levels, the first level is divided into two modules, which are used to extract feature maps of three adjacent scales respectively; the second level will use the two feature maps of different scales calculated in the first level to converge again, and the result is obtained by further relationship modeling of the four levels of different scales of the Swin Transformer network structure.

[0079] Specifically, the adjacent three levels of the four-level side output of the Swin Transformer network structure are input into the cascade pooling layer, that is, the feature maps X of the four levels with different resolutions are converted into i , 1≤i≤4, the adjacent three levels are input respectively; among them, S1={X1,X2,X3} and S2={X2,X3,X4} are used as two groups of inputs, which are input to the cascaded pooling layer in parallel; first, after the first-level pooling layer, the window self-attention calculation is performed on the inputs S1 and S2 to obtain two groups of self-attention results A1 and A2; then, in the feature fusion module, the two groups of self-attention results A1 and A2 are added and fused to obtain a fusion result A with four levels of different scale features, and then the fusion result A with four levels of different scale features is input to the second-level pooling layer, and the self-attention calculation is performed to complete the re-extraction and projection operation of the four-level scale features, and then the feature map X with four different resolutions is used. i The residual is summed with the pooling result and mapped again in the feature space to complete the entire pooling operation process.

[0080] In this embodiment 1, both the main branch network and the auxiliary branch network are supervised by the proxy label after the skeleton line neighbor gradient representation processing; the main branch prediction result Y X and the auxiliary branch prediction result Y X 'In the process of feature fusion and obtaining the skeleton detection result of the image to be detected I, the weighted Euclidean loss is used to transform the main branch prediction result Y X and the auxiliary branch prediction result Y X ’ are connected; wherein, the preset hyperparameters are mainly used to balance the contribution of the main branch network and the auxiliary branch network to the feature space. Since the auxiliary branch only models the features of different scales without further feature decoding, it serves as an auxiliary branch; therefore, the loss function of the dual-branch convergence network model is:

[0081] L=L A +αL B

[0082]

[0083] Among them, L is the total loss of the network; L A is the main branch loss; L B is the auxiliary branch loss; L A and L B Both are weighted Euclidean losses; α is a hyperparameter used to balance the main branch and auxiliary branches; W i is the pixel weight; X i are feature maps of different resolutions; is the proxy label after the skeleton line neighbor gradient representation processing; i is each pixel.

[0084] Experimental results and analysis:

[0085] In order to verify the performance of the dual-branch convergence network model in skeleton detection, training and evaluation were performed on four preset datasets, and compared with the SkeSwin method and the CNN-based method. The evaluation indicator was F-measure. Among the CNN-based methods, DeepFlux and ProMask improved the network proxy label, Hi-Fi and MSB+ adopted a multi-scale image fusion strategy, GeoSkeletonNet used Hough distance to obtain geometric perception capabilities, and AdaLSN adopted a neural network architecture search method. The above methods are all CNN-based methods; the SkeSwin method is based on Transformer.

[0086] The quantitative comparison results of the DCFormer and classic skeleton detection methods are shown in Table 1 below. In Table 1 below, methods 1-9 are all CNN-based methods, methods 10-14 are Swin Transformer-based skeleton detection methods SkeSwin, and methods 15-18 are dual-branch cascade convergence networks DCFormer. Among them, DCFormer-T, DCFormer-S, DCFormer-B and DCFormer-L represent Tiny, Small, Base and Large variants based on Swin Transformer, respectively.

[0087] Table 1 Quantitative comparison of DCFormer and classic skeleton detection methods

[0088]

[0089]

[0090] It should be noted that some method names in Table 1 above are followed by superscripts, which represent their backbone networks. V represents VGG16, R represents ResNet50, and I represents Inception V3. In the DeepFlux method, different suffixes are used. P indicates that the network output needs to be post-processed to obtain the predicted skeleton image, and E indicates that the network itself is an end-to-end network, and the network output result is the predicted skeleton image.

[0091] As can be seen from Table 1 above, the DCFormer surpasses all current CNN methods on four commonly used datasets and reaches the most advanced level (State Of The Art, SOTA); it also verifies that the skeleton detection method based on Transformer still has great potential, and the DCFormer will become a new baseline for skeleton detection tasks; among them, DCFormer-L is quantitatively compared with the most advanced CNN method. The F-measure on the SK-LARGE dataset is 0.794, surpassing the method AdaLSN using neural network structure search; the F-measure on the WH-SYMMAX dataset is 0.903, and the best method also surpasses AdaLSN; the F-measure on the SYM-PASCAL dataset is 0.607, compared with the best CNN method DeepFlux R -E improved by 1.4%; F-measure was 0.550 on the SYMMAX300 dataset, compared with the best CNN method DeepFlux V -E is improved by 1.9%; this shows that DCFormer adopts the Large variant, surpassing all current optimal performance and reaching an advanced level.

[0092] In this embodiment 1, the auxiliary branch network is used to model the feature maps of different scales at all levels to find more semantic information; wherein, the front-stage and back-stage convergence results in the cascade convergence network are visualized, as shown in the attached figure. Figure 5 As shown; among them, Figure 5 a and attached Figure 5 b is the feature map output of the first-level pooling layer, which represents the modeling of the first three-level side output set S1 and the last three-level side output set S2 of the backbone network respectively; it can be found that the feature map A1 after pooling for S1 integrates the semantic information of the low-level feature map, and contains more low-level semantic information: the blurred outline of the object and a small amount of skeleton information; while the feature set A2 after pooling for S2 integrates the semantic information of the high-level feature map, and contains more high-level semantic content: a small amount of outline features and skeleton features. This further shows that the pooling layer has the ability to fuse multi-scale features; Appendix Figure 5 Figure c shows the output result of the second cascade convergence layer. It is obvious that the cascade convergence network integrates multi-scale semantic information of different levels, improves the representation ability of the network model feature space, can accurately locate objects and distinguish backgrounds, and then integrates low-level contour information with high-level skeleton information, and finally presents a state of skeleton area activation and inhibition near the skeleton; from the attached Figure 5 The content shown in verifies that the idea of ​​​​the cascade convergence network to model the relationship between features of different scales can enrich the content of the network semantic feature space, thereby helping the network to better predict the skeleton position.

[0093] In this embodiment 1, large-scale samples and difficult samples are selected from four data sets, and DeepFlux is used respectively. V -P, ProMask, AdaLSN, SkeSwin-L and DCFormer generated predicted images are qualitatively analyzed. The qualitative analysis is as follows Figure 6 As shown in the attached Figure 6 In the figure, the first row shows a difficult sample in the SK-LARGE dataset where the background and object colors are similar. Existing CNN methods are unable to detect the deer's legs. The SkeSwin-L method can roughly detect the leg skeleton, but its multi-scale extraction capabilities limit the detection, resulting in blurry skeletons. DCFormer-L, on the other hand, can effectively distinguish foreground from background and accurately extract the skeleton. The second row shows one of the few difficult examples from WH-SYMMAX. This example is also difficult to distinguish because the horse's legs blend in with the background. CNN-based methods tend to overlook the horse's legs during detection, while DCFormer can effectively distinguish the object from the background. The third row shows an image of a dog from SYM-PASCAL. The object has large body parts, and neither classic CNN methods nor SkeSwin can effectively detect the skeleton. DCFormer can extract the skeleton more accurately. The last row shows a complex example from the SYMMAX300 dataset. This image shows a tiger walking in the jungle. The ground-truth skeleton label is the median axis of the tiger's body lines, which is difficult to detect. However, DCFormer can effectively capture low-level features such as stripes, thus completing stripe median axis detection.

[0094] The skeleton detection method described in this Example 1 utilizes a dual-branch convergence network model to extract the skeleton of the image to be detected; experimental verification shows that DCFormer has a good detection effect on large-scale samples and samples with large object scale changes, thereby improving the skeleton detection performance; specifically, quantitative experiments verify that DCFormer surpasses the existing optimal CNN method on all commonly used data sets and reaches the most advanced level (State Of The Art, SOTA); from a qualitative point of view, for complex natural images with similar background and foreground, DCFormer uses its dual-branch structure to supplement the semantic feature space, which to a certain extent solves this problem well; at the same time, DCFormer inherits the Transformer's ability to capture long-distance features, and better solves the problem of inaccurate skeleton prediction caused by large object scale.

[0095] Example 2

[0096] As attached Figure 7 As shown, this embodiment 2 provides a skeleton detection system, including an input module and a detection output module.

[0097] An input module is used to obtain an image to be detected I; a detection output module is used to input the image to be detected I into a dual-branch convergence network model and output a skeleton detection result of the image to be detected I; wherein the dual-branch convergence network model includes a backbone network, a main branch network, an auxiliary branch network and an output network; the backbone network is used to extract features from the image to be detected I and obtain feature maps X at several levels of different resolutions i ; The main branch network is used to analyze several levels of feature maps X with different resolutions i Perform aggregation to obtain the aggregated feature map F X ; and the aggregate feature map F X Perform pixel regression processing to obtain the main branch prediction result Y X ; The auxiliary branch network is used to analyze several levels of feature maps X with different resolutions i Perform cascade aggregation processing to obtain the aggregated feature map F X '; and the aggregate feature map F X 'Perform pixel regression processing to obtain the auxiliary branch prediction result Y X '; The output network is used to predict the result Y of the main branch X and the auxiliary branch prediction result Y X ' Perform feature fusion to obtain the skeleton detection result of the image to be detected I.

[0098] Example 3

[0099] As attached Figure 8As shown, this embodiment 3 provides a skeleton detection device, including: a memory for storing a computer program; and a processor for implementing the steps of the skeleton detection method when executing the computer program.

[0100] Obtain an image I to be detected; input the image I to be detected into a dual-branch convergence network model, and output a skeleton detection result of the image I to be detected; wherein the dual-branch convergence network model includes a backbone network, a main branch network, an auxiliary branch network and an output network; the backbone network is used to extract features from the image I to obtain feature maps X at several levels of different resolutions i ; The main branch network is used to analyze several levels of feature maps X with different resolutions i Perform aggregation to obtain the aggregated feature map F X ; and the aggregate feature map F X Perform pixel regression processing to obtain the main branch prediction result Y X ; The auxiliary branch network is used to analyze several levels of feature maps X with different resolutions i Perform cascade aggregation processing to obtain the aggregated feature map F X '; and the aggregate feature map F X 'Perform pixel regression processing to obtain the auxiliary branch prediction result Y X '; The output network is used to predict the result Y of the main branch X and the auxiliary branch prediction result Y X ' Perform feature fusion to obtain the skeleton detection result of the image to be detected I.

[0101] Alternatively, when the processor executes the computer program, the functions of each module in the above skeleton detection system are realized, for example:

[0102] An input module is used to obtain an image to be detected I; a detection output module is used to input the image to be detected I into a dual-branch convergence network model and output a skeleton detection result of the image to be detected I; wherein the dual-branch convergence network model includes a backbone network, a main branch network, an auxiliary branch network and an output network; the backbone network is used to extract features from the image to be detected I and obtain feature maps X at several levels of different resolutions i ; The main branch network is used to analyze several levels of feature maps X with different resolutions i Perform aggregation to obtain the aggregated feature map F X ; and the aggregate feature map F X Perform pixel regression processing to obtain the main branch prediction result Y X ; The auxiliary branch network is used to analyze several levels of feature maps X with different resolutions i Perform cascade aggregation processing to obtain the aggregated feature map F X'; and the aggregate feature map F X 'Perform pixel regression processing to obtain the auxiliary branch prediction result Y X '; The output network is used to predict the result Y of the main branch X and the auxiliary branch prediction result Y X ' Perform feature fusion to obtain the skeleton detection result of the image to be detected I.

[0103] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing preset functions, and the instruction segments are used to describe the execution process of the computer program in the skeleton detection device.

[0104] The skeleton detection device can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The skeleton detection device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the above are examples of skeleton detection devices and do not constitute a limitation of the skeleton detection device. The skeleton detection device may include more components than the above, or a combination of certain components, or different components. For example, the skeleton detection device may also include input and output devices, network access devices, buses, etc.

[0105] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the skeleton detection device, and utilizes various interfaces and lines to connect the various parts of the entire skeleton detection device.

[0106] The memory may be used to store the computer program and / or module, and the processor implements various functions of the skeleton detection device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0107] The memory may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0108] Example 4

[0109] This embodiment 4 further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the skeleton detection method are implemented.

[0110] If the modules / units integrated in the skeleton detection system are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0111] Based on this understanding, the present invention can implement all or part of the processes in the above-mentioned skeleton detection method by means of a computer program that instructs related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of the above-mentioned skeleton detection method. The computer program includes computer program code, which can be in source code form, object code form, executable file, or preset intermediate form.

[0112] The computer-readable storage medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0113] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0114] In the present invention, the main branch network is used to perform multi-scale extraction and decoding of skeleton features, and the cascade aggregation network of the auxiliary branch network is used to model the relationship between feature maps of different resolutions; the dual-branch structure is used to enrich the information of the overall semantic space of the network and improve the multi-scale detection capability; specifically, based on SwinTransformer as the backbone network, the powerful multi-scale feature aggregation capability of Sem-FPN is used to decode features, and a parallel self-attention convergence branch is proposed to find the correlation between Transformer features of different scales as a supplement to the semantic space, thereby improving the multi-scale extraction capability of the network; in addition, a shift convolution operation is proposed and applied to the void convolution pooling pyramid module, and the multi-scale feature extraction capability is enhanced through the shift operation between feature maps; experimental verification shows that the method described in the present invention can effectively improve the detection accuracy of large-scale samples, and has reached the most advanced level on four natural scene data sets.

[0115] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention.

Claims

1. A skeleton detection method, characterized in that: include: Obtain an image to be detected I; Inputting the image to be detected I into a dual-branch convergence network model, and outputting a skeleton detection result of the image to be detected I; wherein the dual-branch convergence network model includes a backbone network, a main branch network, an auxiliary branch network and an output network; The backbone network is used to extract features from the image to be detected I and obtain feature maps X at several levels of different resolutions. i ; The main branch network is used to analyze several levels of feature maps X with different resolutions. i Perform aggregation to obtain the aggregated feature map F X ; and the aggregate feature map F X Perform pixel regression processing to obtain the main branch prediction result Y X ; The auxiliary branch network is used to analyze several levels of feature maps X with different resolutions. i Perform cascade aggregation processing to obtain the aggregated feature map F X '; and the aggregate feature map F X 'Perform pixel regression processing to obtain the auxiliary branch prediction result Y X '; The output network is used to predict the result Y of the main branch X and the auxiliary branch prediction result Y X 'Perform feature fusion to obtain the skeleton detection result of the image to be detected I; The main branch network includes a Sem-FPN module, a ShiftASPP module and a main branch classifier module; The Sem-FPN module is used to generate several levels of feature maps X with different resolutions. i Perform channel-aligned convolution processing and perform feature map size recovery operations to obtain several channel-aligned feature maps; sum up several channel-aligned feature maps to obtain an aggregated result; The ShiftASPP module is used to perform shift convolution processing on the aggregation result and then perform hole convolution calculation to obtain the aggregation feature map F X ; The main branch classifier module is used to classify the aggregate feature map F X Perform convolution processing to obtain the main branch prediction result Y X ; The auxiliary branch network includes a first-level convergence layer, a feature fusion module, a second-level convergence layer and an auxiliary branch classifier module; The first-level convergence layer is used to input the feature maps of three adjacent levels with different resolutions in parallel and perform window self-attention calculation to obtain several sets of self-attention results A i ; The feature fusion module is used to combine several sets of self-attention results A i Perform an additive fusion operation to obtain a fusion result A with several levels of different scale features; The second-level pooling layer is used to perform self-attention calculation on the fusion result A with several levels of different scale features to obtain the pooling result; the feature maps X with several levels of different resolutions are i The residual of is summed with the aggregation result and mapped in the feature space to obtain the aggregate feature map F X '; The auxiliary branch classifier is used to classify the aggregated feature map F X 'Convolution processing is performed to obtain the auxiliary branch prediction result Y X ′.

2. A skeleton detection method according to claim 1, characterized in that: The backbone network is a Swin-Transformer network structure.

3. A skeleton detection method according to claim 1, characterized in that: The ShiftASPP module includes four parallel branch modules; each branch module includes a shift convolution module and a dilated convolution pooling pyramid module; The shift convolution module is configured to perform a first convolution operation on the aggregation result to obtain a first convolution feature map; perform a shift operation on the first convolution feature map to obtain a shifted feature map; perform a second convolution operation on the shifted feature map to obtain a second convolution feature map; and perform an inverse shift operation on the second convolution feature map to obtain a shifted convolution feature map; The dilated convolution pooling pyramid module is used to perform dilated convolution calculation on the shifted convolution feature map to obtain an aggregated feature map F X .

4. A skeleton detection method according to claim 1, characterized in that: The loss function of the dual-branch convergence network model is: in, is the total loss of the network; Loss of main branch; is the auxiliary branch loss; α is a hyperparameter used to balance the main branch and the auxiliary branch; W i is the pixel weight; X i are feature maps of different resolutions; is the proxy label after the skeleton line neighbor gradient representation processing; i is each pixel.

5. A skeleton detection method according to claim 1, characterized in that: The image to be detected I is a natural image that has undergone data enhancement.

6. A skeleton detection system, characterized in that: include: An input module is used to obtain an image I to be detected; A detection output module is used to input the image to be detected I into a dual-branch convergence network model and output a skeleton detection result of the image to be detected I; wherein the dual-branch convergence network model includes a backbone network, a main branch network, an auxiliary branch network and an output network; The backbone network is used to extract features from the image to be detected I and obtain feature maps X at several levels of different resolutions. i ; The main branch network is used to analyze several levels of feature maps X with different resolutions. i Perform aggregation to obtain the aggregated feature map F X ; and the aggregate feature map F X Perform pixel regression processing to obtain the main branch prediction result Y X ; The auxiliary branch network is used to analyze several levels of feature maps X with different resolutions. i Perform cascade aggregation processing to obtain the aggregated feature map F X '; and the aggregate feature map F X 'Perform pixel regression processing to obtain the auxiliary branch prediction result Y X '; The output network is used to predict the result Y of the main branch X and the auxiliary branch prediction result Y X 'Perform feature fusion to obtain the skeleton detection result of the image to be detected I; The main branch network includes a Sem-FPN module, a ShiftASPP module and a main branch classifier module; The Sem-FPN module is used to generate several levels of feature maps X with different resolutions. i Perform channel-aligned convolution processing and perform feature map size recovery operations to obtain several channel-aligned feature maps; sum up several channel-aligned feature maps to obtain an aggregated result; The ShiftASPP module is used to perform shift convolution processing on the aggregation result and then perform hole convolution calculation to obtain the aggregation feature map F X ; The main branch classifier module is used to classify the aggregate feature map F X Perform convolution processing to obtain the main branch prediction result Y X ; The auxiliary branch network includes a first-level convergence layer, a feature fusion module, a second-level convergence layer and an auxiliary branch classifier module; The first-level convergence layer is used to input the feature maps of three adjacent levels with different resolutions in parallel and perform window self-attention calculation to obtain several sets of self-attention results A i ; The feature fusion module is used to combine several sets of self-attention results A i Perform an additive fusion operation to obtain a fusion result A with several levels of different scale features; The second-level pooling layer is used to perform self-attention calculation on the fusion result A with several levels of different scale features to obtain the pooling result; the feature maps X with several levels of different resolutions are i The residual of is summed with the aggregation result and mapped in the feature space to obtain the aggregate feature map F X '; The auxiliary branch classifier is used to classify the aggregated feature map F X 'Convolution processing is performed to obtain the auxiliary branch prediction result Y X ′.

7. A skeleton detection device, characterized in that: include: a processor suitable for executing a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the skeleton detection method according to any one of claims 1 to 5 is executed.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the skeleton detection method according to any one of claims 1 to 5 is implemented.

Images