A method and system for keypoint prediction based on 2d-idct

Abstract

The application belongs to the technical field of deep learning, and particularly relates to a key point prediction method and system based on 2D-IDCT, which solves the problem that a large amount of information is lost in continuous multiple upsampling. The method comprises the following steps: obtaining a to-be-detected image as an input image; pre-processing the input image to obtain a pre-processed input image; inputting the pre-processed input image into a trained key point prediction network based on 2D-IDCT to output a key point prediction heat map. The application reconstructs features through IDCT, and the size of the reconstructed features is not limited, so that a larger scale feature map can be obtained, which helps to improve the detection accuracy; and the DCT transformation of GroundTruth is learned implicitly, and focuses on more important low-frequency coefficients, which is more conducive to extracting global features.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a method and system for key point prediction based on 2D-IDCT. Background Art

[0002] Human pose estimation is increasingly widely used. To achieve higher accuracy, heatmap regression-based human pose estimation networks often increase the feature maps output by the backbone to a certain size. Therefore, a method is needed to integrate the features output by the network backbone and restore them to a larger size, so as to improve accuracy while extracting features more efficiently.

[0003] The DCT, or discrete cosine transform, converts spatial signals to the frequency domain and is often used in image processing. Because most natural signal energy is concentrated in the low-frequency portion after the discrete cosine transform, it exhibits a strong "energy concentration" characteristic. The IDCT, its inverse transform, can reconstruct images from the DCT coefficients.

[0004] Pose estimation models based on heatmap regression often require multiple upsampling cycles to restore the backbone output feature map to a larger size to improve accuracy. For example, SimpleBaseline uses three transposed convolutions to increase both the width and height of the backbone output feature map by a factor of eight, achieving high accuracy. AlphaPose also achieves good accuracy by increasing the feature map size through three PixelShuffle upsampling cycles.

[0005] However, the feature map size obtained by the upsampling method can only be 2^n times the width and height of the backbone output feature map, and multiple upsampling is required. The more upsampling times, the greater the computational complexity. The heatmap-based regression method requires a larger feature map size to achieve high precision; continuous upsampling will lose information in the process of increasing the feature size; the network uses a full convolution method, which has limited ability to extract global features and is not conducive to capturing long-distance dependencies. Summary of the Invention

[0006] In order to solve the above-mentioned problem in the prior art, namely, the problem that a large amount of information will be lost due to multiple consecutive upsampling, the present invention provides a key point prediction method and system based on 2D-IDCT, the method comprising:

[0007] Step S100, obtaining an image to be tested as an input image;

[0008] Step S200, preprocessing the input image to obtain a preprocessed input image;

[0009] Step S300, inputting the preprocessed input image into a trained 2D-IDCT-based key point prediction network, and outputting a key point prediction heat map;

[0010] The key point prediction network based on 2D-IDCT consists of a feature extraction module, a fully connected module, an IDCT module and a heatmap module; the fully connected module includes a feature expansion layer, a first fully connected layer Line1, a Relu activation function layer and a second fully connected layer Line2.

[0011] The step S300 specifically includes:

[0012] Step S310, based on the pre-processed input image, obtain a first feature map F1 through a feature extraction module;

[0013] Step S320: Based on the first feature map F1, the first feature map F1 is dimensionally reorganized by the feature expansion layer of the fully connected module to obtain a second feature map F2;

[0014] The second feature map F2 is used as the input of the fully connected layer Line1, and the output of the first fully connected layer Line1 is subjected to the Relu activation function to obtain the third feature map F3;

[0015] The third feature map F3 is used as the input of the second fully connected layer Line2 to obtain the fourth feature map F4;

[0016] Step S330: Based on the fourth feature map F4, an IDCT module is used to perform an IDCT transform on the second global feature length n of the fourth feature map F4 to obtain a fifth feature map F5;

[0017] Perform convolution calculation on the fifth feature map F5 to obtain a sixth feature map F6;

[0018] Step S340 , using a heatmap module to convolve the feature map F6 to a number of image channels equal to the number of key points of the human body posture, thereby obtaining a key point prediction heatmap;

[0019] The fully connected layer Line1 is used to integrate the feature map F2 and learn global features;

[0020] The fully connected layer Line2 is used to output low-frequency coefficients of a specific dimension.

[0021] The step S310 specifically includes:

[0022] Based on the preprocessed input image, feature extraction is performed using the convolutional neural network in the feature extraction module as the backbone. The feature map output by the backbone is denoted as F1. The size of the first feature map F1 is denoted as (bs, c, w, h), where bs is the input batch size, c is the number of channels of F1, and w and h are the width and height of F1, respectively.

[0023] The convolutional neural network specifically includes:

[0024] A network with multi-scale feature extraction capability. Step S320 specifically includes:

[0025] Step S321, reorganize the feature map F1 dimension to obtain the feature map F2, the size of F2 is (bs, c1, l), where c1 is the number of channels of the feature map F2, the feature map F3, the feature map F4, the feature map F5 and the feature map F6, c1 = c / k is an integer, k is the reorganization coefficient, l is the feature vector length of the feature map F2, and l = w × h × k is an integer;

[0026] Step S322: In the fully connected layer Line1, the second feature map F2 is activated by the ReLU function to obtain the third feature map F3, which is expressed as:

[0027] F3 = Relu(Line1(F2));

[0028] Among them, the dimension of F3 is (bs, c1, m), m is the first global feature length;

[0029] Step S323: In the fully connected layer Line2, the third feature map F3 is processed to obtain the fourth feature map F4, which is expressed as:

[0030] F4=Line2(F3);

[0031] Among them, the dimension of F4 is (bs, c1, n), and n is the second global feature length.

[0032] The step S330 specifically includes:

[0033] Step S331: Perform an IDCT transform on the second global feature length n of the fourth feature map F4 to obtain a feature map F5. The dimension of F5 is (bs, c1, w5, h5), where w5 and h5 are the width and height of the fifth feature map F5, respectively, and w5>w and h5>h.

[0034] Step S332: Perform a 3×3 convolution on the fifth feature map F5 to obtain F6, which is expressed as:

[0035] F6=conv3×3(F5).

[0036] The training method of the 2D-IDCT-based key point prediction network specifically includes:

[0037] Step A100: using a training set image as a training input image; the training set image includes key point annotation information;

[0038] Step A200, generating a GroundTruth heat map based on key point annotation information in the training input image;

[0039] Step A300, preprocessing the training set images to obtain preprocessed training input images;

[0040] Step A400: Input the preprocessed training input image into a key point prediction network based on 2D-IDCT, and output a key point prediction heat map of the training set;

[0041] Step A500, calculating the loss function based on the key point prediction heat map and the GroundTruth heat map of the training set;

[0042] Step A600 , repeating the method of steps A300 to A500 until the loss function value is lower than a preset threshold, thereby obtaining a key point prediction model based on 2D-IDCT.

[0043] The loss function value is calculated using L2 loss;

[0044] Another aspect of the present invention provides a system for key point prediction based on 2D-IDCT, the system comprising:

[0045] The image acquisition module to be tested is configured to collect the image to be tested through the image acquisition device;

[0046] Preprocessing input image generation module: configured to preprocess the image to be tested to obtain a preprocessing input image;

[0047] Keypoint prediction heatmap generation module: It is configured to input the preprocessed input image into the trained 2D-IDCT-based keypoint prediction network and output the keypoint prediction heatmap.

[0048] Beneficial effects of the present invention:

[0049] (1) The present invention reconstructs features through IDCT, and the size of the reconstructed features is not limited, so a larger-scale feature map can be obtained, which helps to improve detection accuracy;

[0050] (2) The present invention focuses on the more important low-frequency coefficients by implicitly learning the DCT transform of GroundTruth, which is more conducive to extracting global features;

[0051] (3) The application can restore the feature map to the specified size through one IDCT transformation, without multiple upsampling, reducing the loss of information in the upsampling process.

[0052] (4) The key point prediction method based on 2D-IDCT can be applied to various key point detection tasks, and can comprehensively extract global features of various types of target feature points and accurately predict the feature points. BRIEF DESCRIPTION OF DRAWINGS

[0053] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings:

[0054] Figure 1 is a flowchart of a key point prediction method based on 2D-IDCT. DETAILED DESCRIPTION

[0055] The application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0056] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the accompanying drawings and embodiments.

[0057] The key point prediction method based on 2D-IDCT of the application comprises:

[0058] Step S100, obtaining a to-be-tested image as an input image;

[0059] Step S200, pre-processing the input image to obtain a pre-processed input image;

[0060] Step S300, inputting the pre-processed input image into a trained key point prediction network based on 2D-IDCT, and outputting a key point prediction heat map;

[0061] The key point prediction network based on 2D-IDCT is composed of a feature extraction module, a full connection module, an IDCT module and a heatmap module; the full connection module includes a feature unfolding layer, a first full connection layer Line1, a Relu activation function layer and a second full connection layer Line2.

[0062] In order to more clearly illustrate the key point prediction method based on 2D-IDCT of the application, the following will be combined with Figure 1Each step in the embodiment of the present invention is described in detail.

[0063] In the first embodiment of the present invention, a key point prediction method based on 2D-IDCT is proposed and applied to the case of human posture estimation, such as Figure 1 As shown, each step is described in detail as follows:

[0064] Step S100, obtaining an image to be tested as an input image;

[0065] Step S200, preprocessing the input image to obtain a preprocessed input image;

[0066] In this embodiment, the preprocessing specifically includes:

[0067] Performing data enhancement and normalization processing on the input image to obtain a preprocessed input image.

[0068] Step S300, inputting the preprocessed input image into a trained 2D-IDCT-based key point prediction network, and outputting a key point prediction heat map;

[0069] In this embodiment, the training method of the key point prediction network based on 2D-IDCT specifically includes:

[0070] Step A100: using a training set image as a training input image; the training set image includes key point annotation information;

[0071] Step A200, generating a GroundTruth heat map based on key point annotation information in the training input image;

[0072] Step A300, preprocessing the training set images to obtain preprocessed training input images;

[0073] Step A400: Input the preprocessed training input image into a key point prediction network based on 2D-IDCT, and output a key point prediction heat map of the training set;

[0074] Step A500, calculating the loss function based on the key point prediction heat map and the GroundTruth heat map of the training set;

[0075] In this embodiment, the loss function value is calculated using L2 loss;

[0076] Step A600 , repeating the method of steps A300 to A500 until the loss function value is lower than a preset threshold, thereby obtaining a key point prediction model based on 2D-IDCT.

[0077] In this embodiment, step S300 specifically includes:

[0078] Step S310, based on the pre-processed input image, obtain a first feature map F1 through a feature extraction module;

[0079] In this embodiment, step S310 specifically includes:

[0080] Based on the preprocessed input image, feature extraction is performed using the convolutional neural network in the feature extraction module as the backbone. The feature map output by the backbone is denoted as F1. The size of the first feature map F1 is denoted as (bs, c, w, h), where bs is the input batch size, c is the number of channels of F1, and w and h are the width and height of F1, respectively.

[0081] In this embodiment, the convolutional neural network specifically includes:

[0082] Feature extraction network ResNet50.

[0083] Step S320: Based on the first feature map F1, the first feature map F1 is dimensionally reorganized by the feature expansion layer of the fully connected module to obtain a second feature map F2;

[0084] The second feature map F2 is used as the input of the fully connected layer Line1, and the output of the first fully connected layer Line1 is subjected to the Relu activation function to obtain the third feature map F3;

[0085] The third feature map F3 is used as the input of the second fully connected layer Line2 to obtain the fourth feature map F4;

[0086] In this embodiment, step S320 specifically includes:

[0087] Step S321, reorganize the feature map F1 dimension to obtain the feature map F2, the size of F2 is (bs, c1, l), where c1 is the number of channels of the feature map F2, the feature map F3, the feature map F4, the feature map F5 and the feature map F6, c1 = c / k is an integer, k is the reorganization coefficient, l is the feature vector length of the feature map F2, and l = w × h × k is an integer;

[0088] Step S322: In the fully connected layer Line1, the second feature map F2 is activated by the ReLU function to obtain the third feature map F3, which is expressed as:

[0089] F3 = Relu(Line1(F2));

[0090] Among them, the dimension of F3 is (bs, c1, m), m is the first global feature length;

[0091] Step S323: In the fully connected layer Line2, the third feature map F3 is processed to obtain the fourth feature map F4, which is expressed as:

[0092] F4=Line2(F3);

[0093] The dimension of F4 is (bs, c1, n), where n is the second global feature length, which in this embodiment is represented by the length of the low-frequency coefficient in the feature map.

[0094] Step S330: Based on the fourth feature map F4, an IDCT module is used to perform an IDCT transform on the second global feature length n of the fourth feature map F4 to obtain a fifth feature map F5;

[0095] Perform convolution calculation on the fifth feature map F5 to obtain a sixth feature map F6;

[0096] In this embodiment, step S330 specifically includes:

[0097] Step S331: Perform an IDCT transform on the second global feature length n of the fourth feature map F4 to obtain a feature map F5. The dimension of F5 is (bs, c1, w5, h5), where w5 and h5 are the width and height of the fifth feature map F5, respectively, and w5>w and h5>h.

[0098] Step S332: Perform a 3×3 convolution on the fifth feature map F5 to obtain F6, which is expressed as:

[0099] F6=conv3×3(F5).

[0100] Step S340 , using a heatmap module to convolve the feature map F6 to a number of image channels equal to the number of key points of the human body posture, thereby obtaining a key point prediction heatmap;

[0101] In this embodiment, the key points specifically include:

[0102] Human body joints.

[0103] In this embodiment, the training set images specifically include:

[0104] Images of human bodies and annotations of human joint coordinates in different scenes.

[0105] Based on this embodiment, as a verification, experiments are conducted on the COCO2017 dataset. The evaluation is performed on the COCOval2017 test set with an input resolution of 256x256. The evaluation criteria are the average precision AP and average recall AR based on OKS (Object Keypoints Similarity) to measure the accuracy of the method. The experimental results are shown in Table 1. All methods are completed under the simplebaseline training framework.

[0106] As shown in Table 1, compared with Simlebaseline_res50, the AP and AR are significantly improved after using the method of the present invention.

[0107] Table 1: Comparison of human posture key point prediction network results based on 2D-IDCT

[0108] method heatmap size AP AR Simplebaseline_res50 64×64 72.5 75.5 Ours_res50 80×80 72.9 75.9 Ours_res50 96×96 73.3 76.1

[0109] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0110] The second embodiment of the present invention discloses a method for predicting key points based on 2D-IDCT and applied to facial feature points:

[0111] During the training process of the 2D-IDCT-based key point prediction network, images of various facial expressions marked with facial key points are used as training input images; the distribution rules of each key point in the face and the positions of key points under various expressions are learned through the 2D-IDCT-based key point prediction network; the trained 2D-IDCT-based key point prediction model can accurately predict the positions of facial key points.

[0112] A third embodiment of the present invention discloses a system for key point prediction based on 2D-IDCT, comprising:

[0113] The image acquisition module to be tested is configured to collect the image to be tested through the image acquisition device;

[0114] Preprocessing input image generation module: configured to preprocess the image to be tested to obtain a preprocessing input image;

[0115] Keypoint prediction heatmap generation module: It is configured to input the preprocessed input image into the trained 2D-IDCT-based keypoint prediction network and output the keypoint prediction heatmap.

[0116] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0117] It should be noted that the above embodiment provides a system for key point prediction based on 2D-IDCT, and only illustrates the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiment can be combined into one module or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are merely for distinguishing the modules or steps and are not to be considered as improper limitations of the present invention.

[0118] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0119] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0120] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for key point prediction based on 2D-IDCT, characterized in that: The method comprises: Step S100, obtaining an image to be tested as an input image; Step S200, preprocessing the input image to obtain a preprocessed input image; Step S300, inputting the preprocessed input image into a trained 2D-IDCT-based key point prediction network, and outputting a key point prediction heat map; The key point prediction network based on 2D-IDCT is composed of a feature extraction module, a fully connected module, an IDCT module and a heatmap module; the fully connected module includes a feature expansion layer, a first fully connected layer Line1, a Relu activation function layer and a second fully connected layer Line2; The step S300 specifically includes: Step S310, based on the pre-processed input image, obtain a first feature map F1 through a feature extraction module; Step S320: Based on the first feature map F1, the first feature map F1 is dimensionally reorganized by the feature expansion layer of the fully connected module to obtain a second feature map F2; The second feature map F2 is used as the input of the fully connected layer Line1, and the output of the first fully connected layer Line1 is subjected to the Relu activation function to obtain the third feature map F3; The third feature map F3 is used as the input of the second fully connected layer Line2 to obtain the fourth feature map F4; Step S330: Based on the fourth feature map F4, an IDCT module is used to perform an IDCT transform on the second global feature length n of the fourth feature map F4 to obtain a fifth feature map F5; Perform convolution calculation on the fifth feature map F5 to obtain a sixth feature map F6; In step S340 , the feature map F6 is convolved and converted into an image with a number of channels equal to the number of key points of the human body posture through a heatmap module to obtain a key point prediction heatmap.

2. The method for key point prediction based on 2D-IDCT according to claim 1, characterized in that: The step S310 specifically includes: Based on the preprocessed input image, feature extraction is performed using the convolutional neural network in the feature extraction module as the backbone. The feature map output by the backbone is denoted as F1. The size of the first feature map F1 is denoted as (bs, c, w, h), where bs is the input batch size, c is the number of channels of F1, and w and h are the width and height of F1, respectively.

3. The method for key point prediction based on 2D-IDCT according to claim 2, characterized in that: The convolutional neural network includes: a network with multi-scale feature extraction capability.

4. The method for key point prediction based on 2D-IDCT according to claim 2, characterized in that: Step S320 specifically includes: Step S321, reorganize the feature map F1 dimension to obtain the feature map F2, the size of F2 is (bs, c1, l), where c1 is the number of channels of the feature map F2, the feature map F3, the feature map F4, the feature map F5 and the feature map F6, c1 = c / k is an integer, k is the reorganization coefficient, l is the feature vector length of the feature map F2, and l = w × h × k is an integer; Step S322: In the fully connected layer Line1, the second feature map F2 is activated by the ReLU function to obtain the third feature map F3, which is expressed as: F3 = Relu(Line1(F2)); Among them, the dimension of F3 is (bs, c1, m), where m is the first global feature length; Step S323: In the fully connected layer Line2, the third feature map F3 is processed to obtain the fourth feature map F4, which is expressed as: F4=Line2(F3); Among them, the dimension of F4 is (bs, c1, n), and n is the second global feature length.

5. The method for key point prediction based on 2D-IDCT according to claim 4, characterized in that: Step S330 specifically includes: Step S331: Perform an IDCT transform on the second global feature length n of the fourth feature map F4 to obtain a feature map F5. The dimension of F5 is (bs, c1, w5, h5), where w5 and h5 are the width and height of the fifth feature map F5, respectively, and w5>w, h5>h; Step S332: Perform a 3×3 convolution on the fifth feature map F5 to obtain F6, which is expressed as: F6=conv3×3(F5).

6. The method for key point prediction based on 2D-IDCT according to claim 1, characterized in that: The training method of the 2D-IDCT-based key point prediction network specifically includes: Step A100: using a training set image as a training input image; the training set image includes key point annotation information; Step A200, generating a GroundTruth heat map based on key point annotation information in the training input image; Step A300, preprocessing the training set images to obtain preprocessed training input images; Step A400: Input the preprocessed training input image into a key point prediction network based on 2D-IDCT, and output a key point prediction heat map of the training set; Step A500, calculating the loss function value based on the key point prediction heat map and the GroundTruth heat map of the training set; Step A600 , repeating the method of steps A300 to A500 until the loss function value is lower than a preset threshold, thereby obtaining a key point prediction model based on 2D-IDCT.

7. The method for key point prediction based on 2D-IDCT according to claim 6, characterized in that: The loss function value is calculated using L2 loss.

8. A system for key point prediction based on 2D-IDCT, applied to the method according to any one of claims 1 to 7, characterized in that: The system comprises: The image acquisition module to be tested is configured to collect the image to be tested through the image acquisition device; Preprocessing input image generation module: configured to preprocess the image to be tested to obtain a preprocessing input image; Keypoint prediction heatmap generation module: It is configured to input the preprocessed input image into the trained 2D-IDCT-based keypoint prediction network and output the keypoint prediction heatmap.

Images