Biological key point detection method, device, equipment and storage medium
By acquiring feature images, window marker maps, and keypoint marker maps of images, and combining the Visual-Window-Keypoint Transformer Block and multilayer perceptron, the problem of low efficiency in biological keypoint detection is solved, achieving more efficient and accurate detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-13
- Publication Date
- 2026-07-10
AI Technical Summary
Existing methods for detecting key points in organisms have low detection efficiency and poor detection results, and traditional methods ignore the constraints between key points and multi-scale information.
A biological keypoint detection method is adopted. By acquiring the feature image, window marker image and keypoint marker image of the image to be detected, visual transformation processing, patch merging and keypoint supplementation processing are performed. The Visual-Window-Keypoint Transformer Block structure and multilayer perceptron are used to perform multi-head self-attention operation to improve detection efficiency and effect.
It improves the detection efficiency of key points in organisms and enhances the detection effect. By learning the constraint relationships between key points and multi-scale information, it achieves better detection results.
Smart Images
Figure CN116912938B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and storage medium for detecting key points in organisms. Background Technology
[0002] The goal of biometric keypoint detection is to establish a feature representation of a biometric pose (i.e., the coordinate positions of biometric keypoints such as the head, left hand, and right hand in the image or video) from an image or video. As one of the fundamental algorithms in computer vision, it can be applied to many related fields, such as behavior recognition, human tracking, and gait recognition.
[0003] However, due to the considerable flexibility of living organisms, which can exhibit various postures and shapes, and the fact that the visibility of their key points is greatly affected by clothing, posture, and viewing angle, as well as environmental factors such as occlusion and lighting, constructing an efficient and accurate framework for detecting key points in living organisms remains a highly challenging topic in the field of computer vision.
[0004] Traditional methods using Convolutional Neural Networks (CNNs) rely on their powerful feature extraction capabilities to learn keypoint locations, focusing solely on image features. However, this approach ignores the constraints between keypoints, leading to poor detection performance. Recent advancements have employed hybrid methods combining CNN feature extraction with Transformer neural networks' multi-head self-attention. However, CNN feature extraction operates on fixed regions, resulting in a lack of multi-scale information within the framework, leading to suboptimal detection results. Furthermore, the Transformer neural network performs multi-head self-attention operations across all features (tokens), further reducing detection efficiency. Summary of the Invention
[0005] The purpose of this invention is to provide a method, apparatus, device, and storage medium for detecting key points in organisms, in order to solve the problems of low detection efficiency and poor detection effect of existing methods for detecting key points in organisms.
[0006] This invention provides a method for detecting key points in a biological organism, comprising:
[0007] Obtain the feature image, window marker map, and key point marker map of the image to be detected;
[0008] A heatmap is obtained by performing a first process and a second process based on the feature image, the window marker map, and the key point marker map; the first process includes visual transformation processing of the window marker map and the key point feature map; the second process includes patch merging processing and key point supplementation processing.
[0009] Based on the heat map, key points of the organism are detected, and the detection results are obtained.
[0010] Optionally, obtaining the feature image of the image to be detected includes:
[0011] Acquire the image to be detected;
[0012] The feature image is obtained by extracting features from the image to be detected using a preset first convolutional neural network.
[0013] Optionally, the step of obtaining a heatmap by performing a first processing and a second processing based on the feature image, the window marker map, and the key point marker map includes:
[0014] Initialize the target value for processing, where the target value includes the number of iterations P;
[0015] The following processing is performed in each iteration except the last iteration:
[0016] A first processing is performed on the first feature image, the first window marker image, and the first key point marker image to obtain a first output result. The first output result includes: a second feature image corresponding to the first feature image, a second window marker image corresponding to the first window marker image, and a second key point marker image corresponding to the first key point marker image.
[0017] The first output result is processed in a second way to obtain a second output result, and the second output result is the input information for the next iteration process. The second output result includes a third feature image corresponding to the second feature image, a third window label image corresponding to the second window label image, and a third key point label image corresponding to the second key point label image.
[0018] The following processing is performed during the final iteration:
[0019] The heatmap is obtained by performing a first process on the second output result of the (P-1)th iteration.
[0020] In the case of the first iteration process, the first feature image is the feature image, the first window marker image is the window marker image, and the first key point marker image is the key point marker image.
[0021] Optionally, the target value may also include the number of times the first processing is performed in each iteration, Q.
[0022] The first processing based on the first feature image, the first window marker map, and the first key point marker map to obtain the first output result includes:
[0023] The following processing is performed during each iteration:
[0024] A first processing is performed based on the first target feature image, the first target window marker image, and the first target key point marker image to obtain a third output result, and the third output result is the input information for the next first processing. The third output result includes: a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0025] In the case of the first processing in each iteration, the first target feature image is the first feature image, the first target window marker image is the first window marker image, and the first target key point marker image is the first key point marker image.
[0026] In the case of the last first processing in each iteration, the second target feature image is the second feature image, the second target window marker image is the second window marker image, and the second target key point marker image is the second key point marker image.
[0027] Optionally, the first processing based on the first target feature image, the first target window marker map, and the first target key point marker map to obtain the third output result includes:
[0028] Each of the first target feature images is divided into at least one window; one window corresponds to one first target window marker image;
[0029] Based on the window and the first target window labeling map corresponding to the window, a third target window labeling map is obtained after learning image information from the first target window labeling map;
[0030] Based on the third target window marker map and the first target key point marker map, a third target key point marker map is obtained after learning key point information from the first target key point marker map;
[0031] Using a pre-defined multilayer perceptron, the first target feature image, the third target window marker image, and the third target key point marker image are mapped to obtain a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0032] Optionally, obtaining a third target window label map based on the first target window label map and the first target window label map corresponding to the window, after learning image information from the first target window label map, includes:
[0033] Perform a multi-head self-attention operation on the window to obtain the processed window;
[0034] A multi-head self-attention operation is performed on the first target window labeling map to obtain a processed first target window labeling map;
[0035] The image information of the processed window is learned using the processed first target window marker map to obtain the third target window marker map.
[0036] Optionally, obtaining the third target keypoint marker map after learning keypoint information from the first target keypoint marker map based on the third target window marker map and the first target keypoint marker map includes:
[0037] Perform a multi-head self-attention operation on the third target window marker map and the first target key point marker map to obtain the key point information of the third target window marker map;
[0038] The key point information of the third target window marker map is learned by using the first target key point marker map to obtain the third target key point marker map.
[0039] Optionally, the second processing of the first output result to obtain a second output result includes:
[0040] The second feature image is convolved using a pre-defined second convolutional neural network to obtain a third feature image corresponding to the second feature image; the size of the third feature image is smaller than that of the second feature image.
[0041] The second window labeling image is convolved using a pre-defined second convolutional neural network to obtain a third window labeling image corresponding to the second window labeling image; the number of the second window labeling images is greater than the number of the third window labeling images, and the number of channels in the second window labeling images is less than the number of channels in the third window labeling image;
[0042] The second key point marker image is processed using a preset multilayer perceptron to obtain a third key point marker image corresponding to the second key point marker image; the number of channels in the second key point marker image is less than the number of channels in the third key point marker image.
[0043] This invention also provides a biological key point detection device, comprising:
[0044] The acquisition module is used to acquire the feature image, window marker map, and key point marker map of the image to be detected;
[0045] The processing module is configured to perform a first processing and a second processing based on the feature image, the window marker map, and the key point marker map to obtain a heat map; the first processing includes visual transformation processing of the window marker map and the key point feature map; the second processing includes patch merging processing and key point supplementation processing;
[0046] The detection module is used to detect key points of the organism based on the heat map and obtain the detection results.
[0047] Optionally, the acquisition module includes:
[0048] The acquisition unit is used to acquire the image to be detected;
[0049] The first extraction unit is used to extract features from the image to be detected using a preset first convolutional neural network to obtain the feature image.
[0050] Optionally, the processing module includes:
[0051] An initialization unit is used to initialize the target value to be processed, wherein the target value includes the number of iterations P;
[0052] The first processing unit is used to perform the following processing in each iteration process, except for the last iteration process:
[0053] A first processing is performed on the first feature image, the first window marker image, and the first key point marker image to obtain a first output result. The first output result includes: a second feature image corresponding to the first feature image, a second window marker image corresponding to the first window marker image, and a second key point marker image corresponding to the first key point marker image.
[0054] The first output result is processed in a second way to obtain a second output result, and the second output result is the input information for the next iteration process. The second output result includes a third feature image corresponding to the second feature image, a third window label image corresponding to the second window label image, and a third key point label image corresponding to the second key point label image.
[0055] The following processing is performed during the final iteration:
[0056] The heatmap is obtained by performing a first process on the second output result of the (P-1)th iteration.
[0057] In the case of the first iteration process, the first feature image is the feature image, the first window marker image is the window marker image, and the first key point marker image is the key point marker image.
[0058] Optionally, the target value may also include the number of times the first processing is performed in each iteration, Q.
[0059] The first processing unit is specifically used for:
[0060] The following processing is performed during each iteration:
[0061] A first processing is performed based on the first target feature image, the first target window marker image, and the first target key point marker image to obtain a third output result, and the third output result is the input information for the next first processing. The third output result includes: a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0062] In the case of the first processing in each iteration, the first target feature image is the first feature image, the first target window marker image is the first window marker image, and the first target key point marker image is the first key point marker image.
[0063] In the case of the last first processing in each iteration, the second target feature image is the second feature image, the second target window marker image is the second window marker image, and the second target key point marker image is the second key point marker image.
[0064] Optionally, the first processing unit is specifically used for:
[0065] Each of the first target feature images is divided into at least one window; one window corresponds to one first target window marker image;
[0066] Based on the window and the first target window labeling map corresponding to the window, a third target window labeling map is obtained after learning image information from the first target window labeling map;
[0067] Based on the third target window marker map and the first target key point marker map, a third target key point marker map is obtained after learning key point information from the first target key point marker map;
[0068] Using a pre-defined multilayer perceptron, the first target feature image, the third target window marker image, and the third target key point marker image are mapped to obtain a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0069] Optionally, the first processing unit is specifically used for:
[0070] Perform a multi-head self-attention operation on the window to obtain the processed window;
[0071] A multi-head self-attention operation is performed on the first target window labeling map to obtain a processed first target window labeling map;
[0072] The image information of the processed window is learned using the processed first target window marker map to obtain the third target window marker map.
[0073] Optionally, the first processing unit is specifically used for:
[0074] Perform a multi-head self-attention operation on the third target window marker map and the first target key point marker map to obtain the key point information of the third target window marker map;
[0075] The key point information of the third target window marker map is learned by using the first target key point marker map to obtain the third target key point marker map.
[0076] Optionally, the first processing unit is specifically used for:
[0077] The second feature image is convolved using a pre-defined second convolutional neural network to obtain a third feature image corresponding to the second feature image; the size of the third feature image is smaller than that of the second feature image.
[0078] The second window labeling image is convolved using a pre-defined second convolutional neural network to obtain a third window labeling image corresponding to the second window labeling image; the number of the second window labeling images is greater than the number of the third window labeling images, and the number of channels in the second window labeling images is less than the number of channels in the third window labeling image;
[0079] The second key point marker image is processed using a preset multilayer perceptron to obtain a third key point marker image corresponding to the second key point marker image; the number of channels in the second key point marker image is less than the number of channels in the third key point marker image.
[0080] This invention also provides a biological key point detection device, comprising: a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the steps of the biological key point detection method as described above.
[0081] This invention also provides a readable storage medium storing a program or instructions thereon, which, when executed by a processor, implement the steps in the biological key point detection method as described above.
[0082] This invention also provides an image animation generation apparatus, comprising:
[0083] This invention also provides an image animation generation device, comprising: a transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; when the processor executes the program or instructions, it implements the steps of the image animation generation method as described above.
[0084] This invention also provides a readable storage medium storing a program or instructions thereon, which, when executed by a processor, implement the steps of the image animation generation method as described above.
[0085] The beneficial effects of the above-described technical solution of the present invention are as follows:
[0086] The method for detecting key points in organisms provided by this invention includes: acquiring a feature image, a window marker image, and a key point marker image of an image to be detected; performing a first processing and a second processing based on the feature image, the window marker image, and the key point marker image to obtain a heatmap; the first processing includes performing visual transformation processing on the window marker image and the key point feature image; the second processing includes patch merging processing and key point supplementation processing; and detecting key points in organisms based on the heatmap to obtain detection results can improve the detection efficiency and enhance the detection effect. Attached Figure Description
[0087] Figure 1 A flowchart of a biological key point detection method provided in an embodiment of the present invention;
[0088] Figure 2 This is a schematic diagram illustrating the Visual-Window-Keypoint Transformer Block process provided in an embodiment of the present invention;
[0089] Figure 3 This is an overall framework diagram of the biological key point detection method provided in the embodiments of the present invention;
[0090] Figure 4 A flowchart illustrating the biological key point detection method provided in this embodiment of the invention;
[0091] Figure 5 This is a schematic diagram of the structure of the biological key point detection device provided in an embodiment of the present invention;
[0092] Figure 6 This is a schematic diagram of the structure of the biological key point detection device provided in an embodiment of the present invention. Detailed Implementation
[0093] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.
[0094] To address the problems of low detection efficiency and poor detection results in existing methods for detecting key points in organisms, this invention provides a method, apparatus, device, and storage medium for detecting key points in organisms.
[0095] like Figure 1 As shown, this embodiment of the invention provides a method for detecting key points in a biological organism, including:
[0096] Step 101: Obtain the feature image, window marker map, and key point marker map of the image to be detected.
[0097] In this embodiment of the invention, the human body is used as the example organism.
[0098] In this step, the image to be detected is the input image H×W×3 (generally H=256, W=192 or H=384, W=288), where H represents the image height, W represents the image width, and 3 indicates that the image is an RGB 3D image.
[0099] This step involves feature extraction from the input image to obtain the image to be detected. A feature image, namely ( (Number of tokens), with C channels, and initialization... The system consists of one window token map and K keypoint token maps, with C channels, where M is an integer greater than or equal to 1, K is an integer greater than or equal to 1, and C is an integer greater than or equal to 1. Preferably, K is 17.
[0100] Step 102: Based on the feature image, the window marker map, and the key point marker map, perform a first processing and a second processing to obtain a heat map; the first processing includes visual transformation processing of the window marker map and the key point feature map; the second processing includes patch merging processing and key point supplementation processing.
[0101] In this step, the first processing is performed by designing a Visual-Window-Keypoint TransformerBlock structure, and the second processing is a patch merging and keypoint expansion operation. By performing the first and second processing on the feature image, window marker map and keypoint marker map, K keypoint tokens are regressed into K heatmaps.
[0102] Step 103: Detect key points of the organism based on the heat map and obtain the detection results.
[0103] In this step, K heatmaps are used to detect key points of organisms, and the detection results are obtained, which can improve the detection efficiency of key points of organisms and enhance the detection effect.
[0104] In an optional embodiment of the present invention, obtaining the feature image of the image to be detected includes:
[0105] Acquire the image to be detected;
[0106] The feature image is obtained by extracting features from the image to be detected using a preset first convolutional neural network.
[0107] Optionally, the preset first convolutional neural network is an overlapping convolutional neural network. That is, in this optional embodiment, for the input image H×W×3, an overlapping convolutional network (with a kernel size of 8×6 and a stride of (4, 3)) is used for feature extraction to obtain... A feature image, namely ( (Number of tokens), with a channel count of C.
[0108] Then, initialize The system consists of one window token map and K keypoint token maps, with a channel count of C.
[0109] In an optional embodiment of the present invention, the step of obtaining a heatmap by performing a first processing and a second processing based on the feature image, the window marker map, and the key point marker map includes:
[0110] The target value is initialized, which includes the number of iterations P, where P is an integer greater than or equal to 1. In this embodiment, the number of stages is initialized to 4, namely stage1, stage2, stage3 and stage4, that is, the number of iterations P is initialized to 4.
[0111] The following processing is performed in each iteration except the last iteration:
[0112] A first processing is performed on the first feature image, the first window marker image, and the first key point marker image to obtain a first output result. The first output result includes: a second feature image corresponding to the first feature image, a second window marker image corresponding to the first window marker image, and a second key point marker image corresponding to the first key point marker image.
[0113] The first output result is processed in a second way to obtain a second output result, and the second output result is the input information for the next iteration process. The second output result includes a third feature image corresponding to the second feature image, a third window label image corresponding to the second window label image, and a third key point label image corresponding to the second key point label image.
[0114] The following processing is performed during the final iteration:
[0115] The heatmap is obtained by performing a first process on the second output result of the (P-1)th iteration.
[0116] In the case of the first iteration process, the first feature image is the feature image, the first window marker image is the window marker image, and the first key point marker image is the key point marker image.
[0117] Specifically, in the first stage (stage1), according to tokens The Visual-Window-Keypoint Transformer Block process is performed on 1 window token and K keypoint tokens to obtain the first output result, including The second feature image corresponding to each token, The second window marker graph corresponding to each of the K window tokens and the second keypoint marker graph corresponding to each of the K keypoint tokens, based on... The second feature image corresponding to each token, Patch Merging and Keypoint Expansion operations are performed on the second window marker image corresponding to K window tokens and the second keypoint marker image corresponding to K keypoint tokens to obtain the third feature image corresponding to the second feature image, the third window marker image corresponding to the second window marker image, and the third keypoint marker image corresponding to the second keypoint marker image. This information (the third feature image corresponding to the second feature image, the third window marker image corresponding to the second window marker image, and the third keypoint marker image corresponding to the second keypoint marker image) is used as input information for the Visual-Window-Keypoint Transformer Block process in the second stage (stage 2). In stage 2, the Visual-Window-Keypoint Transformer Block process is performed based on the input information to obtain the first output result. Patch Merging and Keypoint Expansion operations are then performed on the first output result to obtain the second output result. This second output result is used as input information for the Visual-Window-Keypoint Transformer Block process in stage 3. In stage 3, the Visual-Window-Keypoint Transformer Block process is performed based on the input information to obtain the Visual-Window-Keypoint Transformer... The first output of the Block process is processed by Patch Merging and Keypoint Expansion operations to obtain a second output. This second output serves as the input information for the Visual-Window-Keypoint Transformer Block process in stage 4. In stage 4, the Visual-Window-Keypoint Transformer Block process is executed based on the input information, yielding the first output of the Visual-Window-Keypoint Transformer Block process. This first output includes... The second feature image corresponding to each token The second window marker map corresponding to K window tokens and the second keypoint marker map corresponding to K keypoint tokens are obtained by regression processing of the second keypoint marker map corresponding to K keypoint tokens. The K heat maps are used to predict the coordinates of key points of organisms.
[0118] Furthermore, the target value also includes the number of times the first processing is performed in each iteration, Q, where Q is an integer greater than or equal to 1. In this optional embodiment, each stage is initialized to include Q blocks, that is, each stage performs the first processing Q times. Specifically, each stage is initialized to have {s0,s1,s2,s3} blocks respectively.
[0119] The first processing based on the first feature image, the first window marker map, and the first key point marker map to obtain the first output result includes:
[0120] The following processing is performed during each iteration:
[0121] A first processing is performed based on the first target feature image, the first target window marker image, and the first target key point marker image to obtain a third output result, and the third output result is the input information for the next first processing. The third output result includes: a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0122] In the case of the first processing in each iteration, the first target feature image is the first feature image, the first target window marker image is the first window marker image, and the first target key point marker image is the first key point marker image.
[0123] In the case of the last first processing in each iteration, the second target feature image is the second feature image, the second target window marker image is the second window marker image, and the second target key point marker image is the second key point marker image.
[0124] Specifically, a Visual-Window-Keypoint Transformer Block process is performed in each block of each stage. During the first processing, the first feature image, the first window marker image, and the first keypoint marker image are used as input information for the Visual-Window-Keypoint Transformer Block process. The Visual-Window-Keypoint Transformer Block process is performed based on the input information to obtain the third output result of the Visual-Window-Keypoint Transformer Block process. The third output result of the previous Visual-Window-Keypoint Transformer Block process is the input information for the next Visual-Window-Keypoint Transformer Block process, until the last Visual-Window-Keypoint Transformer Block process of the stage is performed. The output result obtained is the second feature image, the second window marker image, and the second keypoint marker image.
[0125] The following details the first processing step, namely the Visual-Window-Keypoint Transformer Block flow. Please refer to [link / reference]. Figure 2 The first processing based on the first target feature image, the first target window marker map, and the first target key point marker map to obtain the third output result includes:
[0126] Each of the first target feature images is divided into at least one window; each window corresponds to a first target window marker map. In this embodiment, for each first target feature image, the 2D feature map X is... F ∈R h×w×C Divide it into A window of size M×M is represented as X=[X1,X2,...,X...]. n Each window has a corresponding window token W. i Thus, X is obtained. i =[X i W i ];
[0127] After Layer Normalization (LN), based on the window and its corresponding first target window token map, a third target window token map is obtained after learning image information from the first target window token map. Optionally, in this embodiment, this step includes: performing a multi-head self-attention operation (Visual-Window Tokens Attention) on the window to obtain a processed window; using the first target window token map to learn image information from the processed window to obtain the third target window token map. Specifically, for n X... i ([X i W i Perform multi-head self-attention operations separately to obtain the processed window (window X). i And the processed first target window token W i , window token W i Extracting (learning) Windows X i From the image information, n third target window tokens are obtained;
[0128] After LN, based on the third target window token map and the first target keypoint token map, a third target keypoint token map is obtained after learning keypoint information from the first target keypoint token map. Optionally, in this embodiment, this step includes: performing multi-head self-attention (Window-Keypoint Tokens Attention) on the third target window token map and the first target keypoint token map to obtain keypoint information of the third target window token map; and using the first target keypoint token map to learn the keypoint information of the third target window token map to obtain the third target keypoint token map. Specifically, n W... i (Window tokens) and K keypoint tokens are combined to perform multi-head self-attention operations. Keypoint tokens learn key point information from the image, and constraint relationships are learned between keypoint tokens. Window tokens enable the interaction of image information between windows.
[0129] After LN, a pre-defined multi-layer perception (MLP) is used to map the first target feature image X, the third target window marker image W, and the third target key point marker image K, resulting in a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image. The formulaic representation of these steps is as follows:
[0130] [X',W']=VW-MSA(LN([X;W]))
[0131] [W";K']=WK-MSA(LN([W';K]))
[0132] [X; W; K]=MLP(LN([X'; S"; K']))
[0133] The first formula above represents performing a VW-MSA operation on X and W to obtain X' and W', where X represents the image token feature, which is directly calculated from the image, i.e., the first target feature image; W refers to the window feature, a learnable parameter that extracts all visual features within a window and varies according to the number of windows, i.e., the third target window marker image; K is the keypoint feature, i.e., the third target keypoint marker image, which is 17 in the COCO dataset. VW-MSA stands for Visual-Window-MSA, and MSA stands for Measurement Systems Analysis.
[0134] The second formula above indicates that W' and K undergo a WK-MSA operation to obtain W”,K', where WK-MSA is Window-Keypoint-MSA.
[0135] The third formula above indicates that X, W, K are processed by MLP to obtain new X, W, K (i.e., X', S", K').
[0136] The following details the process of performing the second processing, namely Patch Merging and Keypoint Expansion operations. Specifically, the second processing of the first output result to obtain the second output result includes:
[0137] The second feature image is convolved using a preset second convolutional neural network to obtain a third feature image corresponding to the second feature image. The size of the third feature image is smaller than that of the second feature image. Specifically, in this embodiment, the second feature image is convolved using a preset second convolutional neural network with a kernel size of 3, a stride of 2, an input channel number of C, and an output channel number of 2×C. This reduces the size of the second feature image to half of its original size to obtain the third feature image.
[0138] The second window token image is convolved using a preset second convolutional neural network to obtain a third window token image corresponding to the second window token image. The number of the second window token images is greater than the number of the third window token images, and the number of channels of the second window token images is less than the number of channels of the third window token images. Specifically, in this embodiment, the second feature image is convolved using a preset second convolutional neural network with a kernel size of 3, a stride of 2, an input channel number of C, and an output channel number of 2×C. This reduces the number of window tokens (second window token images) to one-quarter of the original number and increases the number of channels to twice the original number, thus obtaining the third window token image.
[0139] The second keypoint marker map is processed using a preset multi-layer perception to obtain a third keypoint marker map corresponding to the second keypoint marker map. The number of channels in the second keypoint marker map is less than the number of channels in the third keypoint marker map. Specifically, in this embodiment, the number of channels in the K keypoint tokens (second keypoint marker map) is doubled to obtain the third keypoint marker map.
[0140] The following is combined with Figure 3 and Figure 4 The specific process of the biological key point detection method provided in the embodiments of the present invention is as follows:
[0141] Step 1: Obtain the image as input. For the input image H×W×3 (generally H=256, W=192 or H=384, W=288), use overlapping convolutions for feature extraction (convolution kernel size is 8×6, stride is (4, 3)), to obtain... Feature images (i.e.) (number of tokens), with C channels;
[0142] Step 2: Initialization There are N window tokens and K keypoint tokens, and the number of channels is C, namely Covolutional Embedding Keypoint Initialization;
[0143] Step 3: Initialize the number of stages to 4, namely stage1, stage2, stage3, and stage4. Each stage has {s0, s1, s2, s3} blocks, i = 0, j = 0, where s0 is 4, s1 is 4, s2 is 12, and s3 is 4;
[0144] Step 4.1: If i < stage and j < s i , perform the process of Visual-Window-Keypoint TransformerBlock, ++j, and go to Step 4.1;
[0145] Step 4.2: If i < stage and i + 1 < stage and j = s i , perform Patch Merging and KeypointExpansion operations, ++i, j = 0, and go to Step 4.1;
[0146] Step 5: Use the classification module (MLP Head) to regress the K keypoint tokens into K heatmaps for predicting the coordinates of the key points.
[0147] The method for detecting key points of an organism provided by the embodiment of the present invention proposes a multi-scale and efficient human key point detection framework based on Transformer. It uses Transformer to learn the constraint relationships between key points, improving the detection effect; restricts the multi-head self-attention in a local window, improving the detection efficiency; at the same time, constructs a hierarchical framework to learn the multi-scale information of the image to achieve the optimal detection effect.
[0148] Among them, the designed Visual-Window-Keypoint Transformer Block structure, that is, in the first step, window tokens are used to extract information, and in the second step, keypoint information is learned from the window tokens. This structure can significantly reduce the computational amount of global multi-head self-attention, and after replacing the keypoint tokens with task-specific tokens, it can be extended to other application scenarios; constructs a hierarchical Transformer detection framework to obtain multi-scale learning of key points, and thus obtains better results.
[0149] Such as Figure 5As shown, this embodiment of the invention also provides a biological key point detection device, comprising:
[0150] The acquisition module 501 is used to acquire the feature image, window marker image and key point marker image of the image to be detected;
[0151] Processing module 502 is used to perform a first processing and a second processing based on the feature image, the window marker map, and the key point marker map to obtain a heat map; the first processing includes visual transformation processing of the window marker map and the key point feature map; the second processing includes patch merging processing and key point supplementation processing;
[0152] The detection module 503 is used to detect key points of the organism based on the heat map and obtain the detection results.
[0153] Optionally, the acquisition module 501 includes:
[0154] The acquisition unit is used to acquire the image to be detected;
[0155] The first extraction unit is used to extract features from the image to be detected using a preset first convolutional neural network to obtain the feature image.
[0156] Optionally, the processing module 502 includes:
[0157] An initialization unit is used to initialize the target value to be processed, wherein the target value includes the number of iterations P;
[0158] The first processing unit is used to perform the following processing in each iteration process, except for the last iteration process:
[0159] A first processing is performed on the first feature image, the first window marker image, and the first key point marker image to obtain a first output result. The first output result includes: a second feature image corresponding to the first feature image, a second window marker image corresponding to the first window marker image, and a second key point marker image corresponding to the first key point marker image.
[0160] The first output result is processed in a second way to obtain a second output result, and the second output result is the input information for the next iteration process. The second output result includes a third feature image corresponding to the second feature image, a third window label image corresponding to the second window label image, and a third key point label image corresponding to the second key point label image.
[0161] The following processing is performed during the final iteration:
[0162] The heatmap is obtained by performing a first process on the second output result of the (P-1)th iteration.
[0163] In the case of the first iteration process, the first feature image is the feature image, the first window marker image is the window marker image, and the first key point marker image is the key point marker image.
[0164] Optionally, the target value may also include the number of times the first processing is performed in each iteration, Q.
[0165] The first processing unit is specifically used for:
[0166] The following processing is performed during each iteration:
[0167] A first processing is performed based on the first target feature image, the first target window marker image, and the first target key point marker image to obtain a third output result, and the third output result is the input information for the next first processing. The third output result includes: a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0168] In the case of the first processing in each iteration, the first target feature image is the first feature image, the first target window marker image is the first window marker image, and the first target key point marker image is the first key point marker image.
[0169] In the case of the last first processing in each iteration, the second target feature image is the second feature image, the second target window marker image is the second window marker image, and the second target key point marker image is the second key point marker image.
[0170] Optionally, the first processing unit is specifically used for:
[0171] Each of the first target feature images is divided into at least one window; one window corresponds to one first target window marker image;
[0172] Based on the window and the first target window labeling map corresponding to the window, a third target window labeling map is obtained after learning image information from the first target window labeling map;
[0173] Based on the third target window marker map and the first target key point marker map, a third target key point marker map is obtained after learning key point information from the first target key point marker map;
[0174] Using a pre-defined multilayer perceptron, the first target feature image, the third target window marker image, and the third target key point marker image are mapped to obtain a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0175] Optionally, the first processing unit is specifically used for:
[0176] Perform a multi-head self-attention operation on the window to obtain the processed window;
[0177] A multi-head self-attention operation is performed on the first target window labeling map to obtain a processed first target window labeling map;
[0178] The image information of the processed window is learned using the processed first target window marker map to obtain the third target window marker map.
[0179] Optionally, the first processing unit is specifically used for:
[0180] Perform a multi-head self-attention operation on the third target window marker map and the first target key point marker map to obtain the key point information of the third target window marker map;
[0181] The key point information of the third target window marker map is learned by using the first target key point marker map to obtain the third target key point marker map.
[0182] Optionally, the first processing unit is specifically used for:
[0183] The second feature image is convolved using a pre-defined second convolutional neural network to obtain a third feature image corresponding to the second feature image; the size of the third feature image is smaller than that of the second feature image.
[0184] The second window labeling image is convolved using a pre-defined second convolutional neural network to obtain a third window labeling image corresponding to the second window labeling image; the number of the second window labeling images is greater than the number of the third window labeling images, and the number of channels in the second window labeling images is less than the number of channels in the third window labeling image;
[0185] The second key point marker image is processed using a preset multilayer perceptron to obtain a third key point marker image corresponding to the second key point marker image; the number of channels in the second key point marker image is less than the number of channels in the third key point marker image.
[0186] It should be noted that the biological key point detection device provided in the embodiments of the present invention is a device capable of performing the above-described biological key point detection method. Therefore, all embodiments of the above-described biological key point detection method are applicable to this device and can achieve the same or similar technical effects.
[0187] like Figure 6 As shown, this embodiment of the invention also provides a biological key point detection device, including: a processor 601; and a memory 603 connected to the processor 601 via a bus interface 602. The memory 603 is used to store the programs and data used by the processor 601 when performing operations, and the processor 601 calls and executes the programs and data stored in the memory 603.
[0188] The transceiver 604 is connected to the bus interface 602 and is used to receive and send data under the control of the processor 601. Specifically, the processor 601 is used to read the program in the memory 603 and execute the following processes:
[0189] Obtain the feature image, window marker map, and key point marker map of the image to be detected;
[0190] A heatmap is obtained by performing a first process and a second process based on the feature image, the window marker map, and the key point marker map; the first process includes visual transformation processing of the window marker map and the key point feature map; the second process includes patch merging processing and key point supplementation processing.
[0191] Based on the heat map, key points of the organism are detected, and the detection results are obtained.
[0192] Optionally, the processor 601 is configured to:
[0193] Acquire the image to be detected;
[0194] The feature image is obtained by extracting features from the image to be detected using a preset first convolutional neural network.
[0195] Optionally, the processor 601 is configured to:
[0196] Initialize the target value for processing, where the target value includes the iteration number P;
[0197] The following processing is performed in each iteration except the last iteration:
[0198] A first processing is performed on the first feature image, the first window marker image, and the first key point marker image to obtain a first output result. The first output result includes: a second feature image corresponding to the first feature image, a second window marker image corresponding to the first window marker image, and a second key point marker image corresponding to the first key point marker image.
[0199] The first output result is processed in a second way to obtain a second output result, and the second output result is the input information for the next iteration process. The second output result includes a third feature image corresponding to the second feature image, a third window label image corresponding to the second window label image, and a third key point label image corresponding to the second key point label image.
[0200] The following processing is performed during the final iteration:
[0201] The heatmap is obtained by performing a first process on the second output result of the (P-1)th iteration.
[0202] In the case of the first iteration process, the first feature image is the feature image, the first window marker image is the window marker image, and the first key point marker image is the key point marker image.
[0203] Optionally, the target value may also include the number of times the first processing is performed in each iteration, Q.
[0204] The processor 601 is specifically used for:
[0205] The following processing is performed during each iteration:
[0206] A first processing is performed based on the first target feature image, the first target window marker image, and the first target key point marker image to obtain a third output result, and the third output result is the input information for the next first processing. The third output result includes: a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0207] In the case of the first processing in each iteration, the first target feature image is the first feature image, the first target window marker image is the first window marker image, and the first target key point marker image is the first key point marker image.
[0208] In the case of the last first processing in each iteration, the second target feature image is the second feature image, the second target window marker image is the second window marker image, and the second target key point marker image is the second key point marker image.
[0209] Optionally, the processor 601 is specifically used for:
[0210] Each of the first target feature images is divided into at least one window; one window corresponds to one first target window marker image;
[0211] Based on the window and the first target window labeling map corresponding to the window, a third target window labeling map is obtained after learning image information from the first target window labeling map;
[0212] Based on the third target window marker map and the first target key point marker map, a third target key point marker map is obtained after learning key point information from the first target key point marker map;
[0213] Using a pre-defined multilayer perceptron, the first target feature image, the third target window marker image, and the third target key point marker image are mapped to obtain a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
[0214] Optionally, the processor 601 is specifically used for:
[0215] Perform a multi-head self-attention operation on the window to obtain the processed window;
[0216] A multi-head self-attention operation is performed on the first target window labeling map to obtain a processed first target window labeling map;
[0217] The image information of the processed window is learned using the processed first target window marker map to obtain the third target window marker map.
[0218] Optionally, the processor 601 is specifically used for:
[0219] Perform a multi-head self-attention operation on the third target window marker map and the first target key point marker map to obtain the key point information of the third target window marker map;
[0220] The key point information of the third target window marker map is learned by using the first target key point marker map to obtain the third target key point marker map.
[0221] Optionally, the processor 601 is specifically used for:
[0222] The second feature image is convolved using a pre-defined second convolutional neural network to obtain a third feature image corresponding to the second feature image; the size of the third feature image is smaller than that of the second feature image.
[0223] The second window labeling image is convolved using a pre-defined second convolutional neural network to obtain a third window labeling image corresponding to the second window labeling image; the number of the second window labeling images is greater than the number of the third window labeling images, and the number of channels in the second window labeling images is less than the number of channels in the third window labeling image;
[0224] The second key point marker image is processed using a preset multilayer perceptron to obtain a third key point marker image corresponding to the second key point marker image; the number of channels in the second key point marker image is less than the number of channels in the third key point marker image.
[0225] Among them, Figure 6 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 601) and memory (memory 603). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides a user interface 605. A transceiver 604 may be multiple elements, including transmitters and receivers, providing units for communicating with various other devices over a transmission medium. Processor 601 is responsible for managing the bus architecture and general processing, and memory 603 may store data used by processor 601 during operation.
[0226] In addition, specific embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the biological key point detection method as described above.
[0227] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0228] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.
[0229] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0230] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for detecting key points in a biological organism, characterized in that, include: Obtain the feature image, window marker map, and key point marker map of the image to be detected; A heatmap is obtained by performing a first process and a second process based on the feature image, the window marker map, and the key point marker map; the first process includes visual transformation processing of the window marker map and the key point feature map; the second process includes patch merging processing and key point supplementation processing. Based on the heat map, key points of the organism are detected, and the detection results are obtained. The step of obtaining a heatmap by performing first and second processing based on the feature image, the window marker map, and the key point marker map includes: Initialize the target value for processing, where the target value includes the iteration number P; The following processing is performed in each iteration except the last iteration: A first processing is performed on the first feature image, the first window marker image, and the first key point marker image to obtain a first output result. The first output result includes: a second feature image corresponding to the first feature image, a second window marker image corresponding to the first window marker image, and a second key point marker image corresponding to the first key point marker image. The first output result is processed in a second way to obtain a second output result, and the second output result is the input information for the next iteration process. The second output result includes a third feature image corresponding to the second feature image, a third window label image corresponding to the second window label image, and a third key point label image corresponding to the second key point label image. The following processing is performed during the final iteration: The heatmap is obtained by performing a first process on the second output result of the (P-1)th iteration. In the case of the first iteration process, the first feature image is the feature image, the first window marker image is the window marker image, and the first key point marker image is the key point marker image; The step of performing a second processing on the first output result to obtain a second output result includes: The second feature image is convolved using a pre-defined second convolutional neural network to obtain a third feature image corresponding to the second feature image; the size of the third feature image is smaller than that of the second feature image. The second window labeling image is convolved using a pre-defined second convolutional neural network to obtain a third window labeling image corresponding to the second window labeling image; the number of the second window labeling images is greater than the number of the third window labeling images, and the number of channels in the second window labeling images is less than the number of channels in the third window labeling image; The second key point marker image is processed using a preset multilayer perceptron to obtain a third key point marker image corresponding to the second key point marker image; the number of channels in the second key point marker image is less than the number of channels in the third key point marker image.
2. The method for detecting key points in a biological organism according to claim 1, characterized in that, The acquisition of the feature image of the image to be detected includes: Acquire the image to be detected; The feature image is obtained by extracting features from the image to be detected using a preset first convolutional neural network.
3. The method for detecting key points in a biological organism according to claim 1, characterized in that, The target value also includes Q, the number of times the first processing is performed in each iteration; The first processing based on the first feature image, the first window marker map, and the first key point marker map to obtain the first output result includes: The following processing is performed during each iteration: A first processing is performed based on the first target feature image, the first target window marker image, and the first target key point marker image to obtain a third output result, and the third output result is the input information for the next first processing. The third output result includes: a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image. In the case of the first processing in each iteration, the first target feature image is the first feature image, the first target window marker image is the first window marker image, and the first target key point marker image is the first key point marker image. In the case of the last first processing in each iteration, the second target feature image is the second feature image, the second target window marker image is the second window marker image, and the second target key point marker image is the second key point marker image.
4. The method for detecting key points in a biological organism according to claim 3, characterized in that, The first processing based on the first target feature image, the first target window marker map, and the first target key point marker map to obtain the third output result includes: Each of the first target feature images is divided into at least one window; one window corresponds to one first target window marker image; Based on the window and the first target window labeling map corresponding to the window, a third target window labeling map is obtained after learning image information from the first target window labeling map; Based on the third target window marker map and the first target key point marker map, a third target key point marker map is obtained after learning key point information from the first target key point marker map; Using a pre-defined multilayer perceptron, the first target feature image, the third target window marker image, and the third target key point marker image are mapped to obtain a second target feature image corresponding to the first target feature image, a second target window marker image corresponding to the first target window marker image, and a second target key point marker image corresponding to the first target key point marker image.
5. The method for detecting key points in a biological organism according to claim 4, characterized in that, The step of obtaining a third target window label map based on the window and the first target window label map corresponding to the window, after learning image information from the first target window label map, includes: Perform a multi-head self-attention operation on the window to obtain the processed window; A multi-head self-attention operation is performed on the first target window labeling map to obtain a processed first target window labeling map; The image information of the processed window is learned using the processed first target window marker map to obtain the third target window marker map.
6. The method for detecting key points in an organism according to claim 4, characterized in that, The step of obtaining the third target keypoint marker map after learning keypoint information from the first target keypoint marker map based on the third target window marker map and the first target keypoint marker map includes: Perform a multi-head self-attention operation on the third target window marker map and the first target key point marker map to obtain the key point information of the third target window marker map; The key point information of the third target window marker map is learned by using the first target key point marker map to obtain the third target key point marker map.
7. A device for detecting key points in a living organism, characterized in that, include: The acquisition module is used to acquire the feature image, window marker map, and key point marker map of the image to be detected; The processing module is configured to perform a first processing and a second processing based on the feature image, the window marker map, and the key point marker map to obtain a heat map; the first processing includes visual transformation processing of the window marker map and the key point feature map; the second processing includes patch merging processing and key point supplementation processing; The detection module is used to detect key points of the organism based on the heat map and obtain the detection results; The processing module includes: An initialization unit is used to initialize the target value to be processed, wherein the target value includes the number of iterations P; The first processing unit is used to perform the following processing in each iteration process, except for the last iteration process: A first processing is performed on the first feature image, the first window marker image, and the first key point marker image to obtain a first output result. The first output result includes: a second feature image corresponding to the first feature image, a second window marker image corresponding to the first window marker image, and a second key point marker image corresponding to the first key point marker image. The first output result is processed in a second way to obtain a second output result, and the second output result is the input information for the next iteration process. The second output result includes a third feature image corresponding to the second feature image, a third window label image corresponding to the second window label image, and a third key point label image corresponding to the second key point label image. The following processing is performed during the final iteration: The heatmap is obtained by performing a first process on the second output result of the (P-1)th iteration. In the case of the first iteration process, the first feature image is the feature image, the first window marker image is the window marker image, and the first key point marker image is the key point marker image; The first processing unit is specifically used for: The second feature image is convolved using a pre-defined second convolutional neural network to obtain a third feature image corresponding to the second feature image; the size of the third feature image is smaller than that of the second feature image. The second window labeling image is convolved using a pre-defined second convolutional neural network to obtain a third window labeling image corresponding to the second window labeling image; the number of the second window labeling images is greater than the number of the third window labeling images, and the number of channels in the second window labeling images is less than the number of channels in the third window labeling image; The second key point marker image is processed using a preset multilayer perceptron to obtain a third key point marker image corresponding to the second key point marker image; the number of channels in the second key point marker image is less than the number of channels in the third key point marker image.
8. A biological key point detection device, comprising: A transceiver, a processor, a memory, and a program or instructions stored in the memory and executable on the processor; characterized in that, when the processor executes the program or instructions, it implements the steps of the biological key point detection method as described in any one of claims 1 to 6.
9. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps in the biological key point detection method as described in any one of claims 1 to 6.