A bird posture monitoring device based on attention and multi-scale dual mechanisms

Abstract

This invention discloses a bird pose monitoring device based on a dual mechanism of attention and multi-scale, comprising a VHR-BirdPose network. The VHR-BirdPose network includes: a ViT attention subnetwork for extracting and outputting attention features from the input original bird image; an HRNet multi-scale subnetwork for capturing and outputting bird pose features at different scales from the input original bird image; and a feature fusion unit for fusing the attention features output from the ViT attention subnetwork and the bird pose features at different scales output from the HRNet multi-scale subnetwork, outputting the fused bird features represented by a heatmap. This invention belongs to the field of image information processing technology and is capable of estimating bird pose.

Description

Technical Field

[0001] This invention relates to the field of image information processing technology, and in particular to a bird posture monitoring device based on a dual mechanism of attention and multi-scale. Background Technology

[0002] Pose estimation refers to the process of automatically identifying and analyzing the posture, movements, and motions of humans or animals using computer vision and machine learning techniques. Bird pose estimation holds significant importance and application value in bird research and conservation. For bird behavior research, pose estimation helps researchers gain a deeper understanding of bird behavior patterns, flight methods, and communication methods. Bird pose estimation is an important prerequisite for tasks such as skeleton-based behavior recognition, behavior analysis, and object tracking in birds. Bird pose estimation faces several challenges and difficulties, including the vast diversity of bird species, the variety of postures, and the complexity of posture structures.

[0003] Bird postures exhibit complex structures and fine-grained variations, demanding high accuracy and robustness from posture estimation algorithms capable of accurately capturing subtle changes. Current posture estimation methods struggle to address the challenges and difficulties inherent in bird posture estimation. Summary of the Invention

[0004] The purpose of this invention is to provide a bird posture monitoring device based on attention and multi-scale dual mechanisms, which can realize complex feature representation, temporal modeling, and accurate and stable posture estimation.

[0005] To achieve the above objectives, the present invention provides a bird posture monitoring device based on a dual mechanism of attention and multi-scale, comprising a VHR-BirdPose network, wherein the VHR-BirdPose network has:

[0006] The ViT attention subnetwork is used to extract attention features from the input raw bird image and output them.

[0007] The HRNet multi-scale subnetwork is used to capture bird pose features at different scales in the input raw bird image and output them.

[0008] The feature fusion unit is used to fuse the attention features output by the ViT attention subnetwork and the bird pose features of different scales output by the HRNet multi-scale subnetwork, and output the fused bird features represented by a heatmap.

[0009] Furthermore, the ViT attention subnetwork includes:

[0010] The image patch embedding layer is used to cut the input original bird image of size H×W×C into image patches of width and height P; where H×W is the resolution of the original bird image and C is the number of channels of the original bird image.

[0011] The first linear layer, with C input channels and D output channels, is used to extract images of length from each image patch. A feature block sequence with D channels;

[0012] The location fusion layer adds the feature block sequence to a location embedding of equal length, consisting of randomly initialized learnable parameters, to obtain the image sequence. ;

[0013] The encoder module includes N stacked attention modules for processing image sequences layer by layer. The feature information of each image is extracted, and the extracted feature information is used as the attention feature extracted by the ViT attention sub-network;

[0014] The attention module has the following features:

[0015] The fourth linear layer, with D input channels and 3D output channels, is used to process image sequences. Mapped to the first input sequence Then take the first input sequence It is divided into three equal parts along the channel dimension, and finally further divided into h parts along the channel dimension, all with the same size. The query, key, and value;

[0016] The attention mechanism has h heads, and each head is used to compute on a set of input Query, Key, and Value to obtain... Then merge the h calculation results The second output sequence is obtained. ;

[0017] The multilayer perceptron structure consists of a second linear layer with 4D output channels and a third linear layer with D output channels stacked together, with the second output sequence... The third output sequence is obtained through a multilayer perceptron structure. Thus, the fourth output sequence is obtained. Then, by analyzing the fourth output sequence Upsampling is performed to obtain the attention features extracted by the ViT attention sub-network. .

[0018] Furthermore, the VHR-BirdPose network types include S-type, B-type, L-type, and XL-type, among which:

[0019] The S-shaped VHR-BirdPose network has 106M parameters and includes 12 ViT attention subnetworks, each of which includes 6 attention modules. The width of the HRNet multi-scale subnetwork is 32.

[0020] The B-type VHR-BirdPose network has 181M parameters and includes 12 ViT attention subnetworks, each of which includes 12 attention modules. The width of the HRNet multi-scale subnetwork is 32.

[0021] The L-shaped VHR-BirdPose network has 181M parameters and includes 16 ViT attention subnetworks, each of which includes 12 attention modules. The width of the HRNet multi-scale subnetwork is 32.

[0022] The XL-type VHR-BirdPose network has 333M parameters and includes 16 ViT attention subnetworks, each of which contains 24 attention modules. The width of the HRNet multi-scale subnetwork is 32.

[0023] Furthermore, the HRNet multi-scale subnetwork includes:

[0024] The first convolutional layer, consisting of two kernels of size 3, a stride of 2, and 64 output channels, takes an original bird image of size H×W×3 as input and outputs a first feature map downsampled by 4 times. ;

[0025] The first type of residual block has 4 layers and 256 output channels. The first feature map... After passing through each layer of the first type of residual block, a second feature map is formed. ;

[0026] The second convolutional layer has a kernel size of 3, a stride of 1, and 32 output channels. The second feature map... The third feature map is formed through the second convolutional layer. ;

[0027] The third convolutional layer has a kernel size of 3, a stride of 2, and 64 output channels. The second feature map... The fourth feature map is formed through the third convolutional layer. ;

[0028] The second type of residual block has 4 layers and 32 output channels;

[0029] The third type of residual block has 4 layers and 64 output channels;

[0030] The first pairwise fusion unit is used to fuse the third feature map. The feature map formed by the second type of residual block and the fourth feature map The first fused feature map is obtained by pairwise fusion of the feature maps formed by the third type of residual blocks. Second fusion feature map ;

[0031] The fourth convolutional layer has a kernel size of 3, a stride of 2, and 128 output channels. The second fused feature map... The fourth feature map is formed through the fourth convolutional layer. ;

[0032] The fourth type of residual block has 4 layers and 128 output channels;

[0033] The second pairwise fusion unit is used to fuse the first fusion feature map. The feature map formed by the second type of residual block and the second fused feature map The feature map and the fourth feature map formed by the third type of residual block The third fused feature map is obtained by pairwise fusion of the feature maps formed by the fourth type of residual blocks. Fourth fusion feature map and the fifth fusion feature map ;

[0034] The fifth convolutional layer has a kernel size of 3, a stride of 2, and 256 output channels. The fifth fused feature map... The fifth feature map is formed through the fifth convolutional layer. ;

[0035] The third pairwise fusion unit is used to fuse the third fusion feature map. Using the feature map of the second type of residual block and the fourth fused feature map Using the feature map of the third type of residual block and the fifth fusion feature map Through the feature maps of the fourth type of residual block and the fifth feature map The sixth fused feature map is obtained by pairwise fusion of the feature maps of the first type of residual blocks. 7th fusion feature map Eighth fusion feature map Ninth fusion feature map ;

[0036] The fourth pairwise fusion unit is used to fuse the sixth feature map. Using the feature map of the second type of residual block and the seventh fusion feature map Using the feature map of the third type of residual block and the eighth fusion feature map Using the feature map of the third type of residual block and the ninth fusion feature map The feature maps of the fourth type of residual block are fused pairwise to obtain the feature maps of the four resolutions. , as the output feature of the HRNet multi-scale subnetwork.

[0037] Furthermore, the pairwise fusion operation of the first pairwise fusion unit is repeated once, and each pairwise fusion operation specifically includes:

[0038] First, the third feature map The feature map obtained by the convolutional layer with a kernel size of 3 and a stride of 2 and the fourth feature map Add them together, and then add the fourth feature map. The feature map obtained by combining a convolutional layer with a kernel size of 1 and a stride of 1 with an upsampling operation and the third feature map Add them together to obtain the first fused feature map. Second fusion feature map .

[0039] Furthermore, the pairwise fusion operation of the second pairwise fusion unit is repeated four times, and each pairwise fusion operation specifically includes:

[0040] First, obtain the first fused feature map. The feature map formed by the second type of residual block and the second fused feature map The feature map and the fourth feature map formed by the third type of residual block The highest-resolution feature map and the lowest-resolution feature map formed from the fourth type of residual blocks are then combined with the lowest-resolution feature map by passing the highest-resolution feature map through a convolutional layer with a kernel size of 3 and a stride of 2. This results in the third fused feature map. Fourth fusion feature map and the fifth fusion feature map .

[0041] Furthermore, the pairwise fusion operation of the third pairwise fusion unit is repeated twice, and each pairwise fusion operation specifically includes:

[0042] First obtain the third fusion feature map The feature map formed by the second type of residual block and the fourth fused feature map The feature map formed by the third type of residual block and the fifth fused feature map The feature map and the fifth feature map formed by the fourth type of residual block The highest-resolution feature map and the lowest-resolution feature map are obtained from the feature maps formed by the first type of residual blocks. Then, the highest-resolution feature map is passed through a convolutional layer with a kernel size of 3 and a stride of 2 to obtain the feature map, which is then added to the lowest-resolution feature map to obtain the sixth fused feature map. 7th fusion feature map Eighth fusion feature map Ninth fusion feature map .

[0043] Furthermore, the pairwise fusion operation of the fourth pairwise fusion unit is repeated twice, and each pairwise fusion operation specifically includes:

[0044] First, fuse the seventh feature map Eighth fusion feature map Ninth fusion feature map The feature map obtained by combining a convolutional layer with a kernel size of 3 and a stride of 2 with an upsampling operation is fused with the sixth feature map. Adding them together yields bird posture features at different scales. .

[0045] Furthermore, the feature fusion unit specifically includes:

[0046] Channel-dimensional concatenation unit, used to concatenate the attention features extracted by the ViT attention subnetwork. Bird posture characteristics at different scales By concatenating along the channel dimension, we obtain fused features that integrate attention and multi-scale dual mechanisms. ;

[0047] A fully connected layer with c output channels is used to receive fused features. This is transformed into a heatmap, which is the final output of the model. Where c is the number of keypoints in the pose estimation task.

[0048] Furthermore, the bird posture monitoring device based on attention and multi-scale dual mechanisms also includes a network parameter optimization model, whose loss function... It is expressed as shown in the following formula (1):

[0049] (1)

[0050] In the formula, n represents the number of skeletal nodes, which are in the following order: head center, left eye, right eye, top of front of mouth, left of back of mouth, right of back of mouth, bottom of front of mouth, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, middle and rear of body, left hip, right hip, left knee, right knee, left ankle, right ankle, top of tail, middle of tail, and rear of tail. and These are the width and height pixel values ​​of the heatmap, respectively. The method predicts the output heatmap. This is a ground truth heatmap. For the first The weights of each key point.

[0051] This invention is used to extract high-resolution and global features, which can effectively handle the diversity of bird postures and improve the accuracy and robustness of posture estimation. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the bird posture monitoring device based on attention and multi-scale dual mechanisms according to an embodiment of the present invention. Detailed Implementation

[0053] In the accompanying drawings, the same or similar reference numerals are used to denote the same or similar elements or elements having the same or similar functions. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0054] The bird posture monitoring device based on attention and multi-scale dual mechanisms provided in this invention includes a VHR-BirdPose network. The VHR-BirdPose network has a ViT (Vision Transformer) attention subnetwork, an HRNet multi-scale subnetwork, and a feature fusion unit, wherein:

[0055] The ViT attention subnetwork, implemented using ViT, is used to extract attention features from the input raw bird image and output them.

[0056] The HRNet multi-scale sub-network is implemented using HRNet. It is used to capture and output bird pose features at different scales in the input raw bird image.

[0057] The feature fusion unit is used to fuse the attention features output by the ViT attention subnetwork and the bird pose features of different scales output by the HRNet multi-scale subnetwork, and output the fused bird features represented by a heatmap.

[0058] The embodiments of the present invention utilize the ViT attention sub-network to enhance the attention of key points and extract richer semantic information, and combine the HRNet multi-scale sub-network to improve the accuracy of bird pose estimation, so that the method has better performance and characteristics in bird pose estimation tasks.

[0059] In one embodiment, the ViT attention subnetwork includes a patch embedding layer, a first linear layer, a location fusion layer, and an encoder module. The patch embedding layer segments the input original bird image of size H×W×C into patches of width and height P. Here, H×W is the resolution of the original bird image, and C is the number of channels in the original bird image. The first linear layer has C input channels and D output channels, and is used to extract a length of P from each patch. A feature block sequence with D channels is used. The position fusion layer adds the feature block sequence to a position embedding of equal length, composed of randomly initialized learnable parameters, to obtain the image sequence. The encoder module includes N stacked attention modules, used to process the image sequence layer by layer through the N attention modules. The feature information of each image is extracted, and the extracted feature information is used as the attention feature extracted by the ViT attention sub-network.

[0060] As a preferred embodiment of the attention module, the attention module has a fourth linear layer, an attention mechanism, and a multilayer perceptron (MLP) structure, wherein: the fourth linear layer has D input channels and 3D output channels, and is used to process image sequences. Mapped to the first input sequence Then take the first input sequence It is divided into three equal parts along the channel dimension, and finally further divided into h parts along the channel dimension, all with the same size. The query, key, and value are used. The attention mechanism enables the neural network model to focus on useful features; in this case, it enhances the model's attention to key features of birds. The number of attention mechanisms is h, and each attention mechanism is used to compute a set of query, key, and value inputs, resulting in... Then merge the h calculation results The second output sequence is obtained. The multilayer perceptron architecture consists of a second linear layer with 4D output channels and a third linear layer with D output channels stacked together. The second output sequence... The third output sequence is obtained through a multilayer perceptron structure. Thus, the fourth output sequence is obtained. Then the fourth output sequence Organized into Finally, by analyzing the fourth output sequence Upsampling is performed to obtain the attention features extracted by the ViT attention sub-network. .

[0061] It should be noted that those skilled in the art can also make changes to the composition structure of the attention module disclosed in the above embodiments to obtain other structural forms of the attention module, as long as it can achieve the function of extracting feature information from the image.

[0062] In the above embodiments, the ViT attention subnetwork is introduced as a feature extractor to capture the global dependencies of the image, enhance the attention to key points of birds, and improve the accuracy of pose estimation.

[0063] In one embodiment, the formula used by each attention mechanism to compute a set of Query, Key, and Value from the input is as follows:

[0064]

[0065] In the formula, Q corresponds to Query, V corresponds to Value, and K corresponds to Key.

[0066] In one embodiment, the VHR-BirdPose network includes S-type, B-type, L-type, and XL-type networks, wherein:

[0067] The S-shaped VHR-BirdPose network has 106M parameters and includes 12 ViT attention subnetworks, each of which has 6 attention modules. The width of the HRNet multi-scale subnetwork is 32.

[0068] The B-type VHR-BirdPose network has 181M parameters and includes 12 ViT attention subnetworks, each of which has 12 attention modules. The width of the HRNet multi-scale subnetwork is 32.

[0069] The L-shaped VHR-BirdPose network has 181M parameters and includes 16 ViT attention subnetworks, each of which contains 12 attention modules. The width of the HRNet multi-scale subnetwork is 32.

[0070] The XL-type VHR-BirdPose network has 333M parameters and includes 16 ViT attention subnetworks, each of which contains 24 attention modules. The width of the HRNet multi-scale subnetwork is 32.

[0071] By setting the number of heads in the ViT attention subnetwork, the number of layers of attention modules contained in each ViT attention subnetwork, and the width of the HRNet multi-scale subnetwork, VHR-BirdPose networks with different parameters can be obtained to suit different tasks.

[0072] In one embodiment, the HRNet multi-scale sub-network includes a first convolutional layer, a first type of residual block (BottleNeck), a second convolutional layer, a third convolutional layer, a second type of residual block, a third type of residual block, a first pairwise fusion unit, a fourth convolutional layer, a fourth type of residual block, a second pairwise fusion unit, a fifth convolutional layer, a third pairwise fusion unit, and a fourth pairwise fusion unit, wherein:

[0073] The first convolutional layer consists of two convolutional kernels of size 3, a stride of 2, and 64 output channels. It takes an original bird image of size H×W×3 as input and outputs a first feature map downsampled by 4 times. .

[0074] The first type of residual block has 4 layers and 256 output channels. The first feature map... After passing through each layer of the first type of residual block, a second feature map is formed. .

[0075] The second convolutional layer has a kernel size of 3, a stride of 1, and 32 output channels. The second feature map... The third feature map is formed through the second convolutional layer. .

[0076] The third convolutional layer is a convolutional layer with a kernel size of 3, a stride of 2, and 64 output channels. The second feature map... The fourth feature map is formed through the third convolutional layer. .

[0077] The second type of residual block has 4 layers and 32 output channels.

[0078] The third type of residual block has 4 layers and 64 output channels.

[0079] The first pairwise fusion unit is used to combine the third feature map The feature map formed by the second type of residual block and the fourth feature map The first fused feature map is obtained by pairwise fusion of the feature maps formed by the third type of residual blocks. Second fusion feature map The term "pairwise fusion" can be understood as adding the corresponding elements of two feature maps to obtain the fused feature map.

[0080] The fourth convolutional layer has a kernel size of 3, a stride of 2, and 128 output channels. The second fused feature map... The fourth feature map is formed through the fourth convolutional layer. .

[0081] The fourth type of residual block has 4 layers and 128 output channels.

[0082] The second pairwise fusion unit is used to combine the first fusion feature map The feature map formed by the second type of residual block and the second fused feature map The feature map and the fourth feature map formed by the third type of residual block The third fused feature map is obtained by pairwise fusion of the feature maps formed by the fourth type of residual blocks. Fourth fusion feature map and the fifth fusion feature map .

[0083] The fifth convolutional layer has a kernel size of 3, a stride of 2, and 256 output channels. The fifth fused feature map... The fifth feature map is formed through the fifth convolutional layer. .

[0084] The third pairwise fusion unit is used to combine the third fusion feature map. Using the feature map of the second type of residual block and the fourth fused feature map Using the feature map of the third type of residual block and the fifth fusion feature map Through the feature maps of the fourth type of residual block and the fifth feature map The sixth fused feature map is obtained by pairwise fusion of the feature maps of the first type of residual blocks. 7th fusion feature map Eighth fusion feature map Ninth fusion feature map .

[0085] The fourth pairwise fusion unit is used to fuse the sixth feature map. Using the feature map of the second type of residual block and the seventh fusion feature map Using the feature map of the third type of residual block and the eighth fusion feature map Using the feature map of the third type of residual block and the ninth fusion feature map The feature maps of the fourth type of residual block are fused pairwise to obtain the feature maps of the four resolutions. , as the output feature of the HRNet multi-scale subnetwork.

[0086] Of course, those skilled in the art can also modify the composition structure of the HRNet multi-scale sub-network disclosed in the above embodiments to obtain other structural forms of the HRNet multi-scale sub-network, as long as it can achieve the function of capturing bird posture features at different scales in the image.

[0087] This embodiment fully utilizes the advantages of HRNet's multi-scale sub-network in extracting multi-scale features. By retaining high-resolution and low-resolution features simultaneously through multiple parallel branches, it improves the ability to capture pose details, thereby enhancing the recognition accuracy and robustness of the method model of this invention.

[0088] In one embodiment, the pairwise fusion operation of the first pairwise fusion unit may be repeated once, but is not limited to this one time, and each pairwise fusion operation specifically includes:

[0089] First, the third feature map The feature map obtained by the convolutional layer with a kernel size of 3 and a stride of 2 and the fourth feature map Add them together, and then add the fourth feature map. The feature map and the third feature map are obtained by using a convolutional layer with a kernel size of 1 and a stride of 1, combined with an upsampling operation. Add them together to obtain the first fused feature map. Second fusion feature map .

[0090] In one embodiment, the pairwise fusion operation of the second pairwise fusion unit may be repeated four times, but is not limited to, and each pairwise fusion operation specifically includes:

[0091] First, obtain the first fused feature map. The feature map formed by the second type of residual block and the second fused feature map The feature map and the fourth feature map formed by the third type of residual block The highest-resolution feature map and the lowest-resolution feature map formed from the fourth type of residual blocks are then combined with the lowest-resolution feature map by passing the highest-resolution feature map through a convolutional layer with a kernel size of 3 and a stride of 2. This results in the third fused feature map. Fourth fusion feature map and the fifth fusion feature map .

[0092] In one embodiment, the pairwise fusion operation of the third pairwise fusion unit may be repeated twice, but is not limited to this: each pairwise fusion operation specifically includes:

[0093] First obtain the third fusion feature map The feature map formed by the second type of residual block and the fourth fused feature map The feature map formed by the third type of residual block and the fifth fused feature map The feature map and the fifth feature map formed by the fourth type of residual block The highest-resolution feature map and the lowest-resolution feature map are obtained from the feature maps formed by the first type of residual blocks. Then, the highest-resolution feature map is passed through a convolutional layer with a kernel size of 3 and a stride of 2 to obtain the feature map, which is then added to the lowest-resolution feature map to obtain the sixth fused feature map. 7th fusion feature map Eighth fusion feature map Ninth fusion feature map .

[0094] In one embodiment, the pairwise fusion operation of the fourth pairwise fusion unit may be repeated twice, but is not limited to this: Each pairwise fusion operation specifically includes:

[0095] First, fuse the seventh feature map Eighth fusion feature map Ninth fusion feature map The feature map obtained by combining a convolutional layer with a kernel size of 3 and a stride of 2 with an upsampling operation is fused with the sixth feature map. Adding them together yields bird posture features at different scales. .

[0096] In one embodiment, the feature fusion unit specifically includes a channel-dimensional splicing unit and a fully connected layer, wherein:

[0097] Channel-dimensional splicing units are used to concatenate the attention features extracted by the ViT attention subnetwork. Bird posture characteristics at different scales By concatenating along the channel dimension, we obtain fused features that integrate attention and multi-scale dual mechanisms. .

[0098] The fully connected layer has c output channels, which are used to receive fused features. This is transformed into a heatmap, which is the final output of the model. Where c represents the number of keypoints in the pose estimation task.

[0099] In one embodiment, the bird posture monitoring device based on attention and multi-scale dual mechanisms also includes a network parameter optimization model, whose loss function is... It is expressed as shown in the following formula (1):

[0100] (1)

[0101] In the formula, n represents the number of skeletal nodes, which are in the following order: head center, left eye, right eye, top of front of mouth, left of back of mouth, right of back of mouth, bottom of front of mouth, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, middle and rear of body, left hip, right hip, left knee, right knee, left ankle, right ankle, top of tail, middle of tail, and rear of tail. and These are the width and height pixel values ​​of the heatmap, respectively. The method predicts the output heatmap. This is a ground truth heatmap. For the first The weights of each key point.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An attention and multi-scale dual mechanism-based bird posture monitoring device, characterized in that, The VHR-BirdPose network comprises: a ViT attention subnetwork, which is used for extracting attention features of an input original bird image and outputting; the ViT attention subnetwork comprises an image block embedding layer, a first linear layer, a position fusion layer and an encoder module, wherein the image block embedding layer cuts the input image into image blocks, and the encoder module comprises N stacked multi-head attention modules, which are used for extracting global dependency features; an HRNet multi-scale subnetwork, which is used for capturing bird posture features of different scales in the input original bird image and outputting; the HRNet multi-scale subnetwork comprises a multi-branch convolution layer, a residual block and at least one two-by-two fusion unit, which are used for maintaining high-resolution features and realizing multi-scale feature interaction through the fusion unit; a feature fusion unit, which is used for fusing the attention features output by the ViT attention subnetwork and the bird posture features of different scales output by the HRNet multi-scale subnetwork, and outputting fused bird features represented by a heat map; the fusion comprises channel dimension splicing, and the heat map is used for bird posture estimation.

2. The attention and multi-scale dual mechanism based bird posture monitoring device of claim 1, wherein, In the ViT attention subnetwork; The image block embedding layer is used for cutting the input original bird image with a size of HxWxC into image blocks with a width and a height of P; wherein HxW is the resolution of the original bird image, and C is the channel number of the original bird image; The first linear layer has C input channels and D output channels, and is used to extract images of length L = HW / P from each image patch. 2 A feature block sequence with D channels; The position fusion layer is used to add the feature block sequence and the position embedding of the same length and composed of randomly initialized learnable parameters to obtain an image sequence The encoder module is used for extracting feature information of each image in the image sequence x1 through N layers of attention modules layer by layer, and the extracted feature information is used as the attention features extracted by the ViT attention subnetwork; The attention module has: a fourth linear layer with D input channels and 3D output channels for mapping the image sequence x1 into a first input sequence The first input sequence x ′ 1 is split into three equal parts in the channel dimension, and finally split into h parts in the channel dimension with size Query, Key and Value; a number of attention mechanisms, h, each of which is used to compute an input Query, Key, and Value to obtain The h results are then combined to obtain a second output sequence A multi-layer perceptron structure stacked by a second linear layer with an output channel number of 4D and a third linear layer with an output channel number of D, a second output sequence x ′ 2Obtaining a third output sequence by a multi-layer perceptron structure Thus obtaining a fourth output sequence Again, performing upsampling on the fourth output sequence Obtaining an attention feature extracted by the ViT attention subnetwork 3. The attention and multi-scale dual mechanism based bird posture monitoring device of claim 2, wherein, The types of the VHR-BirdPose network include S type, B type, L type and XL type, wherein: The S type VHR-BirdPose network has a parameter amount of 106M, and comprises 12 ViT attention subnetworks, each of which comprises 6 layers of attention modules, and the HRNet multi-scale subnetwork has a width of 32; The B type VHR-BirdPose network has a parameter amount of 181M, and comprises 12 ViT attention subnetworks, each of which comprises 12 layers of attention modules, and the HRNet multi-scale subnetwork has a width of 32; The L type VHR-BirdPose network has a parameter amount of 181M, and comprises 16 ViT attention subnetworks, each of which comprises 12 layers of attention modules, and the HRNet multi-scale subnetwork has a width of 32; The XL type VHR-BirdPose network has a parameter amount of 333M, and comprises 16 ViT attention subnetworks, each of which comprises 24 layers of attention modules, and the HRNet multi-scale subnetwork has a width of 32.

4. The attention and multi-scale dual mechanism based bird posture monitoring device of claim 2, wherein, The HRNet multi-scale subnetwork specifically comprises: The first convolutional layer is a convolutional layer with two layers of convolutional kernels with a size of 3, a step of 2, and an output channel number of 64, which is used for inputting an original bird image with a size of HxWx3, and outputs a first feature map with a down-sampling of 4 times The first type of residual block has 4 layers, and the number of output channels is 256. After the first feature map x1 passes through each layer of the first type of residual block in turn, a second feature map is formed a second convolutional layer with a kernel size of 3, a stride of 1, and 32 output channels, to generate a second feature map x ′ 1by the second convolutional layer, a third feature map x a third convolutional layer which is a convolutional layer with a kernel size of 3, a stride of 2, and an output channel number of 64, the second feature map x ′ 1by the third convolutional layer, a fourth feature map x a second type of residual block, which has 4 layers and an output channel number of 32; a third type of residual block, which has 4 layers and an output channel number of 64; a first pairwise fusion unit configured to fuse the third feature maps the feature maps formed by the second type of residual block with the fourth feature maps the feature maps formed by the third type of residual block pairwise fused to obtain first fused feature maps and second fused feature maps a fourth convolutional layer with a convolution kernel size of 3, a stride of 2, and 128 output channels, the second fused feature map The fourth feature map is formed by the fourth convolutional layer a fourth type of residual block, which has 4 layers and an output channel number of 128; a second pairwise fusion unit configured to fuse the first fusion feature maps a feature map formed by the second type of residual block, a second fusion feature map a feature map formed by the third type of residual block, a fourth feature map fuse the feature maps formed by the fourth type of residual block pairwise to obtain a third fusion feature map a fourth fusion feature map and a fifth fusion feature map a fifth convolutional layer which is a convolutional layer with a convolution kernel size of 3, a stride of 2, and an output channel number of 256, and a fifth fused feature map The fifth feature map is formed by the fifth convolutional layer a third pairwise fusion unit configured to fuse third feature maps a fourth fused feature map by the feature maps of the second type of residual block a fifth fused feature map by the feature maps of the third type of residual block a fifth feature map by the feature maps of the fourth type of residual block a sixth fused feature map by pairwise fusion of the feature maps of the first type of residual block a seventh fused feature map an eighth fused feature map a ninth fused feature map a fourth pairwise fusion unit configured to fuse the sixth fusion feature map a seventh fusion feature map by fusing the feature maps of the second type of residual block an eighth fusion feature map by fusing the feature maps of the third type of residual block a ninth fusion feature map by fusing the feature maps of the third type of residual block a feature of fusing four resolution feature maps by fusing the feature maps of the fourth type of residual block as the output feature of the HRNet multi-scale subnetwork.

5. The attention and multi-scale dual mechanism based bird posture monitoring device of claim 4, wherein, The pairwise fusion operation of the first pairwise fusion unit is repeated once, and each time the pairwise fusion operation specifically includes: The third feature map The fourth feature map is added, and the fourth feature map The third feature map is added, and the first fusion feature map and the second fusion feature map 6. The attention and multi-scale dual mechanism based bird posture monitoring device of claim 4, wherein, The pairwise fusion operation of the second pairwise fusion unit is repeated four times, and each time the pairwise fusion operation specifically includes: The first fusion feature map is obtained The feature map formed by the second type of residual block, the second fusion feature map The feature map formed by the third type of residual block, the fourth feature map The feature map with the highest resolution and the feature map with the lowest resolution among the feature maps formed by the fourth type of residual block, and the feature map with the highest resolution is added to the feature map obtained by the convolution layer with the convolution kernel size of 3 and the step size of 2, to obtain the third fusion feature map The fourth fusion feature map And the fifth fusion feature map 7. The attention and multi-scale dual mechanism based bird posture monitoring device of claim 4, wherein, The pairwise fusion operation of the third pairwise fusion unit is repeated twice, and each time the pairwise fusion operation specifically includes: The third fusion feature map is obtained first The feature map formed by the second type of residual block, the fourth fusion feature map The feature map formed by the third type of residual block, the fifth fusion feature map The feature map formed by the fourth type of residual block, the fifth feature map The highest resolution feature map and the lowest resolution feature map in the feature map formed by the first type of residual block, and the highest resolution feature map is added to the feature map obtained by the convolution layer with the convolution kernel size of 3 and the step of 2, to obtain the sixth fusion feature map The seventh fusion feature map The eighth fusion feature map The ninth fusion feature map 8. The attention and multi-scale dual mechanism based bird posture monitoring device of claim 4, wherein, The pairwise fusion operation of the fourth pairwise fusion unit is repeated twice, and each time the pairwise fusion operation specifically includes: The seventh fusion feature map is first obtained by adding the first feature map and the second feature map The eighth fusion feature map is obtained by adding the third feature map and the fourth feature map The ninth fusion feature map is obtained by adding the fifth feature map and the sixth feature map The feature map obtained by the convolution layer with the convolution kernel size of 3 and the step of 2 in cooperation with the up-sampling operation is added to the sixth fusion feature map to obtain the bird pose feature x of different scales hrnet .

9. The attention and multi-scale dual mechanism based bird posture monitoring device of any one of claims 2-8, wherein, The feature fusion unit specifically includes: a channel dimension splicing unit for splicing attention features x extracted by the ViT attention subnetwork vit and bird posture features x of different scales hrnet In the channel dimension splicing, the fusion features of the fusion of attention and multi-scale double mechanisms are obtained a fully connected layer with c output channels for receiving the fused features x fuse , into a heat map that is the final output of the model 10. The attention and multi-scale dual mechanism based bird posture monitoring device of claim 9, wherein, Also included is a network parameter optimization model, whose loss function Loss MSE is represented as shown in the following formula (1): where n is the number of skeleton joints, and the skeleton joints are in order head mid, left eye, right eye, mouth front top, mouth back left, mouth back right, mouth front bottom, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, body mid back, left hip, right hip, left knee, right knee, left ankle, right ankle, tail top, tail mid, tail back, W H H H are the width and height pixel values of the heat map, respectively, is the method prediction output heat map, is the ground truth heat map, W i is the weight of the ith key point.

Images