Pig skeleton extraction method and evaluation method based on coding and decoding network

Abstract

The application discloses a pig skeleton extraction method and evaluation method based on a coding and decoding network, and comprises the following steps: a coding and decoding network structure is built based on the combination of a ResNet50 residual network and a U-Net semantic segmentation network, a gate module of an attention mechanism is introduced at the same time to enhance feature extraction results, and a key point heat map H containing position information is generated; while the key point heat map H is generated, a key point offset is predicted, a key point offset feature matrix and the output key point heat map H are aggregated by using a Hough voting mechanism, the accurately positioned key point position is remapped to an original image, and a pig skeleton is extracted after the key points are connected. The application has the beneficial effects that the pig skeleton can be extracted in real time under the condition that the illumination condition is unstable and the pig moves violently, the average root mean square error is 23.43 pixels for a video with an input resolution of 1280x720, and the skeleton extraction accuracy is 85.27%.

Description

Technical Field

[0001] This invention relates to the technical field of animal skeleton extraction, and in particular to a method for extracting pig skeletons based on encoding and decoding networks and an evaluation method for extracting pig skeletons based on encoding and decoding networks. Background Technology

[0002] Skeletal information can accurately reflect the posture of newborn pigs and the changes in various parts of the pig's body during movement, providing an important reference for pig movement tracking, gait recognition, and behavior detection. Therefore, the automatic identification and extraction of pig skeletons in complex actual pigsty environments has profound significance for improving the economic benefits of farms and pig welfare.

[0003] Current methods for extracting animal skeletons primarily employ model-based and morphological approaches. In recent years, deep learning technology has made significant strides in skeleton extraction and keypoint detection, particularly in human pose estimation and skeleton extraction. Numerous studies have been conducted on deep learning-based extraction of human or animal skeletons, achieving promising results. However, research related to pig skeleton extraction is scarce. The poor lighting conditions in actual pigsties, making it difficult to distinguish the pig from the background in images, significantly complicates pig skeleton extraction. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by this invention is to propose a method for extracting pig skeletons based on encoding and decoding networks, which can accurately and quickly extract pig skeletons in actual pigsty scenarios, laying the foundation for subsequent pig posture recognition and behavior analysis.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a method for extracting a pig skeleton based on an encoding / decoding network, comprising the following steps: collecting various pig posture data and constructing an original dataset; after augmenting the original dataset, dividing it into a training set and a test set; establishing a skeleton model and manually annotating key points on the skeleton; constructing an encoding / decoding network structure based on a combination of a ResNet50 residual network and a U-Net semantic segmentation network, and simultaneously introducing a gating module with an attention mechanism to enhance the feature extraction results; sequentially passing the enhanced feature extraction results through key point classification and Gaussian filtering to generate a key point heatmap H containing location information; while generating the key point heatmap H, predicting the key point offsets, and using a Hough voting mechanism to aggregate the key point offset feature matrix and the output key point heatmap H to obtain the precisely located key point positions, then remapping them onto the original image, connecting the key points, and extracting the pig skeleton.

[0008] Preferably, the skeletal model includes the following steps: establishing a pig skeletal model based on the anatomical definition of pig joints; dividing the skeletal model into an upper body region and a lower body region, with a total of 14 key points; defining 6 points in the upper body region, including the head and nose in the head region and the tail, hips, spine center point, and shoulders in the main body region; and dividing the lower body region into 8 points, representing the elbow and hoof joints of the limbs; and defining the connection relationships between the key skeletal points.

[0009] Preferably, the encoding / decoding network structure includes an encoding unit and a decoding unit; the encoding unit retains the main feature extraction components of ResNet50 and removes the average pooling layer and fully connected layer for classification; the decoding unit defines three decoding blocks, each of which receives the fused feature matrix Merge, performs dimensionality reduction through convolution operations with strides of 1 and 3*3 to obtain the feature matrix FC, and then passes it through the sigmoid activation function to obtain a keypoint classification feature map Y with a size of 1 / 4 of the original image and k channels.

[0010] Preferably, the introduction of the attention mechanism includes the following steps: combining the gating module of the attention mechanism, applying attention to the skip connections used for feature fusion; and redistributing the weights of different regions of the shallow features by guiding the shallow features with deep features, assigning high weights to key point features and low weights to background features.

[0011] Preferably, the keypoint heatmap H includes the following generation steps: the number of channels and the size of the keypoint classification feature map Y and the keypoint heatmap H are consistent; the confidence level of each pixel position on the keypoint heatmap H is used to represent the response to the keypoint, and the higher the confidence level, the greater the response to the keypoint; the positions of different keypoints on the heatmap are determined, and the peak point (Y) on the corresponding channel of the keypoint classification feature map Y is traversed and searched. max (k)), as shown in equation (1):

[0012] Y max (k)=Max(P k (Y i,j )) (1)

[0013] In equation (1), k = 1, 2, ... 14 are the key point numbers, P k Y represents the confidence level. i,j This represents a pixel on the keypoint classification feature map Y; a circular region S with a radius of R pixels and centered at the peak point is generated on the heatmap. k Set the center point H of each circular region center The confidence level is set to 1, and the generated circular regions are smoothed by Gaussian filtering to achieve gradual attenuation from the region to the center, resulting in the heat map H of the key points containing K circular regions, as shown in equations (2) and (3) below:

[0014] S k (H i,j )={||H i,j -Y max (k)||≤R,R=4} (2)

[0015]

[0016] In equation (2), H i,j This represents a pixel on the heatmap. In equation (3), F represents the Gaussian filter function, sigma is set to 4, except for S k The value is 0 for other locations outside the designated area.

[0017] Preferably, the prediction of keypoint offset includes the following steps: regressing the keypoint offset using the Block 4 convolutional block of the encoding unit to predict the keypoint offset feature matrix O; the size of the keypoint offset feature matrix O is related to the thermal... Figure 1 In this matrix, each element represents the horizontal and vertical offset of the corresponding pixel on the heatmap from its nearest key point. Any point F within the key point region of the pig is labeled, satisfying the conditions as shown in equations (4) and (5).

[0018] O[F,k]=G k -F*S (4)

[0019]

[0020] In equations (4) and (5), F represents any pixel on the heatmap, k = 1, 2, ..., 14 is the key point number, and G... k This represents the true value of the k-th key point location, S represents the scaling ratio of the original image to the heatmap size, set to 4, and T represents the distance threshold, set to 17 pixels.

[0021] Preferably, the aggregation of the keypoint heatmap H includes the following steps: using the Hough voting mechanism to aggregate the keypoint offset feature matrix O and the heatmap; after the keypoint heatmap H is generated, its corresponding keypoint confidence P k It has been determined that the circular region S k Each candidate position h within k The voting weight of (i,j) depends on the offset. k (s,t) and the distance between the current position and other positions within the region are given by the following formula (6):

[0022]

[0023] In equation (6), (s,t) represents other pixels in the candidate region, G(·) represents the bilinear interpolation kernel, and N represents the circular region S. k All pixels within the range, where H represents the heatmap.

[0024] Preferably, during network training, heatmap generation and keypoint offset regression employ the sigmoid cross-entropy classification loss function and HuberLoss, respectively, and the two are added together with certain weights to obtain the joint loss function. total The following equations are given respectively: (7), (8) and (9):

[0025]

[0026] loss total =loss Y weight Y +loss O weight O (9)

[0027] In equation (7), a represents the probability that the keypoint is predicted as positive, y represents the label value of the keypoint (1 for positive examples and 0 for negative examples); in equation (8), y represents the true value of the keypoint location, f(x) represents the predicted value, and δ is the parameter of HuberLoss and is set to 1; in equation (9), weight Y The weights represent the weights of the loss function used to generate the heatmap.O This represents the weights of the keypoint offset loss function.

[0028] Preferably, when the weight is 0.05, the root mean square error is minimized and the network model performs optimally.

[0029] An evaluation method for extracting pig skeletons based on an encoding / decoding network includes the above extraction method and the following steps: the training process is accelerated using CUDA, the model is iterated 80,000 times, the batch size is 1, the first-order Adam optimizer is used for optimization, and the learning rate adopts a segmented strategy, as shown in the following formula (10):

[0030]

[0031] The average processing time and PCK for each image frame are calculated using Euclidean pixel distance, a pig skeleton extraction algorithm based on an encoding / decoding network, and the following Equation 8 as evaluation metrics for pig skeleton extraction:

[0032]

[0033] In equation (11), i represents the key point number, k represents the k-th threshold, and d i d represents the Euclidean distance between the predicted and actual values ​​of the i-th keypoint. def The scale factor is the Euclidean distance from the pig's head to its tail, T. k The threshold is set manually; the mainstream evaluation standard PCK@0.3 is adopted, which represents the comparison between the true value and the predicted value when T=0.3.

[0034] The beneficial effects of this invention are: it can extract the skeleton of a newborn pig in real time under unstable lighting conditions and intense pig movement. For a video with an input resolution of 1280x720, the average root mean square error is 23.43 pixels and the skeleton extraction accuracy is 85.27%. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0036] Figure 1 This is a schematic diagram illustrating the key points of pigs as described in this invention;

[0037] Figure 2 This is a schematic diagram of the key point heatmap generation model based on the encoding / decoding network described in this invention;

[0038] Figure 3 This is a schematic diagram of the key point offset optimization and skeleton extraction described in this invention;

[0039] Figure 4 This is a schematic diagram illustrating the key point offset optimization effect analysis described in this invention;

[0040] Figure 5 This is a schematic diagram illustrating the relationship between the loss value and the weights of the loss function as described in this invention;

[0041] Figure 6 This is a schematic diagram illustrating the relationship between the root mean square error and the weights of the loss function as described in this invention;

[0042] Figure 7 This is a schematic diagram of the pig skeleton extraction results described in this invention. Detailed Implementation

[0043] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0044] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0046] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0047] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0048] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0049] Example 1

[0050] Pig skeleton extraction is fundamental for estimating pig posture and movement analysis. However, unstable lighting conditions in pigsties and vigorous pig movement lead to low extraction accuracy and long processing times. To address these issues, this embodiment proposes a pig skeleton extraction method based on an encoder-decoder network.

[0051] Specifically, the process involves first dividing the pig into 14 skeletal key points based on anatomical principles and skeletal features, and then establishing a skeletal model based on these key points. Next, key points are manually annotated in the collected pig videos. A ResNet50 residual network and a U-Net semantic segmentation network are combined to construct an encoder-decoder network structure, and an attention mechanism is introduced to improve feature extraction results. While generating a keypoint heatmap, the keypoint offset is predicted, and the two are weighted and aggregated using a Hough voting mechanism to achieve accurate skeleton extraction. Experimental results show that the skeleton extraction accuracy is 85.27% when the PCK (Percentage of Correct Keypoints) threshold is equal to 0.3. Compared to the ResNet50 residual network, with a similar average processing time, the error is reduced by 38.21 pixels, and the skeleton extraction accuracy is improved by 22.67%. Furthermore, the pig skeleton extraction method proposed in this embodiment can provide a technical reference for further research on pig behavior.

[0052] Furthermore, this embodiment will be specifically described in three stages: data acquisition, pig skeleton annotation, and pig skeleton extraction algorithm. As follows:

[0053] 1. Data Acquisition Phase

[0054] 1.1 Video Acquisition and Preprocessing

[0055] In this embodiment, data was collected using the camera of an iPhone SE2 at Zhejiang Qingzhu Agricultural and Animal Husbandry Co., Ltd. Thirty 120-day-old Large White pigs were selected as the experimental subjects, and a total of 50 individual pig videos, ranging from 10 to 60 seconds in length, were collected at a frame rate of 30 frames per second and a resolution of 1280x720 pixels.

[0056] To reduce the significant amount of repetitive and redundant work generated by annotating similar frames in videos, the K-Means clustering algorithm was used when manually annotating key points of the pig skeleton. This algorithm groups similar frames in the video into clusters, and then selects one frame from each cluster as the keyframe image, thus constructing an original dataset of 603 images representing various pig poses. Simultaneously, the Python Imague data augmentation library was used to perform operations such as horizontal flipping, rotation cropping, perspective transformation, and blur filtering to augment the pig dataset (1765 images). The augmented pig dataset was then divided into training and testing sets at a 4:1 ratio.

[0057] 1.2 Labeling of pig skeletons

[0058] The pig dataset contains skeletal annotation information. This embodiment primarily establishes a pig skeletal model based on the anatomical definition of pig joints, and manually annotates key points on the skeleton, such as... Figure 1 As shown, the skeletal model is divided into an upper body region and a lower body region, with a total of 14 key points. The upper body region has 6 points, including the head area (head, nose) and the main body area (tail, hips, center point of the spine, shoulders), while the lower body region has 8 points, representing the elbow and hoof joints of the limbs. The connection relationships between the skeletal key points and their corresponding abbreviations are shown in Table 1 below.

[0059] Table 1: Skeleton keypoint connection relationship

[0060]

[0061] 2. Pig skeleton extraction algorithm

[0062] The pig skeleton extraction algorithm consists of two parts: keypoint heatmap generation and skeleton extraction. First, the algorithm extracts pig features using an encoder-decoder network, generating a Gaussian response heatmap containing keypoint location information. To improve the localization accuracy of keypoints, the keypoint offset is optimized by calculating the keypoint offset feature matrix and then weighted and aggregated with the keypoint heatmap using a Hough voting mechanism to obtain the corrected keypoints. These corrected keypoints are then mapped back to the original image and connected to extract the pig skeleton.

[0063] 2.1 Key Point Heatmap Generation Model Based on Encoding / Decoding Network

[0064] Pigs are relatively large and have similar body colors, making them more sensitive to scale information during feature extraction. This embodiment uses a ResNet50 deep transfer learning model pre-trained on the ImageNet dataset as the core algorithm, integrating the design principles of the U-Net semantic segmentation network to propose a keypoint heatmap generation model based on an encoder-decoder network. Its framework is as follows: Figure 2 As shown. Figure 2 In the encoding unit, Blocks 1, 2, 3, and 4 have dimensions that are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively, with 256, 512, 1024, and 2048 channels. The attention gating module A receives feature vectors from both the encoding and decoding modules and obtains attention coefficients through feature concatenation (add), thus redistributing weights. In the decoding unit, Merge1, Merge2, and Merge3 have the same dimensions and number of channels as Blocks 1, 2, and 3, respectively.

[0065] The model consists of an encoding / decoding structure (encoding unit and decoding unit) and a gating module A incorporating an attention mechanism. Through the encoding / decoding structure, the model effectively fuses semantic features at different scales, reducing image detail loss and achieving high accuracy. Simultaneously, during the multi-scale semantic information fusion process, a gating module incorporating an attention mechanism is introduced, using higher-order features to guide lower-order features, enhancing the feature extraction results for keypoints. The feature extraction results are then processed through keypoint classification and Gaussian filtering to finally generate a keypoint heatmap containing location information.

[0066] The feature encoding unit retains the main feature extraction components of ResNet50, removing the average pooling layer and fully connected layer for classification. The feature decoding unit is designed with three decoding blocks to fully utilize features of different orders. Each decoding block receives the fused feature matrix Merge, performs dimensionality reduction through convolution operations with strides of 1 and 3*3 to obtain the feature matrix FC, and finally passes it through the sigmoid activation function to obtain the keypoint classification feature map Y with a size of 1 / 4 of the original image and k channels (i.e., the number of keypoints).

[0067] Unlike traditional attention modules that apply attention to the input feature channels, the gating module A, which incorporates an attention mechanism, applies attention to the skip connections used for feature fusion. Based on this, it enables deep features to guide shallow features, redistributes weights to different regions of shallow features, assigns high weights to key point features and low weights to background features, and further enhances the extraction effect of key point features.

[0068] The number of channels and size of the keypoint heatmap H are consistent with those of the classification feature map Y. The confidence score of each pixel location in this map represents the degree of response to the keypoint; the higher the confidence score, the greater the response to the keypoint. Determining the locations of different keypoints on the heatmap requires iterating through the corresponding channels of feature map Y to find the peak points (Y0, Y1, Y2). max (k)), see equation (1) below:

[0069] Y max (k)=Max(P k (Y i,j )) (1)

[0070] And generate a circular region S with a radius of R pixels centered at this point on the heatmap. k Set the center point H of each circular region center The confidence level was set to 1, and Gaussian filtering was applied.

[22] The region is smoothed to achieve a gradual decay from the region to the center. Finally, a key point heatmap containing K circular regions is obtained, as shown in equations (2) and (3):

[0071] S k (H i,j )={|H i,j -Y max (k)|≤R,R=4} (2)

[0072]

[0073] In equation (1), k = 1, 2, ... 14 are the key point numbers, P k Y represents the confidence level. i,j H represents a pixel on feature map Y; in equation (2), H i,j This represents a pixel on the heatmap; in equation (3), F represents the Gaussian filter function, sigma is set to 4, except for S k The value is 0 for other locations outside the designated area.

[0074] 2.2 Keypoint Offset Optimization and Skeleton Extraction

[0075] To reduce the offset error caused by reduced resolution during heatmap generation, the keypoint offsets are regressed using the Block 4 convolutional block of the encoding unit in the shared heatmap generation model. This predicts the keypoint offset feature matrix O. Then, the Hough voting mechanism is used to aggregate the keypoint offset feature matrix and the output heatmap to obtain a 2-D Hough activation map that accurately locates the keypoints. Figure 3 As shown in the figure. Based on this, the key point positions obtained from the 2D-hough activation map are remapped back to the original image, and the pig skeleton is extracted after connecting the key points.

[0076] The size and thermal properties of the keypoint offset feature matrix Figure 1 In this matrix, each element represents the horizontal and vertical offset of the corresponding pixel on the heatmap from its nearest key point. In practice, only any point F within the key point region of the pig is labeled, thus satisfying conditions (Equations 4 and 5).

[0077] O[F,k]=G k -F*S (4)

[0078]

[0079] In equations (4) and (5), F represents any pixel on the heatmap, k = 1, 2, ..., 14 is the key point number, and G... k This represents the true value of the k-th key point location, S represents the scaling ratio of the original image to the heatmap size, set to 4, and T represents the distance threshold, set to 17 pixels.

[0080] This embodiment employs the Hough voting mechanism to aggregate the keypoint offset feature matrix and heatmap. Hough voting is an algorithm based on the generalized Hough transform, which uses local features to vote on the target center and estimates the center's location by finding the maximum value in the voting space.

[0081] After the key point heatmap is generated, the corresponding key point confidence level P is... k It has been determined that the circular region S k Each candidate position h within k The voting weight of (i,j) depends primarily on the offset. k (s,t) and the distances between the current location and other locations within the region are shown in Equation (6):

[0082]

[0083] In equation (6), (s,t) represents other pixels in the candidate region, G(·) represents the bilinear interpolation kernel, and N represents the circular region S. k All pixels within the range, where H represents the heatmap.

[0084] Example 2

[0085] This embodiment proposes a method for extracting and evaluating pig skeletons based on an encoding / decoding network. The results and analysis of the above embodiment's method for extracting pig skeletons based on an encoding / decoding network are presented as follows:

[0086] All experiments were conducted on a server with an operating system of Ubuntu 18.04, a quad-core Intel(R) Xeon(R) CPU @ 2.30GHz, and an NVIDIA Tesla P100 graphics card. The training process was accelerated using CUDA, with a total of 80,000 iterations and a batch size of 1. The first-order Adam optimizer was used for optimization. A segmented learning rate strategy was adopted, as shown in (7):

[0087]

[0088] Euclidean pixel distance, the average time taken for the algorithm to process each image frame, and PCK (percentage of correct key points, Equation 8), which is widely used in human pose estimation, were used as evaluation metrics for pig skeleton extraction.

[0089]

[0090] In equation (8), i represents the key point number, k represents the k-th threshold, and d i d represents the Euclidean distance between the predicted value and the actual value of the i-th keypoint. def T represents the scale factor, which in this paper is the Euclidean distance from the pig's head to its tail. k This is a manually set threshold; the smaller the value, the stricter the evaluation standard. δ(·) represents 1 when the condition is met, and 0 otherwise. This embodiment adopts the mainstream evaluation standard PCK@0.3, which represents the comparison between the true value and the predicted value when T=0.3.

[0091] Experimental results show that when the PCK (Percentage of Correct Keypoints) threshold is equal to 0.3, the skeleton extraction accuracy is 85.27%. Compared with the ResNet50 residual network, with a similar average processing time, the error is reduced by 38.21 pixels, and the skeleton extraction accuracy is improved by 22.67%.

[0092] Example 3

[0093] To verify the extraction performance of the model of this invention, it was compared with three basic network models, ResNet50, ResNet101, and Res-UNet, under the same experimental conditions. The effects of increasing network depth (ResNet101), fusing with U-Net semantic segmentation network (Res-UNet), and incorporating a gating module with an attention mechanism (AttentionRes-UNet) on model performance were tested. The test results are shown in Table 2 below:

[0094] Table 2: Accuracy analysis of different network models.

[0095]

[0096] As shown in Table 3, when using a 50-layer ResNet network, the root mean square error of the skeleton keypoint prediction is 61.64 pixels, and the skeleton extraction accuracy is 62.60%. However, increasing the ResNet network depth to 101 layers did not significantly improve any of the model's metrics, but the detection time per frame increased by 0.011. This indicates that a network depth of 50 layers is sufficient for the skeleton extraction requirements of this paper, and increasing the number of network layers does not further improve the model's performance.

[0097] For the network fused with U-Net (Res-UNet), although the processing time increased due to the increased number of parameters, its root mean square error decreased by 22 pixels and the skeleton extraction accuracy improved by 14.43% compared to the ResNet50 residual network. This result is because the model, combined with the U-Net network, absorbs features from lower-level convolutional networks, preserving more keypoint details. After introducing a gating module incorporating an attention mechanism, the extraction performance was further improved compared to the encoder-decoder network, with a root mean square error reduction of 16.21 pixels and a skeleton extraction accuracy improvement of 8.24%. Furthermore, for the difficult-to-detect lower body region, the overall error was reduced by 30%. The existing skeleton extraction results demonstrate that, based on the encoder-decoder network structure, the introduction of the attention mechanism effectively emphasizes features belonging to keypoints and further mines semantic information in difficult-to-detect areas, thus improving the network's detection accuracy.

[0098] Example 4

[0099] This embodiment illustrates the effect of keypoint offset optimization. To verify the impact of keypoint offset optimization on reducing prediction error, the model without keypoint offset optimization is compared with the model in this paper. The results are shown in Table 3 below.

[0100] Table 3: Accuracy analysis of key point offset optimization.

[0101]

[0102] As shown in Table 3, although calculating the keypoint offset led to an increase in computational cost, the root mean square error (RMSE) with keypoint offset optimization decreased by 17.6%, and the skeleton extraction accuracy improved by 7.31%. Furthermore, the RMS error decreased by 28.2% when the keypoint confidence was greater than 0.6. This indicates that regression of keypoint offset can effectively correct the position of keypoints, and has a better correction effect for keypoints with higher confidence.

[0103] Reference Figure 4The illustration shows a comparison of the skeleton extraction algorithm performance with and without keypoint offset optimization. It demonstrates that keypoint offset optimization effectively reduces the offset error of predicted keypoints and improves the detection accuracy of skeleton keypoints. In particular, it significantly improves the detection of pig elbows and hooves. This is because the elbows and hooves of pigs are close together, and they are prone to occlusion and merging during walking. Even small offsets can lead to inaccurate keypoint prediction positions and overlap. The keypoint offset optimization method in this embodiment effectively solves the problems caused by these situations.

[0104] Example 5

[0105] This embodiment further illustrates the impact of loss function weights on the extraction results. During the network training process of the pig skeleton extraction method, the heatmap generation and keypoint offset regression respectively adopt the sigmoid cross-entropy classification loss function (Equation 9) and HuberLoss (Equation 10), and the two are added together with certain weights to obtain the joint loss function loss. total (Equation 11):

[0106]

[0107] loss total =loss Y weight Y +loss O weight O (11)

[0108] In equation (9), a represents the probability that the keypoint is predicted as positive, y represents the label value of the keypoint (1 for positive examples and 0 for negative examples). In equation (10), y represents the ground truth value of the keypoint location, f(x) represents the predicted value, and δ is the parameter of HuberLoss, set to 1. In equation (11), weight Y The weights represent the weights of the loss function used to generate the heatmap. O The weights of the keypoint offset loss function are represented. To verify the impact of different weights of the two loss functions on skeleton extraction performance, experiments were conducted with different keypoint offset loss function weights set when the weight of the heatmap generation loss function was 1. The relationship between the loss function weights and the changes in loss value and root mean square error is shown in the figure. Figure 5 and Figure 6 As shown.

[0109] Depend on Figure 5It can be seen that the three loss functions with different weights all decrease to low values ​​in a short period of time at the beginning of training. During the decrease, the loss functions experience corresponding jumps due to changes in the learning rate gradient, but eventually tend to stabilize. This indicates that the hyperparameters of the model are set reasonably, the model converges, and the loss value is positively correlated with the weights of the keypoint offset loss function; the larger the weight, the larger the loss value. Figure 6 It can be seen that when the weight is 0.05, the root mean square error is the smallest and the network model performs best. However, when the weight is 0.03, although the loss value is the smallest, the root mean square error fluctuates more drastically, and the model accuracy is reduced. This may be because the classification loss function generated by the heatmap is overemphasized, resulting in a different convergence rate than the keypoint offset regression loss function. At the same time, when the weight is too large, i.e., 0.1, the error will increase because the heatmap generation loss function does not converge well, and the model performance will also be poor.

[0110] Example 6

[0111] This embodiment describes the results and analysis of pig skeleton extraction. Five short video clips (5-10 seconds each) containing different behaviors such as walking, running, standing, crouching, and turning were selected from the collected video as test subjects. One frame was extracted every second (30 frames) to obtain the results. Figure 7 The image sequence shown verifies the extraction results of the extraction method of the present invention in a real-world scenario.

[0112] As shown in the figure, this method can effectively detect the skeletal structure and key point information of pigs in different postures under different lighting and viewing angles. For behaviors such as walking and crouching, where key points of the limbs are not easily observed, this method does not produce false detections of invisible key points. When pigs perform movements with large body amplitude changes, such as running and turning, this method can still extract the corresponding key point areas well.

[0113] This invention proposes a method for extracting pig skeletons based on an encoder-decoder network. This method is based on the ResNet50 residual network and incorporates the design principles of the U-Net semantic segmentation network, constructing an encoder-decoder network structure. Attention mechanisms and transfer learning methods are integrated to further improve network performance. To improve the detection accuracy of key points on the pig skeleton, the key point offset caused during heatmap generation is optimized. A Hough voting mechanism and a key point offset matrix are used to vote on candidate key point regions on the heatmap, and the region with the highest number of votes is identified as the key point. Finally, the key point regions on the heatmap are mapped back to the original image and concatenated to extract the pig skeleton.

[0114] Experimental results show that the method of the present invention can extract the skeleton of a pig in real time under unstable lighting conditions and intense pig movement. For a video with an input resolution of 1280x720, the average root mean square error is 23.43 pixels and the skeleton extraction accuracy is 85.27%.

[0115] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can be implemented using standard programming techniques—including a non-transitory computer-readable storage medium configured with a computer program, wherein such a storage medium causes the computer to operate in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0116] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program comprises a plurality of instructions executable by one or more processors.

[0117] Furthermore, the method can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described herein, the invention also includes the computer itself. A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on a display.

[0118] As used herein, the terms “component,” “module,” “system,” etc., are intended to refer to a computer-related entity, which may be hardware, firmware, a combination of hardware and software, software, or running software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, a running thread, a program, and / or a computer. As an example, an application running on a computing device and the computing device itself can both be components. One or more components may reside in a running process and / or thread, and components may be located in a single computer and / or distributed among two or more computers. Furthermore, these components are capable of execution from various computer-readable media having various data structures thereon. These components may communicate locally and / or remotely via signals, such as based on one or more data packets (e.g., data from a component that interacts with a local system, another component in a distributed system, and / or signals that interact with other systems via a network such as the Internet).

[0119] It should also be understood that the present invention is described through embodiments, and the embodiments are only clear and complete descriptions of the technical solutions proposed in the claims of the present invention, that is, explanations of the claims. Therefore, when judging whether the technical solutions recorded in the specification of the present invention are sufficiently disclosed, the purpose and core essence of the solutions defined by the claims should be fully considered. There are necessarily other technical problems in the specification that are unrelated to the core technical problem solved by this embodiment. The corresponding technical features and technical solutions are not included in the essence of this embodiment and are non-essential technical features. Therefore, the implicit disclosure can be referred to. Those skilled in the art can fully implement them by combining existing technology and common knowledge. Therefore, there is no need to describe them in detail.

[0120] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for pig skeleton extraction based on a coding-decoding network, characterized in that: The method comprises the following steps, Collecting live pig posture data and constructing an original data set; The original data set is divided into a training set and a test set after data set augmentation; Establishing a skeleton model and manually labeling key points on the skeleton; A Res-UNet structure is built based on the combination of a ResNet50 residual network and a U-Net semantic segmentation network, and a gating module of an attention mechanism is introduced to enhance the feature extraction result; The coding-decoding network structure includes an encoding unit and a decoding unit, the encoding unit retains the feature extraction main body of ResNet50, and removes the classification average pooling layer and the full connection layer, the decoding unit defines three decoding blocks, each decoding block accepts a fusion feature matrix Merge, performs a convolution operation with a step of 1 and a size of 3*3 to reduce the dimension, obtains a feature matrix FC, and then passes through a sigmoid activation function to obtain a key point classification feature map Y with a size of 1 / 4 of the original picture and a channel number k; The attention mechanism includes the following steps: combining the gating module A of the attention mechanism, applying attention to the skip connection used for feature fusion; through the guidance of deep features to shallow features, the weights of different regions of the shallow features are redistributed, and high weights are given to key point features and low weights are given to background features; The enhanced feature extraction result is sequentially subjected to key point classification and Gaussian filtering to generate a key point heat map H containing position information, the key point heat map H includes the following generation steps: the channel number and size of the key point classification feature map Y and the key point heat map H remain the same; the confidence of each pixel position on the key point heat map H represents the response degree of the key point, and the higher the confidence, the greater the response to the key point; The key point heat map H is generated while predicting the key point offset, and the key point offset feature matrix and the output key point heat map H are aggregated by using the Hough voting mechanism to obtain the accurately positioned key point position, which is then remapped to the original picture, and the key points are connected to extract the live pig skeleton; The prediction of the key point offset includes the following steps, The offset of the key point is regressed by the convolution block Block4 of the encoding unit to predict the key point offset feature matrix O; The size of the key point offset feature matrix O is consistent with the heat map, and the value of each element in the matrix is the offset of the corresponding pixel point on the heat map to the nearest key point in the horizontal and vertical directions; Any point F belonging to the region of each key point of the live pig is labeled, that is, the conditions are met, such as formulas (4) and (5): O[F, k] = G k - F * S (4) In formula (4) and formula (5), F represents an arbitrary pixel point on a heat map, k = 1, 2,..., 14 is a key point serial number, G k represents a true value of the kth key point position, S represents a scale ratio of the original image to the heat map size, which is set to 4, and T represents a distance threshold value, which is set to 17 pixels. The skeleton model includes the following establishment steps, The skeleton model of the pig is established according to the definition of the pig joints in anatomy; The skeleton model is divided into an upper body region and a lower body region, a total of 14 key points, the upper body region defines 6 points, including the head, nose and tail of the main body region, hip, spine center point and shoulder, and the lower body region is divided into 8 points, representing the elbow and hoof joints of the four limbs; The connection relationship between the key points of the skeleton is defined; In the network training process, the heat map generation and the key point offset regression respectively adopt sigmoid cross-entropy classification loss function and HuberLoss, and the two are added with a certain weight to obtain a joint loss function loss total , as follows formula (7), (8) and (9): loss total = loss Y * weight Y + loss O * weight O (9) In formula (7), a represents the probability of key point prediction as a positive class, and y represents the label value of the key point, with 1 for a positive example and 0 for a negative example; in formula (8), y represents the true value of the key point position, f(x) represents the predicted value, and δ is a parameter of HuberLoss and is set to 1; in formula (9), weight Y represents the weight of the heat map generation loss function, weight O represents the weight of the key point offset loss function, and when the weight is 0.05, the root mean square error is the smallest, and the network model is the best.

2. The pig skeleton extraction method based on a codec network according to claim 1, characterized in that: The key point heat map H also includes the following generation steps, determining different key point positions on the heat map, traversing to find a peak point (Y max (k)) as follows: Y max (k) = Max(P k (Y i,j )) (1) k = 1, 2,..14 is the key point serial number, P k represents the confidence, Y i,j represents a certain pixel point on the key point classification feature map Y; A circular region S with a peak point as a center and a radius of R pixels is generated on the heat map k The confidence of each circular region center point H center is set to 1, the generated circular region is smoothed by Gaussian filtering, gradual attenuation of the region to the center is realized, and the key point heat map H containing K circular regions is obtained, as follows (2) and (3): S k (H i,j )={||H i,j -Y max (k)||≤R,R=4} (2) H in formula (2) i,j represents a certain pixel point on the heat map, F in formula (3) represents a Gaussian filter function, sigma is set to 4, and other positions are 0 except S k region.

3. The pig skeleton extraction method based on a codec network according to claim 1, characterized in that: The aggregation of the key point heat map H includes the following steps, The key point offset feature matrix 0 and the heat map are aggregated by using a Hough voting mechanism; After the key point heat map H is generated, the key point confidence P corresponding to the key point heat map H is generated k It has been determined that each candidate position h k Inside the circular region S k The voting weight of (i,j) depends on the offset amount offset k (s,t) and the distance between the current position and other positions in the region, as shown in the following formula (6): In formula (6), (s, t) represents other pixel points of the candidate region, G(·) represents a bilinear interpolation kernel, N represents all pixel points in the circular region S, and H represents a heat map. k In formula (6), (s, t) represents other pixel points of the candidate region, G(·) represents a bilinear interpolation kernel, N represents all pixel points in the circular region S, and H represents a heat map.

4. An evaluation method for extracting a pig skeleton based on a codec network, characterized by: The extraction method according to any one of claims 1-3, further comprising the following steps, The training process uses cuda acceleration, the model is iterated 80000 times in total, the batch size is 1, a first-order adam optimizer is used for optimization, and a segmented strategy is used for the learning rate, as shown in the following formula (10): The average time consumption of each image frame and PCK are used as evaluation indexes of the pig skeleton extraction algorithm based on the encoding and decoding network by using the Euclidean pixel distance, as shown in the following formula (11): In formula (11), i represents the key point serial number, k represents the kth threshold value, d i represents the Euclidean distance between the predicted value and the real value of the ith key point, d def represents the scale factor, i.e. the Euclidean distance from the pig head to the tail, T k is a threshold value set artificially; The mainstream evaluation standard PCK@0.3 is adopted, which represents the comparison of the true value and the predicted value Euclidean distance when T=0.3.

Images