A group pig posture recognition method based on a generative adversarial network

Abstract

The application discloses a group pig posture recognition method based on a generative adversarial network, and comprises the following steps: collecting a group pig initial data set and preprocessing the initial data set; constructing a generative adversarial network model comprising three parts of a generator, a discriminator and a posture estimation module arranged in sequence; the posture estimation module decodes a heat map output by the generator, extracts corresponding key point positions presumed in a graph, and maps the key point positions back to an original graph according to a proportion; and finally, according to the distribution of the key point positions, independent group pig postures are assembled to complete recognition. Experimental results show that the group pig key points can be detected with an mAP of 74.09%, and for the key points that are shielded, the PCK index can reach 82.0% at most.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of group pig posture recognition, and particularly relates to a group pig posture recognition method based on a generative adversarial network. BACKGROUND

[0002] Animal behavior is closely related to its health and welfare, however, manual observation of animal behavior in large-scale farms is not only time-consuming and laborious, but also produces huge labor costs. In recent years, deep convolutional neural networks have made significant progress in human pose estimation. For example, continuous multi-resolution semantic fusion and high-resolution heat map representation are recently used to effectively infer the poses of multiple people, and a deconvolution module is added after the backbone network to predict heat maps, which achieves excellent performance with simple design. These methods mainly follow the strategy of deep neural network regression of human part heat maps, and show good feature representation learning ability.

[0003] There are also human pose estimation methods applied to animals such as chickens, cows, and mice. However, there are few studies on group pigs, and only methods detect key points on the back of pigs, while avoiding the detection of difficult but important elbow and hoof key points of the limbs. This may be because when estimating the posture of group pigs in the actual group pig environment, there are often the following problems: the pig's body shape and color are similar, the limbs have high similarity, and it is difficult to distinguish; the pig house area is small and the group pigs have the characteristics of aggregation, resulting in serious elbow and hoof occlusion; and pigs are quadruped mammals, and the limbs far from the camera are easily occluded by themselves. Occlusion has always been a difficult problem in pose estimation, because the model lacks visual information of the occluded key points and is difficult to detect. SUMMARY

[0004] This section is intended to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, the abstract and the title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0005] In view of the above existing problems, the present application is proposed.

[0006] Therefore, the technical problem solved by the present application is that the aggregation of group pigs causes occlusion, making it difficult to estimate the posture of group pigs.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: a method for group pig pose recognition based on generative adversarial networks (GANs), comprising the following steps: collecting an initial dataset of group pigs and preprocessing the initial dataset; constructing a GAN model comprising a generator, a discriminator, and a pose estimation module arranged sequentially, and training the GAN model using the preprocessed dataset; inputting the data to be recognized into the trained GAN model, whereby the generator outputs a heatmap, the discriminator analyzes the difference between the heatmap and the actual results, learns the geometric topological features of the skeleton, and then infers the occluded key points based on the skeleton's geometric structure; the pose estimation module decodes the heatmap output by the generator, extracts the inferred key point positions from the image, and maps them back to the original image proportionally; finally, based on the distribution of the key point positions, assembling them into independent group pig poses to complete the recognition.

[0008] Preferably, the generator uses HrNetV2 as its backbone network. The network consists of four stages, including the following steps: after receiving an input RGB image, the network goes through an initial stage containing two 3*3 convolutional layers and a pooling layer to generate an image with a scale equal to the original image size. Figure 1 The network consists of a high-resolution subnet of 8x8; each stage gradually increases the resolution of the subnet from high to low through downsampling, and the multi-resolution subnets are connected in parallel. Throughout the process, the parallel multi-resolution subnets extract features through a 3x3 convolutional layer and a residual block, and repeatedly exchange information by adding convolutional layers together; finally, in the output stage, the network upsamples and connects the other low-resolution convolutions as features, and uses a transposed convolutional layer with low information redundancy and high mapping ability to upsample the features to obtain a heatmap for encoding keypoint confidence information and a partial affinity field (PAF) for encoding keypoint connection relationships.

[0009] Preferably, the heatmap size is 1 / 4 of the original image resolution, and the number of channels is k, where k represents the number of key points. The PAF captures the relationship between parts by encoding each joint vector, and its resolution is the same as the heatmap, with k*2 channels representing each joint component on the x and y planes.

[0010] Preferably, the training of the generative adversarial network is an adversarial process between the generator and the discriminator, expressed by the loss functions of equations (1) and (2) below:

[0011]

[0012] L Discriminator =L real (G,X)-gt*L fake (S,X) (2)

[0013] A weight variable gt with interval size [0,1] is introduced in the network, as shown in the following formula (3), which represents the degree of emphasis on L fake when the performance of the generator is better than that of the discriminator (L fake <βL real ) gt will increase, so that the loss function L fake has a higher weight ratio in the training process, so that the discriminator will be trained more in identifying the generated heat map, when the performance of the discriminator is better than that of the generator (L fake >βL real ), gt will decrease to slow down the training speed of the discriminator, so that the generator can catch up with the pace of the discriminator.

[0014] g t+1 =g t +μ(βL real (S,C)-L fake (S,X)) (3)

[0015] In the formula, C represents the generated heat map, X represents the ground truth, S represents the fake heat map, G represents the real heat map, μ and β are hyperparameters, which are set to 0.0001 and 0.5 respectively, and are used to control the corresponding weights.

[0016] Preferably, the pose estimation module comprises the following steps: after obtaining the heat map regenerated through the adversarial game, the first step is to extract the position of the key point corresponding to the pixel point with the peak value of confidence in the heat map; the local highest peak in the heat map is found through non-maximum suppression, and the position is mapped back to the original coordinate space according to the size of the heat map scaling; wherein the DARK method is used to optimize the offset error caused by the change of scale in the position mapping process, and only the first and second derivatives of each pixel position on the heat map are needed to obtain more accurate pixel positions. The specific formula definition in the mapping is as follows formula (4) and (5):

[0017] loc p =stride*(peaks+offsets p )+stride / 2 (4)

[0018] offsets p =(D″(p)) -1 D′(p) (5)

[0019] Wherein loc p represents the position of the predicted key point p on the original image, peaks represents the local highest peak of the key point p on the heat map, offsets prepresents the offset calculated using the DARK method, D' and D" represent the first and second derivatives respectively, stride represents the size of the scaling and is 4.

[0020] Preferably, the pose estimation module includes the following steps for the matching part between the key points and the individuals: using the partial affinity field obtained from the generator training, calculating the PAF score between each pair of key points by line integral to obtain the cost matrix of each pair of key points, and obtaining the best match of each key point according to the Hungarian algorithm; taking the minimum data-driven graph as the maximum spanning tree, searching the graph after network evaluation, gradually expanding the minimum skeleton with strong recognition edges by the order of the ability size of separating two distributions, forming up to 9 increasingly redundant graphs; selecting the graph that maximizes the purity and the score of connecting key points together as the animal assembly graph; and finally searching the connected components in the graph to find the nodes belonging to a certain individual to obtain the final pose of the pig group.

[0021] Preferably, for each training image, the generative adversarial network model predicts output through end-to-end training, and after the training is completed, the discriminator part will be removed, and the output of the converged generator is sent to the pose estimation module.

[0022] Preferably, the preprocessing of the initial data set includes the following steps: using the K-MEANS clustering method to classify similar frames in the collected data into a cluster, and in turn selecting a frame from each cluster as a key frame image to construct the initial data set of various poses of live pigs; labeling the key points of each pig in the initial data set, including distinguishing the left and right elbows and hooves, a total of 14 key points, and dividing the labeled key points into two categories: visible key points and occluded key points; and dividing the training set and the test set according to the ratio of 8:2.

[0023] Preferably, the generative adversarial network model can detect the key points of the pig group with an mAP of 74.09%, and for the occluded key points, the PCK index can reach up to 82.0%.

[0024] The application has the beneficial effects that the generative adversarial network model is proposed to solve the pig posture estimation problem, and has stronger robustness for pig body occlusion, overlap and distortion, is composed of a generator and a discriminator with similar structures, means a zero-sum game between them, the generator adopts HRNET as a skeleton network, is responsible for predicting a heat map of pig body key points based on image features, and the discriminator adopts a simpler network structure, can distinguish unreasonable postures, and provides useful prompts for the generator to improve the heat map; in the final inference link, the DARK method is used to improve the error loss in the heat map decoding process, and the PAF partial affinity field and the minimum spanning tree method are used to assemble the key points, and the experimental results show that the mAP of the key points of the group pigs can reach 74.09%, and for the key points that are occluded, the PCK index can reach 82.0% at most. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0026] Figure 1 The schematic diagram of the actual pig house scene and the camera position distribution described in the application;

[0027] Figure 2 The schematic diagram of the key point distribution described in the application;

[0028] Figure 3 The schematic diagram of the key point annotation example described in the application;

[0029] Figure 4 The schematic diagram of the key point annotation when mutual occlusion occurs described in the application;

[0030] Figure 5 The schematic diagram of the key point annotation when self-occlusion occurs described in the application;

[0031] Figure 6 The schematic diagram of the overall key point distribution described in the application;

[0032] Figure 7 The schematic diagram of the "elbow" key point distribution described in the application;

[0033] Figure 8 The schematic diagram of the "hoof" key point distribution described in the application;

[0034] Figure 9 The schematic diagram of the posture estimation flow chart described in the application;

[0035] Figure 10 A schematic diagram of the network structure of the generator of the present application;

[0036] Figure 11 A schematic diagram of the generator loss function curve of the present application;

[0037] Figure 12 A schematic diagram of the discriminator loss function curve of the present application;

[0038] Figure 13 A schematic diagram of the highly twisted and staged situation of the present application;

[0039] Figure 14 A schematic diagram of the occlusion situation of the present application;

[0040] Figure 15 A schematic diagram of the severe exposure situation of the present application;

[0041] Figure 16 A schematic diagram of the pig individual blur situation of the present application;

[0042] Figure 17 A schematic diagram of the two body parts being severely occluded of the present application;

[0043] Figure 18 A schematic diagram of some failed results of the challenging scene of the present application. DETAILED DESCRIPTION

[0044] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details presented herein. In other instances, well-known methods have not been described in detail in order to avoid obscuring the present application. Therefore, the specific details set forth hereinafter are merely exemplary of the present application and are not intended to limit the present application for the purpose of utilization. Other embodiments of the present application, beyond those described herein, will be apparent to those skilled in the art from this disclosure.

[0045] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details presented herein. In other instances, well-known methods have not been described in detail in order to avoid obscuring the present application. Therefore, the specific details set forth hereinafter are merely exemplary of the present application and are not intended to limit the present application for the purpose of utilization. Other embodiments of the present application, beyond those described herein, will be apparent to those skilled in the art from this disclosure.

[0046] Secondly, the "one embodiment" or "embodiment" referred to herein can include specific features, structures or characteristics in at least one implementation of the present application. The "in one embodiment" appearing in different places in the specification does not all refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments.

[0047] The application is described in detail in conjunction with the schematic diagram. In the detailed description of the embodiments of the application, the cross-sectional view of the device structure is partially enlarged without the general proportion for the convenience of illustration, and the schematic diagram is only an example, which should not limit the scope of protection of the application herein. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in actual production.

[0048] Meanwhile, in the description of the application, it should be noted that the terms "upper, lower, inner and outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore it cannot be understood as a limitation of the application. In addition, the terms "first, second or third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0049] In the application, unless otherwise explicitly specified and limited, the terms "mounting, connection, connection" should be understood broadly, for example: it can be fixed connection, detachable connection or integral connection; it can also be mechanical connection, electrical connection or direct connection, it can also be indirectly connected through intermediate medium, or it can be the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0050] Embodiment 1

[0051] Posture estimation as an effective method of behavior detection has attracted widespread attention in the field of animal health and welfare detection in recent years. By monitoring the individual behavior of pigs over time, it will help to detect diseases early, allowing earlier and more effective intervention. However, posture estimation of group pigs is a challenging computer vision problem, because in the actual farming scene, the aggregation of group pigs will cause occlusion, making it difficult for the model to recognize due to the lack of sufficient semantics. This embodiment proposes a group pig posture estimation model based on generative adversarial network. The generator and discriminator in the model promote the learning of prior information about the structure of the pig body in an adversarial game, so as to output a reasonable posture of the skeleton structure. In addition, a new pig posture estimation dataset is established, which not only contains basic back key points, but also contains small key points such as elbows, hooves and tails, as well as challenging occluded key point annotations.

[0052] When estimating the posture of pigs in a real group pig environment, the following challenges often exist: the pig's body color is similar, the limbs have high similarity, and it is difficult to distinguish; the pig house area is small and the group pigs have the characteristics of aggregation, resulting in serious elbow and hoof occlusion; pigs are quadruped mammals, and the limbs far from the camera are easily occluded by themselves. Among the above problems, occlusion has always been a difficult problem in pose estimation, because the model will lack visual information of the occluded key points and it is difficult to detect. However, human vision can learn the diversity and infinity of object shape structure from observation. Even under extreme occlusion, the potential posture can be inferred and the untrustworthy one can be excluded. The embodiment proposes an effective generative adversarial network framework to infer the occluded key points in the graph. The intention is to judge whether the predicted pose is geometrically reasonable through the discriminator in the generative adversarial network. If the predicted pose can "deceive" the discriminator and make it believe that its prediction is reasonable, then the network has successfully learned the geometric features of the pig skeleton structure. In addition, HrNetV2 is used as the generator backbone network, and the DARK method is used to optimize the decoding process of the heat map. The problem of similar color and high similarity of limbs can be effectively solved.

[0053] The embodiment proposes a process to achieve this goal, including image acquisition, data set preparation, and construction of the pose estimation model. A model based on a generative adversarial network is designed to deal with the highly occluded scene in the actual pig breeding environment, which is the first time in animal pose estimation. The model learns the implicit geometric structure of the pig body through the generative adversarial network, infers the position of the occluded key points, and is significantly better than other pose estimation methods. In addition, HrNetV2 is used as the skeleton network of the model, and improvements are made on this basis. The decoding part of the heat map is also improved.

[0054] Specifically, the embodiment is explained by including the following parts: data set acquisition, data set preparation (i.e. preprocessing), pose estimation model, the generative adversarial network model learns the implicit geometric structure of the pig body through the generative adversarial network, infers the position of the occluded key points, and is significantly better than other pose estimation methods, as follows:

[0055] (1) Collect the environment.

[0056] The experimental site of the embodiment is Zhejiang Qingzhu Agriculture and Animal Husbandry Co., Ltd. Video data from July to October 2021 is collected as the basis for the research. The test station is mainly responsible for the evaluation and registration of sow traits, and a total of 20 stalls are provided. The actual scene of each stall is as follows Figure 1The size of the pigsty is 3.5m*5m, the height is from 2.2m to 2.8m, and the overall space is trapezoidal. Each pigsty is provided with a window, and the light condition is good during the day, but there may be excessive exposure. In addition to the middle activity area, an automated water trough and feeding device are provided on the left and right sides (areas prone to obstruction), respectively. To facilitate the collection of various postures of the pig body in all directions, we selected two of them and installed four Hikvision Smart256 cameras at the top of each column in four different directions. The device can collect 30 frames / s of video with a resolution of 1920x1280 pixels.

[0057] (2) Dataset.

[0058] In this embodiment, 14 (7 per column) long white pigs of 60-day-old age in the above experimental environment were used as experimental objects, and 200 videos of about 5 minutes long were collected as basic data in the time period of 07:00-17:00. From the basic data, 40 videos of pig groups within the camera field of view and active behavior were selected. The K-MEANS clustering method was used to group similar frames in the video into a cluster, and a frame was selected from each cluster as a key frame image in turn, thereby constructing an original data set of various postures of live pigs, a total of 2000. Figure 2 As shown, each skeleton contains basic key points such as head, nose, tail, shoulder, back, etc., and also includes 14 key points such as elbow and hoof to distinguish left and right. The key point name mapping table is as follows Table 1:

[0059] Table 1: Key point name mapping table.

[0060]

[0061] In particular, this dataset is used to evaluate the key point detection ability of the model under the occlusion condition. For the occluded key point, as long as its approximate position can be guessed, it will still be labeled, as shown in Figures 3-5 The labeled key points are divided into two categories: visible key points and occluded key points.

[0062] In general, the live pig dataset contains 2000 images and 74203 labeled key points, and is divided into training and test sets in a ratio of 8:2. In particular, the dataset contains challenging elbow and hoof key points and key points under different scene occlusions, which can effectively verify the detection effect of the model under occlusion. As shown in Figure 6 The occluded key points account for 12.308% (9133) of the whole, and the visible key points account for 87.692% (65070) of the whole. For the elbow and hoof key points that are prone to occlusion, as shown in Figures 7-8As shown, the proportion of occlusion in the data set reached 20.362% (3989) and 19.975% (3869) respectively.

[0063] (3) Pose estimation model.

[0064] The existing animal pose estimation method has achieved good results, but when performing group pig pose estimation in the actual breeding scene, due to the influence of camera view angle and group pig living habits and other factors, the key points are very easy to be occluded. In view of this problem, the embodiment introduces a generative adversarial network (GAN) to improve the recognition accuracy of the model to the occluded key points. By Figure 9 It can be known that the model is composed of three parts in total, a key point generator, a key point discriminator, and finally a pose estimation part. The key point generator is a high-resolution representation neural network with HrNetV2 as the basic skeleton. After inputting the RGB image, the network will output a heat map with a resolution size of 1 / 4 of the original image and a channel number k (k is the number of key points). Each pixel point on the heat map represents the confidence of each key point appearing at the pixel position, and the interval size is [0, 1]. The network structure of the key point discrimination network D is similar to that of the generation network, but unlike the generator G, the network needs to input the "fake" heat map output by the generator and the real heat map, and its purpose is to analyze the difference between the heat map predicted by the generator and the actual result, learn the geometric topological structure features of the skeleton, and then better predict the occluded key points according to the skeleton geometric structure. The pose estimation part is responsible for decoding the heat map output by the generator, extracting the corresponding key point position, and mapping it back to the original image according to the proportion. Finally, according to the distribution of the key point position, it is assembled into an independent group pig pose.

[0065] Figure 9 Three stages of pig pose estimation are described. Different stages are represented by different background colors. A pose estimation network based on hrnetv2 is used as a generator, and with a discriminator, the input heat map is reconstructed to distinguish between generated "fake" heat maps and real heat maps. Finally, the results obtained by the generator and the discriminator are sent to the pose estimation module, and the final pose of the pig is obtained by the DARK and PAF methods.

[0066] More specifically, as follows:

[0067] Generator: The generator aims to learn the mapping from RGB images to keypoint heatmaps of a pig skeleton, generating as realistic and accurate heatmaps as possible. However, keypoints on a pig skeleton, such as the tail and snout, occupy a relatively small proportion of the body, while the limbs have high similarity in shape. Therefore, the generator needs to fully understand global and local semantic information when extracting image features. Thus, this embodiment uses HrNetV2 as the backbone network of the generator and has made fine-tuning adjustments, such as... Figure 10 As shown. The network consists of four stages. After inputting an RGB image, it goes through an initial stage containing two 3x3 convolutional layers and a pooling layer, producing an image with a scale equal to the original. Figure 1 / 8 high-resolution subnet. Then, in each subsequent stage, the resolution of the subnet is gradually increased from high to low by downsampling, and the multi-resolution subnets are connected in parallel. Throughout the process, the parallel multi-resolution subnet extracts features through a 3*3 convolutional layer and a residual block, and repeatedly exchanges information by adding convolutional layers, which not only integrates high-resolution features and low-resolution features, but also maintains a high-resolution heatmap representation. Finally, in the output stage, the network upsamples and connects several other low-resolution convolutions as features, and uses transposed convolutional layers with low information redundancy and high mapping ability to upsample the features to obtain a heatmap for encoding key point confidence information and a PAF for encoding key point connection relationships. The heatmap size is 1 / 4 of the original image resolution, and the number of channels is k, where k represents the number of key points. PAF[1] is a method that has been widely used in bottom-up estimation of human pose. It captures the relationship between body parts by encoding each joint vector. Its resolution size is the same as the heatmap, and the number of channels is k*2, representing each joint component on the x,y plane. Compared to HrNet, it integrates features from various levels at the end, improving global semantic understanding. Finally, in addition to calculating the basic heatmap loss and PAF loss, an adversarial loss function is also added, the details of which will be explained later.

[0068] Discriminator: The discriminator needs to distinguish the real heatmap from the fake heatmap as much as possible, so as to guide the generator to output more reasonable poses through the way of adversarial game, and improve the detection accuracy of the occluded key points. The fake heatmap and the real heatmap are the output heatmap of the generator and the real value connected with the corresponding RGB image respectively. This can be considered as adding the implicit features of the skeleton to the original RGB image, providing more content for the occluded key points which lack visible information. The discriminator will learn whether the pig skeleton structure described by the heatmap is reasonable and consistent with the pig distribution in the RGB image, and improve the discrimination ability. This leads to an increase in the difficulty of the generator deceiving the discriminator, prompting the generator to output more credible results. Considering that the purpose of the discriminator is only to distinguish the difference between the real heatmap and the fake heatmap, the network structure is simpler than that of the generator. The overall structure still uses HrNet, but only uses one stage of semantic fusion at the end, which can also better complete the discrimination task. For each training image, the generated heatmap and the ground-truth heatmap are fed back to the discriminator, and the corresponding fake loss function and real loss function are calculated.

[0069] The HRNetV2-Deconv network proposed in the embodiment has four stages. The left part basically remains the same as HrNetV2. Each stage produces a low-resolution subnet with 1 / 2 down-sampling and channel number * 2. The resolution and depth of each subnet are [128, 256, 512, 1028] and [1 / 8, 1 / 16, 1 / 32, 1 / 64] respectively. Multi-scale semantic fusion of convolution layers is performed at the end of each stage. The network finally outputs a heatmap with a size of 1 / 8 of the original image and a channel number of the number of key points. Figure 1 / 8, the channel number of the key point is the heatmap. The upper and lower two columns in the legend represent the convolution kernel size and the step size respectively.

[0070] Based on the principle of the generative adversarial network, the training of the model in the embodiment is a process of a generator and a discriminator (the discriminator is also referred to as a discriminator in the embodiment) in an adversarial game. The goal of the generator G is to minimize the distance between the output heat map and the heat map reconstructed by the discriminator, while the discriminator D tries to maximize it. In other words, in order to distinguish the pose, the discriminator needs to capture the essential features of the real pose distribution. At the same time, the generator tries to generate high-quality key point heat maps, so as to deceive the discriminator and make the discriminator reconstruct similar heat maps by detection. The above-mentioned adversarial game process is specifically reflected in the loss function, as shown in equations (1) and (2). For the loss function of the generator network, in addition to the sigmoid cross-entropy classification loss function for calculating the key point heat map loss of itself and the L2 loss of PAF, a fake loss for calculating the generative adversarial loss is added. The fake loss is derived from the loss function of the discriminator, which adjusts the features and spatial dependencies learned by the generator by calculating the difference between the generated “fake” heat map and the label through the L2 loss function, to ensure that the generated key point heat map result conforms to the pig skeleton structure and is reliable. The loss function of the discriminator network is composed of real loss and fake loss. The loss functions of real loss and fake loss are the same, and the difference lies in that the real loss calculates the loss between the “real” heat map and the label. The reason for subtraction is that we expect the real loss to be small enough, so that the discriminator can distinguish the real heat map from the generated heat map generated by the generator. We expect the fake loss to be large enough, so that the generated heat map generated by the generator is close to the real heat map to deceive the discriminator. In addition, in order to prevent the discriminator from converging too fast, a weight variable gt with an interval of [0, 1] is introduced to the fake loss part, which is used to control the balance between the generator and the discriminator during training, to improve the training stability of the generative adversarial network. As shown in equation (3), gt represents the emphasis degree of L fake . When the performance of the generator is better than that of the discriminator (L fake <βL real ), gt will increase, so that the loss function L fake has a higher weight ratio in the training process, so that the discriminator will be trained more in identifying the generated heat map. Similarly, when the performance of the discriminator is better than that of the generator (L fake >βL real ), gt will decrease, so as to slow down the training speed of the discriminator and let the generator catch up with the pace of the discriminator.

[0071]

[0072] L Discriminator =L real (G,X)-gt*Lfake (S, X) (2)

[0073] g t+1 = g t + μ(βL real (S, C)-L fake (S, X)) (3)

[0074] Where C represents the generated heat map, X represents the ground truth, S represents the fake heat map, and G represents the real heat map. μ and β are hyperparameters, which are set to 0.0001, 0.5, and 0.001, respectively. They are used to control the corresponding weights.

[0075] Inference: For each training image, the model proposed in this embodiment predicts these outputs through end-to-end training. After training, the discriminator part will be removed, and the output of the generator after convergence will be sent to the pose estimation module. After obtaining the ReGenerated Heatmap generated through the adversarial game, the pose estimation module first needs to extract the positions of the key points corresponding to the local maximum peaks in the heat map. However, since there may be multiple key points in the image, multiple local maximum peaks will be generated. Therefore, a non-maximum suppression method is used to find the local maximum peak in the heat map, and then the position is mapped back to the original coordinate space according to the size of the heat map scaling. In addition, DARK is used to optimize the offset error caused by the change in scale during the position mapping process. This method only needs to calculate the first and second derivatives of each pixel position in the heat map to obtain more accurate pixel positions. The specific formula definition during mapping is as follows:

[0076] loc p = stride * (peaks + offsets p ) + stride / 2 (4)

[0077] offsets p = (D″(p)) -1 D′(p) (5)

[0078] Where loc p represents the predicted position of the key point p in the original image, peaks represent the local maximum peak of the key point p in the heat map, offsets p represent the offset calculated using the DARK method, D′ and D″ represent the first and second derivatives, respectively, and stride represents the scaling size, which is 4 in this embodiment.

[0079] For the matching part between keypoints and individuals, the PartAffinity Field vector field obtained in the generator training process is needed to calculate the PAF score between each pair of keypoints by line integral, and the cost matrix of each pair of keypoints is obtained, and the best match of each key point is obtained according to the Hungarian algorithm. Then, referring to the processing method in the reference, the minimum data-driven graph is taken as the maximum spanning tree (that is, the subgraph covering all keypoints, the number of edges is the smallest, and the part correlation cost is the largest). Search the graph after network evaluation, and gradually expand the minimum skeleton with strong recognition edges in the order of the size of the ability to separate two distributions, form up to 9 increasingly redundant graphs. And select the graph that maximizes the purity and the score of connecting key points together as the animal assembly graph. Finally, find the nodes belonging to a certain individual by searching the connected components in the graph (in graph theory, these are subsets of key points that can reach each other but do not share connections with any other key points), and obtain the final group pig pose.

[0080] Example 2

[0081] Evaluation method: OKS (key point similarity) is used as the evaluation index of the model in this embodiment, which is widely used in known works, and is inspired by the IoU index in target detection. The purpose is to calculate the similarity between the true value and the predicted human key points. The definition of OKS is as follows:

[0082]

[0083] di represents the Euclidean distance between the detected key point and its corresponding label, vi represents whether the point is a visible point, s represents the scale of the pig body, and k is the manually annotated position offset. According to the OKS score, the corresponding mAP, AP50, AP75, and A90 indicators are calculated, and AP50 represents the ratio of the OKS score greater than the threshold value 0.5 in all OKS scores.

[0084] In addition, the percentage of correct key points (PCKh) is used as an evaluation standard to specifically analyze the recognition accuracy of each key point of the model under occlusion. PCK evaluates the prediction result by measuring the distance between the predicted key point and the label less than a certain threshold of the head length. For example, PCKh@0.5 which indicates the percentage of key points with a distance difference within half the threshold of the head length. PCKh is defined as:

[0085]

[0086] where di represents the Euclidean distance between the predicted value and the true value of the ith key point. di i def ​denotes the scale factor, in this embodiment, the size of the pig's head. δ(*) denotes 1 if the condition is met, otherwise 0. T m is a threshold set artificially, the smaller the value, the more stringent the evaluation standard. The PCKh metric is different for different animals and sizes, and the correct threshold needs to be selected. Using T = 0.5 is a more reasonable threshold.

[0087] Example 3

[0088] To verify the actual effect of the present application, experiments are carried out in this embodiment, which are as follows:

[0089] (1) Experimental setup.

[0090] The model in this paper is trained on the pig data set. Before training, the data set is first data augmented, the batch size is 1, and the first-order adam optimizer is used for optimization. The learning rate of the generator adopts a segmented strategy, the initial learning rate is 1e-4, and is attenuated to 5e-5 and 1e-5 at the 7500th iteration and the 12000th iteration, and the total number of epochs is 15000. The learning rate of the discriminator is set to 1e-5 because the discriminator is more likely to converge during training. All experiments in this paper are carried out on a server with a GPU of Nvidia Telsa P100.

[0091] (2) Model training.

[0092] Figures 11-12 The training process of GANPose on the data set is shown, and all generator losses and discriminator losses reach a balance and remain in a stable state. In Figure 11 , the loss has a significant jump at the 1000th iteration, because the hyperparameter is set from 0.01 to 0.1. The advantage of this is that the generator can pay more attention to the extraction of key point heat map features in the early stage of training, and then adjust the results according to the opinions of the discriminator when the understanding is more sufficient. In Figure 12 , it can be seen that the real loss and the fake loss are initially maintained at a very high level, but as the number of training increases, the gap between the real loss and the fake loss gradually widens until it stabilizes at a higher level, which shows that the weight variable gt plays a certain role, enabling the discriminator to fully distinguish the difference between the fake heat map generated by the generator and the real heat map, achieving Nash equilibrium.

[0093] (3) Quantitative evaluation of the pose estimation model.

[0094] The classical and current advanced animal pose estimation methods are compared in the same test environment and the data set of the present embodiment. The performance of different methods on the OKS evaluation index is shown in Table 2. For the index mAP, the present method has an improvement of 23.15 compared with DeepLabCut, which fully proves that the present method has stronger reasoning ability compared with other methods, especially for high occlusion degree reasoning. In particular, it is also found that for the index AP75, the model also has a significant improvement compared with other methods.

[0095] Table 2: Comparison of OKS performance on pig data set.

[0096]

[0097] (4) Visualization results of the pose estimation model:

[0098] In order to understand how the proposed method achieves the goal of pose estimation with the help of generative adversarial network, the present embodiment verifies the effectiveness of the model of the present embodiment in real scenes and compares it with DeepLabCut. As shown in Figures 13-16 , it can be seen that the model of the present embodiment has a better understanding of the pig's body image, thereby reducing the strange pose. It can estimate the occluded joints, especially the elbow and hoof key points which are easily occluded by themselves, while maintaining high performance on visible joints. Figure 13 for highly twisted and staged situations. Among them, left (a1, b1), middle (a2, b2), right (a3, b3) respectively represent the key extraction results of SLEAP, DeepLabCut and GANPose.

[0099] In the example of Figure 8, the pig’s body is twisted or truncated, resulting in some limbs being invisible or hard to identify in the image. In these cases, DeepLabCut, SLEAP failed to understand some poses, while the method of the present embodiment succeeded. This can be due to the fact that the present embodiment implicitly constrains the body into the discriminator of the present embodiment, and the model is able to infer the correct pose even in the case of a twisted pig body. The model of the present embodiment, which fuses multi-resolution semantic features during training, has a better global understanding of the twisted or truncated individual. In the example of Figure 9, the pig’s limbs are occluded by an automatic feeding device or a fence. Due to the lack of a full understanding of the pig’s skeletal structure, DeepLabCut, SLEAP failed to well predict the occluded keypoints, while the model of the present embodiment, which fuses multi-resolution semantic features during training, has a better global understanding of the occluded parts. It successfully predicts the correct keypoint locations even in some difficult cases. In the examples of Figures 10 and 11, the images are affected by light and blur (which can be due to the fact that flies in the actual site contaminated the camera), resulting in a hard-to-identify pig body, but the model of the present embodiment, which has a high awareness of texture features and global skeletal structure, still performs well in the challenging pig house environment.

[0100] (5) Occlusion analysis:

[0101] Here, the present embodiment presents a detailed analysis of the model’s output when the image is occluded.

[0102] First, the present embodiment compares the performance of the method of the present embodiment and DeepLabCut under occlusion. In the pig dataset of the present embodiment, about 20% of the annotated keypoints are labeled as invisible. The present embodiment displays the results of invisible samples for each keypoint in Table 3. For the body parts without occlusion, the method of the present embodiment improves the baseline by about 9.08% in detection rate. In particular, the method of the present embodiment improves the detection rate of the invisible elbow and hoof by 16% and 18.4%, respectively. This shows the advantage of the method of the present embodiment in handling body parts.

[0103] Next, two examples of severe occlusion of body parts are given in Figure 17 In the first sample, one pig is riding on another pig, and the occluded part is well predicted in the corresponding heat map. Despite the occlusion, the pose heat map generated by the method of the present embodiment is mostly clear, with the Gaussian centers. Figure 17The heat map visualization in our dataset. In (a1), (b1) the method further refines the heat map and corrects the position. (a2), (b2) The prediction of DeepLabCut is mostly accurate, but the heat map shows that the model is not very sure about its answer.

[0104] In the second sample, the two pigs in the middle of the image are stuck to each other, and the method of the embodiment also successfully produces a relatively clear heat map, although the confidence scores of the occluded body parts are relatively low, but overall are still at a relatively high level. This shows that, since the network of the embodiment learns some prior knowledge of the skeletal structure of the pig in training. Therefore, even under some occlusion, it has the ability to predict a reasonable pose. This verifies the motivation of the embodiment for designing the discriminator with a gan.

[0105] Table 3: Detection results (PCK) of invisible key points for each model.

[0106]

[0107] (6) Ablation experiment

[0108] In order to analyze the model of the embodiment in more detail, the embodiment conducts a comprehensive and detailed ablation experiment to evaluate the method proposed in the embodiment. As shown in Table 4, the embodiment studies the influence of different modules on the model.

[0109] First, the embodiment analyzes the influence of the generative adversarial network architecture on the performance of the model, and compares the embodiment with the original method HRNET. As can be seen from Table 4, after adding the generative adversarial network, the backbone network is improved by 7.13mAP, which fully shows the significant improvement of the generative adversarial network architecture on the performance of the model. At the same time, the embodiment also pays attention to the fact that the AP90 index also has a large increase, which shows that the model can have a good performance on the highly occluded key points by using the learned skeletal features, and can make the predicted key points more biased towards the reasonable skeletal structure.

[0110] Secondly, the embodiment also analyzes the DARK module. Through the improvement of the key point heat map encoding method by Taylor expansion, it can be found that the mAP has a slight improvement on the basis of the original. In addition, the embodiment also analyzes the influence of the improved HrNetV2 network on the accuracy of the model. It can be seen that after using the HrNetV2-deconv network, the model has a further understanding of the semantic information of the skeleton, and the fusion efficiency of different resolutions is further improved. In general, these ablation experiments overwhelmingly confirm that each component is effective, and these clues are beneficial to the inference of the invisible joint.

[0111] Table 4: Ablation experiment results.

[0112]

[0113] This embodiment makes a pig dataset for pose estimation captured by multi-view cameras, which covers various poses, occlusions and lighting effects of pigs. Through the comparison and analysis of several experiments, this embodiment verifies the effective estimation of the algorithm in this paper for pose in the actual pig house environment. The human pose estimation dataset has been widely studied and used. However, this is still a blank in the field of pig pose estimation. Therefore, this embodiment makes a pig image dataset with multiple poses and occlusions, hoping to provide some support for other researchers.

[0114] In terms of time consumption, since the model of this embodiment only needs to use the discriminator during training, and the discriminator part is removed during the inference stage, the inference speed only depends on the complexity of the generator network. The results compared with other methods are shown in Table 5. In particular, compared with the top-down method, the parameter amount is less while ensuring the accuracy.

[0115] Table 5: Time consumption of each model.

[0116]

[0117] In the actual application process, this embodiment found that the model performs well when the individual's own parts are occluded, such as walking and squatting, and the elbows and hooves are occluded. When pigs have interactive behaviors, such as riding and gathering, which are challenging scenarios, such as Figure 18 The results of DeepLabCut and GANPose for some failed cases of challenging scenarios are shown in FIG. 5 (a) and (b), respectively. The model accuracy may decrease due to severe mutual occlusion and distorted shape. This may be because the model learns the implicit skeleton structure, but when two individuals are closely connected and the pose is complex, the model may be confused about the skeleton features, resulting in the inability to reasonably pair the key points to the same pig. However, even if the method of this embodiment fails in this case, the method of this embodiment still obtains more reasonable poses than previous methods. In these complex cases, the previous methods will eventually produce some unreasonable results due to the lack of shape constraints.

[0118] The method of the embodiment is also applicable to other four-limbed mammals, like pigs, and there is also a problem that the elbow and hoof key points are difficult to identify and are easily blocked when performing posture estimation on a cow. The elbow and hoof blocking problem of the individual cow is processed through the timing information, but the method of the embodiment has the following two advantages: 1. Since the timing information needs to obtain the content of the previous and subsequent frames, the real-time monitoring capability is essentially lost, and the embodiment does not need to use timing information to infer the blocked key points. 2. The embodiment can realize posture estimation in a group environment. Compared with single-person posture estimation, group posture estimation not only needs to find multiple key points in the image, but also needs to match the key points with individuals.

[0119] The embodiment proposes a generative adversarial network model to solve the posture estimation problem of a group of pigs, and has stronger robustness for the blocking, overlapping and distortion of the pig body. The generator and the discriminator of similar structure constitute a zero-sum game between them. The generator adopts HRNET as the skeleton network, is responsible for predicting the heat map of the pig body key points based on the image features, and the discriminator adopts a simpler network structure, can distinguish unreasonable postures, and provide useful hints for the generator to improve the heat map. In the final inference link, the embodiment uses the DARK method to improve the error loss in the decoding process of the heat map, and uses the PAF partial affinity field and the minimum spanning tree method to assemble each key point. The experimental results show that the model proposed in the embodiment achieves very significant results on the data set.

[0120] It should be appreciated that embodiments of the present application can be implemented or carried out by computer hardware, a combination of hardware and software, or through computer instructions stored in a non-transitory computer readable storage medium. The method can be implemented in a computer program using standard programming techniques - including a non-transitory computer readable storage medium configured with a computer program, wherein the storage medium thus configured causes a computer to operate in a specific and predefined manner - according to the methods described in the specific embodiments and the accompanying drawings. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed application-specific integrated circuit for this purpose.

[0121] Further, the operations of the processes described in this embodiment can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described in this embodiment (or variations and / or combinations thereof) can be implemented under the control of one or more computer systems configured with executable instructions (e.g., computer programs, one or more computer programs, or one or more applications), and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof. The computer programs include machine- readable instructions that can be executed by one or more processors.

[0122] Further, the methods can be implemented in any suitable type of computing platform operatively connected to, including but not limited to, a personal computer, mini-computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platforms, or in communication with charged particle tools or other imaging devices, and the like. Aspects of the present application can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, and the like, such that it can be read by a programmable computer to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. Further, the machine-readable code, or portions thereof, can be transmitted over wired or wireless networks. The present application described in this embodiment includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs implementing the steps described above in conjunction with a microprocessor or other data processor. The present application also includes the computer itself when programmed according to the methods and techniques described in the present application. The computer programs are capable of applying to input data to perform the functions described in this embodiment, thereby transforming the input data to generate output data that is stored to non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In preferred embodiments of the present application, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on a display.

[0123] As used in this application, the terms "component," "module," "system" and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and / or thread of execution and a component can be localized, partially and / or entirely, in one computer or distributed between two or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components can communicate by way of local and / or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and / or across a network such as the Internet with other systems via the signal).

[0124] It should also be understood that the application is described through embodiments, and the examples are only given to clearly and completely explain the technical solutions of the technical scheme proposed by the claims of the application, that is, to explain and illustrate the claims, so when judging whether the technical scheme described in the specification is sufficiently disclosed, the core essence of the technical scheme defined by the claims should be fully considered, and other technical problems unrelated to the core technical problems solved by the embodiments in the specification do not belong to the essence of the embodiments, and are not necessary technical features, so they can be referred to as implicit disclosure, and those skilled in the art can fully realize them by combining the prior art and common knowledge, so there is no need to make detailed descriptions.

[0125] It should be noted that the above examples are only used to illustrate the technical solutions of the present application and not to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, and all should be covered in the scope of the claims of the present application.

Claims

1. A method for pig posture recognition based on a generative adversarial network, characterized in that: The method comprises the following steps of: Collecting an initial data set of group pigs and pre-processing the initial data set; A generative adversarial network model comprising three parts of a generator, a discriminator and a pose estimation module arranged in sequence is constructed, and the generative adversarial network model is trained by using the pre-processed data set; When the data to be recognized is input into the trained generative adversarial network model, the generator outputs a heat map, the discriminator analyzes the difference between the heat map and the actual result, learns the geometric topological structure features of the skeleton, and then infers the occluded key points according to the skeleton geometric structure; The pose estimation module decodes the heat map output by the generator, extracts the inferred corresponding key point positions in the graph, and maps them back to the original graph according to the proportion; Finally, according to the distribution of key point positions, an independent group pig pose is assembled to complete the recognition; The generator takes HrNetV2 as the backbone network of the generator, and the network is divided into four stages, comprising the following steps: After the network inputs an RGB image, an initial stage containing two 3*3 convolution layers and a pooling layer is performed to generate a high-resolution subnetwork with a scale size of 1 / 8 of the original image; Each stage gradually increases the resolution of the subnetwork from high to low through downsampling, and connects the multi-resolution subnetworks in parallel, and in the whole process, the parallel multi-resolution subnetworks extract features through a 3*3 convolution layer and a residual block, and repeatedly exchange information through convolution layer addition; Finally, the network connects the other low-resolution convolutions after upsampling and outputs them as features, and uses a transpose convolution layer with small information redundancy and large mapping capacity to upsample the features to obtain a heat map for encoding key point confidence information and a partial affinity field (PAF) for encoding key point connection relationships; The pose estimation module comprises the following steps: After obtaining the heat map regenerated through the adversarial game, the corresponding positions of the key points in the graph, i.e. the pixel points with peak confidence in the heat map, need to be extracted first; The local highest peak in the heat map is found through non-maximum suppression, and the position is mapped back to the original coordinate space according to the size of the heat map scaling; The distribution-aware coordinate representation (DARK) method is used to optimize the offset error caused by the change of scale in the position mapping process, and only the first and second derivatives of each pixel position on the heat map are needed to obtain more accurate pixel positions. The specific formula definition in the mapping is as follows: loc p = stride * (peaks + offsets p ) + stride / 2 (4) offsets p = (D'(p)) -1 D'(p) (5) where loc p represents the location of the predicted keypoint p on the original image, peaks represents the local maxima of the keypoint p on the heat map, offsets p represents the offsets calculated using the DARK method, D' and D" represent the first and second derivatives, respectively, and stride represents the size of the scaling and is 4.

2. The pig posture recognition method based on the generative adversarial network according to claim 1, characterized in that: The size of the heat map is 1 / 4 of the original image resolution, and the number of channels is k, where k represents the number of key points. The PAF captures the relationship between parts by encoding each joint vector, and its resolution size is the same as that of the heat map, and the number of channels is k*2, representing each joint component in the x and y planes. 3.The herd posture recognition method based on the generative adversarial network according to claim 1, wherein: The training of the generative adversarial network is the adversarial process of the generator and the discriminator, and the loss functions are as shown in the following formulas (1) and (2): L Discriminator = L real (G,X)-gt*L fake (S,X) (2) A weight variable gt with interval [0, 1] is introduced in the network, which represents the emphasis degree of L fake when the performance of the generator is better than that of the discriminator (L fake <βL real ), gt increases, so that the loss function L fake has a higher weight ratio in the training process, resulting in that the discriminator will be trained more in identifying the generated heat map; when the performance of the discriminator is better than that of the generator (L fake >βL real ), gt decreases to slow down the training speed of the discriminator, so that the generator can catch up with the pace of the discriminator; g t+1 = g t + μ(βL real (S,C)-L fake (S,X)) (3) In the formula, C represents the generated thermal map, X represents the real value, S represents the fake thermal map, and G represents the real thermal map. μ, β are hyperparameters, which are set to 0.0001, 0.5, 0.001 respectively, for controlling the corresponding weights. 4.The herd posture recognition method based on the generative adversarial network according to claim 1, wherein: In the pose estimation module, for the matching part between the key points and the individuals, The method comprises the following steps: The PAF score between each pair of key points is calculated by line integral using the partial affinity field obtained from the generator training, and the cost matrix of each pair of key points is obtained, and the best matching of each key point is obtained according to the Hungarian algorithm; The minimum data-driven graph is taken as the maximum spanning tree, and the search is performed on the graph after network evaluation, and the minimum skeleton with strong recognition edges is gradually expanded by the order of the ability size of separating two distributions through edges, and up to 9 increasingly redundant graphs are formed; The graph that maximizes the purity and the score of connecting key points is selected and reserved as the graph of animal assembly; Finally, the nodes belonging to a certain individual are found by searching the connected components in the graph, and the final group pig pose is obtained.

5. The pig posture recognition method based on the generative adversarial network according to claim 1, wherein: For each training image, the generative adversarial network model predicts output through end-to-end training, and after training, the discriminator part is removed, and the output of the converged generator is sent to the pose estimation module. 6.The herd posture recognition method based on the generative adversarial network according to claim 1, wherein: The preprocessing of the initial data set includes the following steps, The K-MEANS clustering method is used to classify similar frames in the collected data into a cluster, and a frame is selected from each cluster in turn as a key frame image to construct the initial data set of each pig posture; The key points of each pig in the initial data set are labeled, including distinguishing the left and right elbows and hooves, a total of 14 key points, and the labeled key points are divided into two categories: visible key points and occluded key points; The training set and the test set are divided in the ratio of 8:

2.

7. The pig posture recognition method based on a generative adversarial network according to claim 1, characterized in that: The generative adversarial network model can detect the key points of the group pig with an mAP of 74.09%, and for the occluded key points, the correct key point percentage PCK index can reach up to 82.0%.

Images