Method and system for extracting cattle bone joint and recognizing posture based on visual sensing

Abstract

The method and system for extracting cattle bone joint and recognizing posture based on visual sensing, through image preprocessing, input image is normalized and sharpened, image details are enhanced, and subsequent processing requirements are adapted; then the improved ResNet network is used to effectively extract image features; then the target detection algorithm GCMT is used for target detection, the image area of the cattle is accurately positioned, the posture recognition is carried out in the area, the designed key point detection network Cattle-PoseNet is used to extract 17 key points of the cattle bone and recognize the posture, finally, the posture output is carried out, the recognized key points are connected according to the biological prior knowledge, and the complete posture graph of the cattle is output. The algorithm and application system for extracting cattle bone joint and recognizing posture based on image processing solve the problems of low accuracy, environmental condition limitation and image quality requirement of the traditional technology, do not need expensive equipment, have low use cost, do not need manual monitoring, and are convenient to apply.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer image vision, and relates to posture recognition of animals, and is a method and system for extracting cow skeletal joints and recognizing postures based on visual sensing. BACKGROUND

[0002] In modern animal husbandry, it is crucial to accurately assess the health and growth status of cows. However, traditional health assessment methods usually rely on manual observation or the use of expensive equipment, and require a lot of time and resources. Therefore, there is a need for a method and system that can automatically, quickly and accurately assess the health status of cows.

[0003] In the field of computer vision and image processing, joint detection and posture estimation technology has made significant progress. These technologies are mainly used in human posture estimation, gesture recognition and other fields, and achieve joint detection and tracking by identifying the key points of the human body or objects. However, there are still obvious deficiencies in the recognition and estimation of skeletal joints and postures of large mammals such as cows. The quality of the detected video or image also has requirements, such as when there are many targets in the image or occlusion occurs, or when there is insufficient light, the existing algorithm still has room for improvement in processing effect.

[0004] Therefore, there is a need for a new method and system for recognizing and extracting cow skeletal joints based on image processing, which can overcome the limitations of traditional methods and achieve automatic, fast and accurate recognition and extraction of cow skeletal joints. SUMMARY

[0005] In order to solve the problems of low accuracy, environmental condition limitation and image quality requirement of traditional technology, the present application provides an algorithm and system for extracting cow skeletal joints and recognizing postures based on image processing. On the one hand, a method for extracting cow skeletal joints and recognizing postures based on deep learning is provided. On the other hand, a complete cow skeletal detection application system is established for actual cow motion recognition.

[0006] The technical solution of the present application is a method for extracting cow skeletal joints and recognizing postures based on visual sensing, comprising the following steps:

[0007] Step 1, collect video images of cows, normalize the images, and perform image preprocessing;

[0008] Step 2, feature extraction is performed on the preprocessed image I, and the feature extraction network uses an improved ResNet network, which combines SE mechanism and spatial pyramid pooling SPP, embeds SE block after residual block, and adds spatial pyramid pooling layer in the last fully connected layer;

[0009] Step 3, according to the extracted features, the target detection algorithm GCMT is used to detect the target of the cow, identify the position of the target cow, obtain the detection frame of the cow, and cut the picture of the cow according to the detection frame for standby;

[0010] The target detection algorithm GCMT adds position information to the features extracted in step 2 after two-dimensional feature vector decomposition, and outputs the prediction result of the target through the GM Transformer module. The GM Transformer module is based on the DETR target detection model, encapsulates the generation of the query object Object Queries into a separate module GCOQ, replaces the first-level multi-head self-attention module in the encoder and decoder structure with a dynamic resolution enhancement attention module DETA, and replaces the second-level self-attention module in the encoder structure with a dynamic multi-scale attention module DMSA; wherein

[0011] GCOQ updates the query object Object Queries by introducing a graph structure between the query objects Object Queries using a graph attention network GAT;

[0012] DETA adds a dynamic filtering layer in the multi-head self-attention, and dynamically enhances or weakens some attention connections based on the position distance in the feature map;

[0013] DMSA assigns weights to feature maps of different scales according to global and local information, combines and fuses feature maps of different scales according to the weights, and then performs self-attention calculation;

[0014] Step 4, joint extraction of cow bones in the detection frame is realized;

[0015] Step 5, convert the detection key points into the posture of the cow.

[0016] Further, step 1 is specifically:

[0017] The collected original image I o is scaled to a unified pixel size, and if the image pixel is lower than the unified pixel, the image is sharpened using an enhanced Laplacian sharpening ELS operator to enhance the image features:

[0018] The ELS operator is a linear combination of a two-dimensional Laplacian operator L and an edge enhancement high-pass filter H to emphasize both boundary and texture details:

[0019] ELS = aL + bH

[0020] Where a and b are adjustable weights, and the ELS operator is applied to the image I o to obtain the sharpened image I sharp as follows:

[0021] I sharp =I o +c·(ELS*I o )

[0022] wherein c is a coefficient for adjusting the sharpening intensity, and c takes a value between 0.3 and 0.5.

[0023] Further, in step 2, the improved ResNet network is referred to as SE-SPP ResNet, which includes an initial convolutional layer, a max-pooling layer, two residual block groups + SE modules, one residual block group + grouped convolution, one residual block group + SE modules, and a spatial pyramid pooling layer + a fully connected layer arranged in sequence.

[0024] Further, the GCOQ module is specifically:

[0025] First, define the query object Object Queries as the nodes of the graph, and define the initial Object Queries matrix as:

[0026]

[0027] wherein N is the number of query objects, and D is the feature dimension of each query object, which is initialized based on the features extracted in step 2 to construct a graph:

[0028] G(V,E)

[0029] wherein V is the node set, corresponding to the query objects; E is the edge set, representing the potential association between the query objects, which is calculated based on the similarity of the Euclidean distance or cosine similarity,

[0030] The features of the query object Object Queries are updated using the graph attention network GAT, and for each node m in the graph, the updated feature h ′ m is calculated in the following manner:

[0031]

[0032] wherein h n is the original or previous iteration feature of node m, N(m) represents the neighbor node set of node m, σ is an activation function, and a mn is the attention coefficient between nodes m and n, which is calculated by the following formula:

[0033]

[0034] W is a learnable weight matrix applied to each node feature, and a is a learnable parameter vector used to calculate the attention coefficient.

[0035] Further, the dynamic resolution enhancement attention module DETA includes a multi-head self-attention module and dynamic content-aware filtering, and the following is achieved:

[0036]

[0037] wherein F l (u,v) is a weight function of the hierarchical l position u relative to v, Q l , K l , V l respectively represent the query Query, key Key, and value Value matrices of the hierarchical l, for normalization, F l (u,v) is a distance-based filter, which is realized by a Gaussian function:

[0038]

[0039] wherein P u and P v are the coordinates of positions u and v on the feature map, and sigma is a parameter for controlling the decay rate.

[0040] Further, in the dynamic multi-scale attention module DMSA, for different scales of features Figure X l use a neural network to calculate fusion weights alpha l according to global and local information, and these weights determine the contribution of each scale of feature maps in the final fusion, and the feature fusion formula is as follows:

[0041]

[0042] wherein X' is the fused feature map, Transform(X l ) represents a set of conversion operations, so that the dimensions of feature maps of all scales are consistent, and the fused feature map is used to generate values and keys in attention calculation for self-attention calculation.

[0043] The application also provides a cattle bone joint extraction and posture recognition system based on visual sensing, which is used to realize the above-mentioned cattle bone joint extraction and posture recognition method based on visual sensing, and comprises:

[0044] An image acquisition module is used to acquire video images of cattle, and the image acquisition device needs at least a resolution of 1080p to ensure image clarity;

[0045] An image preprocessing module is configured to realize image preprocessing of step 1, scale a picture into a picture with a size of 720 pixels*720 pixels, ensure that the size of an input image conforms to the input of a network, and introduce an ELS operator to sharpen the image, so that the image features input into the network are more obvious.

[0046] A target detection module is configured to realize feature extraction of step 2 and target detection of a cow of step 3, and is used for later skeleton joint extraction, the target detection module is loaded with a computer program, and the improved ResNet network in the method of claim 1 and the target detection algorithm GCMT are realized, the improved ResNet network is used to extract a feature map, and the GCMT is used to realize target detection based on the feature map, and a cow skeleton joint is obtained.

[0047] A posture recognition module is configured to realize steps 4 and 5, connect the skeleton joints output from the target detection module into a posture of cow movement, divide the skeleton joints on the cow skeleton into 17 points according to the features of the cow skeleton joints in the skeleton joint extraction part, and establish a new network to extract the 17 skeleton joints, after the 17 skeleton joints are recognized, the skeleton joints are connected according to prior knowledge, and the final posture is output.

[0048] The present application has the following advantages:

[0049] 1. An enhanced Laplacian sharpening operator is established to sharpen the image, which can adapt to target objects of different scales and sizes, so that the image features input into the network are more obvious, and the model has better generalization ability when processing multi-scale images.

[0050] 2. For the target detection of the cow, an improved GCMT target detection algorithm based on the DETR algorithm is proposed, and the GCMT has the following advantages: the GCMT improves the structure of Object Queries, so that the model can better utilize target information and improve the detection performance; the GCMT introduces a spatial multi-head attention mechanism, which can better capture the spatial relationship between targets, thereby improving the accuracy and robustness of target detection; the GCMT adopts a dynamic multi-scale attention mechanism, which can adaptively adjust the attention weight, effectively process targets of different scales, and improve the generalization ability of the algorithm.

[0051] 3、In the detection of cattle skeletal joint, a new image basic feature extraction method is established, which combines SE block and SPP layer into the traditional ResNet architecture, so that the model not only improves the channel level feature adjustment ability, but also increases the spatial scale adaptability, and the combination of the two makes the model more accurately capture useful information when processing complex image tasks. The introduction of SE block helps the model to better generalize to new and unseen data, because it improves the effectiveness of features by learning the dependency between channels. At the same time, the multi-scale feature extraction strategy of SPP layer further enhances the adaptability of the model to different sizes and scale changes.

[0052] 4、The present application establishes a new network to extract cattle skeletal joint, named Cattle-PoseNet, which is composed of cattle pictures, so that its performance is more prominent in extracting cattle skeletal joint; Cattle-PoseNet combines deep learning and traditional computer vision methods, fully utilizes the advantages of deep learning in feature learning, and combines traditional pose estimation methods, so that the algorithm is more robust and reliable; Cattle-PoseNet adopts a multi-scale feature fusion module, which can effectively integrate feature information at different scales and improve the adaptability to cattle of different sizes and proportions; the model structure of Cattle-PoseNet is relatively simple, and the training and deployment are relatively easy, so that the algorithm is more easily popularized and applied in practical application. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The flowchart of the algorithm for extracting cattle skeletal joint and pose recognition in the present application.

[0054] Figure 2 The flowchart of image basic feature extraction in the present application.

[0055] Figure 3 The flowchart of residual block group + grouped convolution in the present application.

[0056] Figure 4 The flowchart of residual block group + compression and activation block in the present application.

[0057] Figure 5 The flowchart of the target detection algorithm in the present application.

[0058] Figure 6 The flowchart of GM Transformer network in the present application.

[0059] Figure 7 The flowchart of GCOQ module in the present application.

[0060] Figure 8 The flowchart of DETA module in the present application.

[0061] Figure 9 This is a flowchart of the DMSA module of the present invention.

[0062] Figure 10 This is a flowchart of the Cattle-PoseNet network of the present invention.

[0063] Figure 11 This is a flowchart of the application system for bovine skeletal joint extraction and posture recognition in this invention. Detailed Implementation

[0064] The following describes specific embodiments of the present invention. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the present invention. That is, in some embodiments of the present invention, these practical details are not essential.

[0065] like Figure 1 As shown, this invention proposes a method for extracting joint points and recognizing postures of a bovine skeleton based on visual sensing, comprising the following steps:

[0066] S1. Acquire color video images of cattle, and perform normalization and image preprocessing.

[0067] Image preprocessing plays a crucial role in bovine skeletal joint extraction and pose recognition tasks. It can help improve image quality, reduce noise, and enhance features, thereby providing better input for subsequent processing steps.

[0068] Because different video recording devices are used in actual production processes, the output image sizes vary. In this invention, the original image I is first... o The image is scaled to 720 pixels by 720 pixels to ensure the input image size matches the network's input. If the image resolution is higher than 720 pixels by 720 pixels, it can be directly scaled down. If the image resolution is lower than 720 pixels by 720 pixels, the image will be enlarged, causing distortion. This invention uses the Enhanced Laplacian Sharpener (ELS) operator to sharpen the image, making its input features more prominent. The specific sharpening method is illustrated through examples.

[0069] The ELS operator is defined as a linear combination of the Laplacian operator L and an edge-enhancing high-pass filter H. As an example, first, the standard two-dimensional Laplacian operator is defined:

[0070]

[0071] Then, a high-pass filter is defined, aiming to enhance the high-frequency part of the image, such as edge details:

[0072]

[0073] The ELS operator is a combination of the two filters, aiming to emphasize both larger boundaries and more subtle texture details:

[0074] ELS = aL + bH (3)

[0075] Where a and b are adjustable weights, commonly set to a = 1 and b = 0.5 in the present invention, so that the edges are moderately enhanced, while the overall contrast and clarity of the image are increased.

[0076] Apply the ELS operator to the image I o , to obtain the sharpened image I sharp as follows:

[0077] I sharp = I o + c·(ELS*I o ) (4)

[0078] Where c is a coefficient that adjusts the intensity of sharpening, usually c takes a value between 0.3 and 0.5, to ensure that the image details are significantly enhanced without excessive introduction of noise.

[0079] S2, image basic feature extraction.

[0080] In order to realize efficient image basic feature extraction in the algorithm of cattle skeletal joint extraction and posture recognition, the present invention proposes an improved ResNet network, called SE-SPP ResNet. Based on the improvement of traditional ResNet, the specific structure is as shown in Figure 2 , which is composed of an initial convolutional layer, a maximum pooling layer, a residual block group, a compression and activation block, a grouped convolution, and a spatial pyramid pooling layer. The specific application and implementation process of the model in the present invention are described in detail below.

[0081] The model input is a color cattle image with a resolution of 720x720 pixels. The image is first subjected to standardization processing, including color normalization and illumination condition adjustment, to reduce the influence of environmental variation on the performance of the model, to obtain the preprocessed image I.

[0082] Since the picture outputted in step S1 is relatively large in size of 720*720, the picture is further reduced in size to facilitate feature extraction and reduce the dimension of calculation. The present application uses an initial convolutional layer to capture the preliminary features of the image and reduce the dimension. The initial convolutional layer uses a convolutional kernel of 7x7, 128 output channels, a step of 2, and edge padding of 3. The purpose of this layer is to quickly reduce the size of the image while extracting preliminary spatial features. After the convolution operation, batch normalization and ReLU activation function are applied to enhance the nonlinear processing capability of the network and stabilize the training process. The max pooling layer uses a max pooling kernel of 3x3 with a step of 2 to further reduce the spatial dimension of the feature map and increase the model's invariance to small displacements. The specific way is as follows:

[0083] The picture is first further reduced in size to facilitate feature extraction and reduce the dimension of calculation. The present application uses an initial convolutional layer to capture the preliminary features of the image and reduce the dimension. The convolutional kernel takes a size of 7*7, and the step is 2. The specific formula is as follows:

[0084] Conv(I)=ReLU(BN(W*I+b)) (5)

[0085] where W and b are the weights and bias of the convolutional kernel, respectively, BN is batch normalization, and ReLU is the activation function. After convolution, the size of the image becomes approximately 360x360 pixels. In order to further reduce the dimension of the features, the present application uses a max pooling layer after the initial convolution. A pooling kernel of 3*3 is selected with a step of 2. After pooling, the size of the image becomes approximately 180x180 pixels.

[0086] The residual block design is as follows: the standard ResNet residual block is used to construct a deep feature extraction network, and an SE block is connected after each residual block. The SE block compresses the spatial information of each channel into a single number through global average pooling operation; through a two-layer fully connected network, the dimension is first reduced and then increased, and the adjustment coefficient of each channel is output through the Sigmoid activation function; the adjustment coefficient obtained by excitation is multiplied by the original channel feature to realize feature re-calibration.

[0087] The specific structure is as follows: Figure 3 In the residual convolution, x is the input feature map. The operations in the residual block usually include two convolutional layers, each followed by a batch normalization layer and a ReLU activation function. The formula is represented as:

[0088]

[0089] where represents the residual function, including two convolutional operations with weights W1, W2 and biases b1, b2, and x is the input feature map. BN is the batch normalization operation. ReLU is the activation function.

[0090] The Squeeze and Excitation block SE explicitly models the importance of each channel, and the SE block makes the network learn how to effectively utilize global information, optimize the feature dependence between channels, and thus improve the richness and accuracy of feature expression. The Squeeze and Excitation block first performs global average pooling on the feature map after residual convolution to achieve Squeeze, then realizes Excitation through a fully connected layer, recalibrates the importance of each channel, and finally multiplies the recalibrated weight back to the original feature map to perform Scale operation. The formula of each part is as follows:

[0091] Squeeze:

[0092]

[0093] Excitation:

[0094]

[0095] Scale:

[0096]

[0097] wherein, is the c-th channel compression feature obtained by global average pooling, and HxW represents the feature map size. s is the excitation weight of all channels obtained by two fully connected layers, W3 and W4 are the weights of the fully connected layer, δ is the ReLU activation function, and σ is the Sigmoid function, represents the set of compression features of all channels, and is represented as a vector, wherein each element corresponds to the compression feature of a channel. s c is the excitation weight of the c-th channel, is the feature map of the c-th channel adjusted by the excitation weight. The output feature map adjusted by the residual connection and the SE is combined, and the formula is:

[0098]

[0099] wherein x is the original input feature map, is the feature map adjusted by the SE block.

[0100] After the combination of two-layer residual modules and compression and activation blocks, there is a module that combines residual blocks with grouped convolution. This module introduces grouped convolution into traditional residual blocks to reduce the consumption of computing resources while maintaining the expressive power of the model. By grouping feature maps and performing convolution on each independent group, grouped convolution greatly reduces the number of parameters and the amount of computation. This not only improves computational efficiency, but also helps the model to be deployed in resource-constrained environments, such as mobile devices and embedded systems commonly used in production environments. In this invention, we divide the input feature map into multiple groups, and each group of features is independently convolved. Finally, these features are combined. The specific module is as follows Figure 4 The input feature map X is divided into G groups, and each group is independently convolved. The convolution kernel W is applied to each group of features. Figure X g where g represents the gth group.

[0101] X g = W g *X g for each group g = 1, 2, …, G (11)

[0102] where W g is the gth group convolution kernel, and X g is the subset of features assigned to the gth group. By grouping feature maps and processing each group of data independently, grouped convolution can effectively reduce the number of parameters, thereby reducing the risk of overfitting and improving computational efficiency.

[0103] Then the convolution results of all groups are combined into a single feature map, as follows:

[0104] Y = Concat (X1, X2, …, X G ) (12)

[0105] Apply batch normalization and ReLU activation function to the combined feature map Y:

[0106] Z = ReLU (BN (Y)) (13)

[0107] This step is a standard operation to normalize the data distribution and introduce nonlinearity, which helps to improve the training process and the generalization ability of the model.

[0108] Finally, the activated feature map Z is directly added to the original input X to form the final output feature map O

[0109] O = Z + X (14)

[0110] ​Then there is another layer of residual block group and the combination of compression and activation block, followed by the combination of spatial pyramid pooling module and fully connected layer, adjusting the size of the picture, outputting the final feature image. The spatial pyramid pooling layer integrates the spatial pyramid pooling layer at the end of the residual network, performs 1x1, 2x2, 4x4 pooling operation to capture the context information of different scales, and splices all the pooling outputs into a fixed length feature vector, ensuring that the dimension of the output feature remains unchanged regardless of the change of the input image. The fully connected layer converts the fixed length feature vector output by the SPP layer into a format suitable for subsequent processing, such as joint position prediction or pose classification label, through the fully connected layer.

[0111] The training details are as follows: the model is trained using cross-entropy loss (classification) or mean square error loss (regression). The Adam optimizer is used, with an initial learning rate of 0.001, and the learning rate decay strategy is used to adapt to the training process. The network hierarchy and hyperparameters such as learning rate, batch size, etc. are adjusted based on the performance on the validation set to achieve the best performance.

[0112] S3, identifying the position of the target cow and cropping the picture of the cow

[0113] The input image is detected for target detection using the GCMT (Graph-based Convolutional Multitask Tracker) algorithm based on the graph, to identify the position of the cow in the image, such as Figure 5 The GCMT algorithm uses a spatial attention mechanism and a dynamic multi-scale attention mechanism, which can effectively capture the spatial relationship between targets and adaptively adjust the attention weight, improving the accuracy and robustness of target detection. In the GCMT algorithm, the features extracted in step 2 are decomposed into two-dimensional feature vectors, and position information is added, and the GM Transformer module outputs the prediction result of the target. The GM Transformer module is similar in structure to the DETR target detection model, with the improvement being the change in the generation method of the query object Object Queries and the structure of the multi-head self-attention mechanism module and the multi-head attention mechanism module, making it more suitable for the task of extracting cow bone joint points in the present application. Since the generation method of the query object Object Queries in DETR has been changed, the present application encapsulates it into a separate module, named GCOQ (Graph-Contextual Object Queries Module), with a structure as shown in Figure 7 .

[0114] GCOQ is a method that uses Graph Neural Networks (GNN) to enhance Object Queries, especially for object detection tasks in complex scenes. This module allows Queries to communicate information with each other by introducing a graph structure between Object Queries, enhancing their expression and judgment ability.

[0115] We first define Object Queries as nodes of a graph. Let the initial Object Queries matrix be:

[0116]

[0117] where N is the number of Queries and D is the feature dimension of each Query. These Queries are initialized based on the features extracted by the neural network initially. On this basis, a graph is constructed:

[0118] G(V,E) (16)

[0119] where V is the node set, corresponding to Object Queries; E is the edge set, representing the potential association between Queries. These edges are calculated based on similarity measures such as Euclidean distance or cosine similarity.

[0120] The invention uses Graph Attention Network (GAT) to update the features of Object Queries. For each node i in the graph, its updated feature h i ′ is calculated as follows:

[0121] h i ′ = σ (∑ j∈N(i) α ij Wh j ) (17)

[0122] where h j is the original or previous iteration feature of node i. N(i) represents the neighbor node set of node i. α ij is the attention coefficient between nodes i and j, calculated as follows:

[0123]

[0124] W is a learnable weight matrix applied to each node feature, a is a learnable parameter vector used to calculate the attention coefficient, and σ is an activation function, such as ReLU, which is selected by the invention.

[0125] After the GAT layer processing, each Query not only retains its original target information, but also integrates the information of its neighbor Queries, achieving context enhancement of information. This enables each Query to more comprehensively consider the surrounding environment and the state of other targets when performing target detection, thereby improving the accuracy and robustness of detection.

[0126] Through the above steps, the GCOQ module effectively utilizes the graph structure to enhance the function of Object Queries, enabling the target detection model to have better performance in complex scenes. This method is particularly suitable for production environments where target interactions are significant, or there are a large number of occlusions and intersections.

[0127] After generating the q, k, and v vectors through the GCOQ module, the Dynamic Resolution Enhanced Attention (DETA) module is entered, as shown in Figure 8 The multi-head attention of DETR is improved to a multi-head self-attention module plus a dynamic content-aware filtering module, which can more effectively process features of various scales and focus on more critical information areas.

[0128] The invention introduces a dynamic content-aware filter, which aims to enhance or weaken certain attention connections based on the content of the features, so that the attention mechanism can focus more on important or information-rich parts of the image. This method is particularly suitable for processing complex visual scenes, where some areas may contain more information about the objects to be detected than other areas. The specific formula is as follows:

[0129]

[0130] where F l (u,v) is the weight function of level l position u relative to v, Q l , K l , V l represent the Query, Key, and Value matrices of level l, respectively, is used for normalization. F l (u,v) realizes dynamic content-aware filtering through weights, using a distance-based filter that considers the spatial distance between two positions in the feature map to emphasize the relationship between adjacent positions while reducing the influence of distant positions. This can be achieved through a Gaussian function:

[0131]

[0132] where P u and P v are the coordinates of position u and v on the feature map, and σ is a parameter that controls the decay rate.

[0133] After passing through the Add&Norm module, the output value v and the query q, key k of the encoder part are input into the dynamic multi-scale attention DMSA module, the structure is as follows Figure 9 This module aims to enhance the model's detection ability for different size targets by adaptively combining features from different scales, especially in complex scenes where the size and scale of the target differ significantly.

[0134] First, use the network architecture FPN with multi-resolution output to generate a series of feature maps {X1, X2, …, X L} from the input image, where X1 is the highest resolution feature map, and X L is the lowest resolution feature map. Then, for each scale of feature Figure X l Use a small neural network to calculate a fusion weight α l according to global and local information, which determines the contribution of each scale feature map in the final fusion. The feature fusion formula is as follows:

[0135]

[0136] where Transform(X l ) is a set of transformation operations to make the dimensions of all scale feature maps consistent for summation. The processed features after fusion are the same as the traditional attention module. It is worth noting that the Q l , K l , V l used in the attention calculation in the dynamic multi-scale attention module DMSA are query, key, and value matrices generated by X ′ , rather than directly input q, k, v directly calculated.

[0137] After passing through the GM Transformer, the target detection of the cow can be realized, and the detection frame classified as a cow is recorded for later use.

[0138] S4, implement joint extraction of cow bones inside the detection frame

[0139] The Cattle-PoseNet network is used for the extraction of the joint points of the cattle skeleton. Cattle-PoseNet is a network specially designed for cattle pose estimation tasks, which can accurately identify the key point positions on the cattle, including the head, tail, limbs and other key parts. The image of the target cattle is input into the Cattle-PoseNet network, and the joint points of the cattle skeleton are extracted through the forward propagation process of the network. The Cattle-PoseNet network contains multiple layers of convolutional neural networks and key point regression layers, which can effectively learn image features and predict the positions of key points on the cattle. The output of the Cattle-PoseNet network includes the coordinate information of each key point on the cattle, such as the positions of the head, tail, limbs and other key points. These coordinate information can be obtained through the output layer of the network, that is, the predicted key point coordinates. The predicted key point coordinate information is mapped to the original image to realize the visualization of the joint points of the cattle skeleton. This can be achieved by drawing the position markers of the key points on the image, thereby visually displaying the recognition results of the cattle pose.

[0140] The present application divides the joint points on the cattle skeleton into 17 points, which are named as head, left eye, right eye, chest, back, left shoulder, left arm, left hand, right shoulder, right arm, right hand, left hip, left leg, left foot, right hip, right leg, and right foot. According to the above characteristics of the cattle skeleton joint points, the present application establishes a new network to extract 17 skeleton joint points, which is named as Cattle-PoseNet, as shown in Figure 10 The network is trained by cattle pictures to make its performance more prominent in extracting cattle skeleton joint points.

[0141] Firstly, the cattle detection picture generated by S3 is scaled to a uniform size, and the SE-SPP ResNet network is used for feature extraction. These two processes are similar to the processes described in S2, and will not be repeated here. After extracting the features, the deep separable convolution module is entered. Deep separable convolution (Depthwise Separable Convolution) is an efficient convolution operation method, which decomposes the standard convolution operation into two smaller operations: depthwise convolution (Depthwise Convolution) and pointwise convolution (Pointwise Convolution). This structure reduces the number of model parameters and reduces the computational complexity, while maintaining or enhancing the performance of the network. Assuming that the input features Figure Xwhere C is the number of channels, H and W are the height and width of the feature map. In the process of depth convolution, a convolution kernel is applied independently for each input channel, which mainly processes spatial features without crossing channel information. The size of the convolution kernel for each channel is 3x3, and the point-by-point convolution is used: a 1x1 convolution kernel is used to aggregate the output channels of the depth convolution. This step is essentially a fully connected layer processing for each position of the feature points, thereby realizing feature integration and dimension transformation. The formula is as follows:

[0142] Y = (X * K depthwise )*K1 ×1 (22)

[0143] where X * K depthwise represents applying a corresponding depth convolution kernel to each channel of X, which is 3x3 in the present application, and K 1×1 is a point-by-point convolution kernel with dimensions C'x Cx 1x 1, used to convert C channels to C' channels.

[0144] Then comes the spatial attention module, which mainly functions to dynamically adjust its focus on the processed feature map, strengthening the features most relevant to the current task (i.e., the key point positions of the cow), while ignoring background noise and irrelevant information. This mechanism helps the network accurately identify key points in various scenarios, especially in complex backgrounds or less-than-ideal lighting conditions. The spatial attention module is usually composed of the following steps: first, the feature map is compressed along the channel direction through average pooling and max pooling layers, simplifying each HxW feature map to a 1-dimensional response. This step aims to aggregate global spatial information. Then, the outputs of average pooling and max pooling are combined and processed through one or more convolution layers (1x1 convolution is used in the present application) to produce a spatial attention map. Next, the values of the attention map are normalized to between 0 and 1 through an activation function to generate attention weights for each spatial position. The overall calculation formula is as follows:

[0145] X out = (σ(Conv 1×1 ([AvgPool(X);MaxPool(X)])))⊙X (23)

[0146] where the input feature map AvgPool and MaxPool represent average pooling and max pooling, respectively, and both have dimensions Conv 1×1 is a 1x1 convolution kernel, σ is a Sigmoid function used to normalize the attention weights to [0, 1], and ⊙ represents element-wise multiplication, which element-wise multiplies the generated spatial attention map with the original feature map (point multiplication) to strengthen the features in the key regions and suppress the features in other regions, X outThe output feature map represents the output, where the weighted features are more prominent in the key regions of the original feature map. By introducing this spatial attention mechanism, Cattle-PoseNet can accurately identify and predict the pose of cattle in various environments, particularly showing high performance in automated pasture management and veterinary diagnosis.

[0147] Then comes the multi-scale feature fusion module, which mainly functions to effectively fuse feature maps from different scales to obtain more rich and representative feature representations. This helps to improve the adaptability of the pose detection model to targets of different scales, enabling it to accurately detect key points on cattle of different sizes and proportions. Let the feature maps from different levels be F1, F2, …, F n The calculation of the multi-scale feature fusion module can be represented as:

[0148]

[0149] where F fusion is the fused feature map, a i is the weight of each feature map, obtained through training.

[0150] The output convolutional layer of the Cattle-PoseNet algorithm is responsible for mapping the learned features of the network to the output space of the target detection task, i.e., predicting the positions of the key points of the cattle. This convolutional layer is usually a 1x1 convolutional kernel that maps the feature maps of the last layer of the network to the final key point coordinates.

[0151] Finally, an activation function is applied to limit the output range, ensuring that the output is within a reasonable range and has appropriate nonlinear characteristics. In Cattle-PoseNet, Softmax normalization is performed on each pixel point to ensure that the sum of the probabilities of all pixel points for each key point is 1:

[0152]

[0153] where Y k represents the heat map of the kth key point, X 4k is the fused feature map obtained by multi-scale feature fusion, and j is used to traverse all pixel positions. In this way, the heat map of the cattle's skeletal joints can be output, and the position with the maximum joint probability can be extracted as the output of the cattle's skeletal joints.

[0154] S5, convert key points to pose

[0155] According to the 17 cattle bone joints output by S4, the joints are connected according to prior knowledge, specifically: the head, left eye and right eye are connected to each other to form a triangle; the left shoulder is connected to the left arm, and the left arm is connected to the left hand to form a line, and the right shoulder, right arm and right hand are also connected in this way; the left hip, left leg and left foot are connected; the right hip, right leg and right foot are connected; then the left shoulder and the right shoulder are connected to the chest respectively, and the left hip and the right hip are connected to the back; finally, the head is connected to the chest, and the chest is connected to the back, so that the posture of the cattle is completely extracted.

[0156] Compared with the prior art, the ELS operator proposed in the application can perform image denoising and enhancement processing on the noise points caused by poor light of the image, so that the algorithm has better anti-noise ability.

[0157] The GM-Transformer proposed in the application introduces a GCOQ module, in which the graph attention network enhances the context enhancement of information, so that better results are obtained in the case of occlusion between the cattle and the background. Meanwhile, the DETA module proposed in the application introduces a dynamic content perception filtering module compared with the self-attention module in the traditional Transformer, so that the attention mechanism can pay more attention to important or information-rich parts of the image, especially in the case of occlusion. The DMSA module of the application introduces a network architecture FPN with multi-resolution output compared with the traditional Transformer, which improves the output effect of the model for different sizes of targets.

[0158] On the basis of target detection, the application proposes a network named Cattle-PoseNet to extract cattle bone joints, introduces a depth separable convolution mechanism to reduce the amount of calculation, thereby reducing the resources consumed by the method and system of the application. Meanwhile, a spatial attention mechanism is introduced to pay more attention to key area information to improve the extraction accuracy of the bone key points. In addition, the application introduces a multi-scale feature fusion module when extracting cattle bone joints to improve the applicability of the model to different sizes of targets, so that it has good robustness in different environments.

[0159] As shown in Figure 11 The application also provides an application system for cattle bone joint extraction and posture recognition based on image processing, comprising:

[0160] Image acquisition module, image acquisition module is the foundation module in the system of cattle skeletal joint extraction and posture recognition, its main responsibility is to obtain high-quality image or video data, for subsequent image processing, joint detection and posture analysis. About image acquisition equipment, preferably at least 1080p (1920x1080) resolution to ensure image clarity, enough to distinguish small joints; The camera should have good low light performance, especially when used in insufficiently lit indoor environments or at night. About image acquisition environment, ensure adequate and uniform lighting, avoid strong backlight or shadow, which may affect the image quality and the accuracy of joint detection; Simplify the background can reduce the interference in image processing, improve the accuracy of joint detection, using a uniform color background wall is a good choice.

[0161] Image preprocessing module is a key link in the system of cattle skeletal joint extraction and posture recognition, which directly affects the efficiency and accuracy of the subsequent modules. The main purpose of this module is to improve the image quality, reduce the noise in processing, and convert the image size format to adapt to the subsequent processing flow. Because the video recording equipment used in the actual production process is different, the output picture size is different, in this invention, first of all, the picture is scaled to 720 pixels*720 pixels size, to ensure that the input image size conforms to the input of the network, at the same time, the ELS operator is introduced to sharpen the image, so that the image features input into the network are more obvious.

[0162] Target detection module is responsible for extracting feature maps from preprocessed images and separating cattle from complex backgrounds for later skeletal joint extraction. In the feature extraction part, this invention introduces an improved ResNet network based on traditional ResNet network, named SE-SPP ResNet, which consists of an initial convolutional layer, a max pooling layer, a residual block group, a compression and activation block, a grouped convolution, and a spatial pyramid pooling layer. Through the combination of these technologies, the SE-SPP ResNet model not only has a significant improvement in feature extraction accuracy and efficiency, but also exhibits excellent performance and adaptability when processing high-resolution images, which is crucial for cattle skeletal joint extraction and posture recognition tasks; In the part of separating cattle from background, this invention proposes a target detection algorithm improved based on DETR algorithm, named GCMT algorithm, which includes an improved transformer structure named GM-Transformer, changes the way of generating Object Queries, changes the structure of multi-head self-attention mechanism module and multi-head attention mechanism module, making it more suitable for the cattle skeletal joint extraction task of this invention.

[0163] The posture recognition module is responsible for extracting the skeleton joints from the attention region output by the target detection module, and connecting them into the posture of the cattle movement and output. In the skeleton joint extraction part, the application divides the joints on the cattle skeleton into 17 points according to the characteristics of the cattle skeleton joints, and establishes a new network to extract the 17 skeleton joints, which is named Cattle-PoseNet. The network training is composed of cattle pictures, so that the performance of the network is more prominent in extracting the cattle skeleton joints. After recognizing the 17 skeleton joints, they are connected according to prior knowledge, and the final posture is output.

[0164] The algorithm and application system for extracting cattle skeleton joints and posture recognition based on image processing solve the problems of low accuracy, environmental condition limitation and image quality requirement of the traditional technology, do not need expensive equipment, have low use cost, do not need manual monitoring, and are convenient to apply.

[0165] The above description is only an embodiment of the application and is not used to limit the application. The application can have various changes and modifications for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the application shall be included in the scope of claims of the application.

Claims

1. A method for extracting joint points and recognizing pose in bovine skeletons based on visual sensing, characterized by: Includes the following steps: Step 1: Acquire video images of cattle, normalize the images, and perform image preprocessing; Step 2, processing the preprocessed image Feature extraction is performed using an improved ResNet network, which combines SE mechanism and spatial pyramid pooling (SPP). SE blocks are embedded after residual blocks, and spatial pyramid pooling layers are added to the final fully connected layer. Step 3: Based on the extracted features, the target detection algorithm GCMT is used to detect the cow, identify the location of the target cow, obtain the detection box of the cow, and crop the cow image according to the detection box for later use. The object detection algorithm GCMT decomposes the features extracted in step 2 into two-dimensional feature vectors, adds positional information, and outputs the prediction results of the target through the GM Transformer module. The GM Transformer module is based on the DETR object detection model. It encapsulates the generation of the query object Object Queries into a separate module GCOQ, replaces the first-level multi-head self-attention module in the encoder and decoder structure with a dynamic resolution enhanced attention module DETA, and replaces the second-level self-attention module in the encoder structure with a dynamic multi-scale attention module DMSA. in GCOQ introduces a graph structure between object queues and uses a graph attention network (GAT) to update the object queues. DETA adds a dynamic filtering layer to multi-head self-attention, dynamically enhancing or weakening certain attentional connections based on positional distances in the feature map. The dynamic resolution-enhanced attention module DETA includes a multi-head self-attention module and dynamic content-aware filtering, implemented as follows: in, It is a level Location relatively The weight function, , , Representing levels respectively The query, key, and value matrix. Used for normalization, For distance-based filters, a Gaussian function is used: in and They are the locations and Coordinates on the feature map It is a parameter that controls the decay rate; DMSA assigns weights to feature maps at different scales of an image based on global and local information, combines the weights to fuse feature maps at different scales, and then performs self-attention calculation. Step 4: Extract the joints of the bovine skeleton within the detection frame; Step 5: Convert the detected key points into the pose of a cow.

2. The method for extracting bovine skeletal joints and recognizing posture based on visual sensing according to claim 1, characterized in that: Step 1 is as follows: The original images collected Scale to a uniform pixel size. If the image pixel count is lower than the uniform pixel count, sharpen the image using the Enhanced Laplacian Sharpening (ELS) operator to enhance image features. The ELS operator is a linear combination of a two-dimensional Laplacian operator L and an edge-enhancing high-pass filter H to simultaneously emphasize boundary and texture details: in, and These are adjustable weights that apply the ELS operator to the image. The image is then sharpened. as follows: in To adjust the sharpening intensity coefficient, The value ranges from 0.3 to 0.

5.

3. The method for extracting bovine skeletal joints and recognizing postures based on visual sensing according to claim 1, characterized in that: In step 2, the improved ResNet network is called SE-SPP ResNet, which includes an initial convolutional layer, a max pooling layer, two residual block groups + SE modules, one residual block group + grouped convolution, one residual block group + SE modules, and a spatial pyramid pooling layer + fully connected layer configured in sequence.

4. The method for extracting bovine skeletal joints and recognizing posture based on visual sensing according to claim 3, characterized in that SE -SPP ResNet specifically refers to: First, the image is reduced in size and preliminary spatial features are extracted using an initial convolutional layer. After the initial convolution, a max pooling layer is used to further reduce the image size, thus completing the second dimensionality reduction of the image. In the residual block group + SE module, the operations in the residual block group include two convolutional layers, each followed by a batch normalization layer and a... The activation function, SE module, first performs global average pooling on the feature map output by the residual block group to achieve compression, then activates it through a fully connected layer, recalibrates the weights of each channel, and then multiplies the recalibrated weights back into the original feature map for scaling operation, outputting the feature map adjusted by SE block, which is then superimposed with the original input feature map as the output feature map of residual block group + SE module. In residual block group + grouped convolution, the input feature map The data is divided into groups, each undergoing independent convolution operations. The convolution results from all groups are then merged into a single feature map. Batch normalization is then applied to the merged feature map. An activation function compares the activated feature map with the input feature map. They are added directly to form the output; Then, after passing through a residual block group + SE module, followed by a combination of spatial pyramid pooling module and fully connected layer, the image size is adjusted, and the final feature image is output, which is the preprocessed image. Extracted feature map.

5. The method for extracting bovine skeletal joints and recognizing posture based on visual sensing according to claim 1, characterized in that G The COQ module is specifically as follows: First, define the query objects (Object Queries) as nodes in the graph. Let the initial Object Queries matrix be: in, It is the number of query objects. These are the feature dimensions of each query object. These query objects are initialized based on the features extracted in step 2 to construct a graph: in, It is a set of nodes, corresponding to the query object; It is a set of edges representing potential associations between query objects. The similarity between these edges is calculated based on measures such as Euclidean distance or cosine similarity. Use a graph attention network (GAT) to update the features of the query objects (Object Queries) for each node in the graph. Its updated features Calculate as follows: in, The corresponding node Features of the original or previous iteration, Represents a node The set of neighboring nodes, It is an activation function. It is a node and The attention coefficient between them is calculated by the following formula: It is a learnable weight matrix applied to the features of each node. It is a learnable parameter vector used to calculate the attention coefficient.

6. The method for extracting bovine skeletal joints and recognizing posture based on visual sensing according to claim 1, characterized in that it is dynamic. In the multi-scale attention module DMSA, for feature maps of different scales... The fusion weights are calculated using a neural network based on global and local information. These weights determine the contribution of feature maps at each scale to the final fusion. The feature fusion formula is as follows: Among them For the fused feature maps, This represents a set of transformation operations that make the feature maps of all scales have the same dimensions, and use the fused feature maps to generate values ​​and keys for attention computation to perform self-attention computation.

7. The method for extracting bovine skeletal joints and recognizing posture based on visual sensing according to claim 1, characterized in that: Step 4 specifically involves dividing the joints of the cow skeleton into 17 points and building a Cattle-PoseNet network to extract these 17 skeletal joints. The Cattle-PoseNet network includes basic feature extraction, depthwise separable convolution, spatial attention operations, and multi-scale feature fusion. After convolution and activation functions, a heatmap is output. First, the images of cows within the detection bounding boxes generated in step 3 are scaled to a uniform size, and features are extracted using an improved ResNet network. Then, they enter the depthwise separable convolutional module to achieve feature integration and dimensionality transformation. After the depthwise separable convolutional module, the spatial attention module is input. First, the feature maps are compressed along the channel direction through average pooling and max pooling layers, simplifying each feature map into a one-dimensional response to aggregate global spatial information. Then, the outputs of average pooling and max pooling are merged and processed through one or more convolutional layers to generate a spatial attention map. Finally, the values ​​of the attention map are normalized to between 0 and 1 through an activation function to generate attention weights for each spatial location. Attention weights are combined to fuse multi-scale features, and the fused features are then convolved and mapped to the output space of the object detection task, i.e., predicting the keypoint locations of the cow. Finally, an activation function is applied to limit the output range, and Softmax normalization is performed on each pixel to ensure that the sum of the probabilities of all pixels at each keypoint is 1. ,for each keypoint in Indicates the first Heatmap of key points It is a feature of fusion. Used to iterate through all pixel positions.

8. The method for extracting bovine skeletal joints and recognizing posture based on visual sensing according to claim 1, characterized in that: In step 5, the key points of the obtained cow skeleton are connected according to prior knowledge to obtain the cow's posture.

9. A system for extracting joint points and recognizing postures of bovine skeletons based on visual sensing, characterized in that: The method for extracting bovine skeletal joints and recognizing postures based on visual sensing as described in claim 1 includes: The image acquisition module is used to acquire video images of cattle. The image acquisition device needs a resolution of at least 1080p to ensure image clarity. The image preprocessing module is used to implement the image preprocessing in step 1. It scales the image to 720 pixels * 720 pixels to ensure that the input image size matches the network input. At the same time, it introduces the ELS operator to sharpen the image, making the image features input to the network more obvious. The target detection module is used to implement the feature extraction in step 2 and the target detection of cattle in step 3. It is used for the subsequent extraction of skeletal joints. The target detection module is loaded with a computer program that implements the improved ResNet network and the target detection algorithm GCMT in the method of claim 1. The improved ResNet network is used to extract feature maps, and GCMT is used to perform target detection based on the feature maps to obtain the skeletal joints of cattle. The pose recognition module is used to implement steps 4 and 5. It connects the skeletal joints output from the target detection module to output the pose of the cow's movement. In the skeletal joint extraction part, based on the characteristics of the cow's skeletal joints, the joints on the cow's skeleton are divided into 17 points, and a new type of network is established to extract the 17 skeletal joints. After identifying the 17 skeletal joints, they are connected according to prior knowledge to output the final pose.

Images