Cow behavior deep learning classification method fusing multi-modal time-frequency features

Abstract

The invention relates to the technical field of deep learning, in particular to a cattle behavior deep learning classification method fusing multi-modal time-frequency features, which comprises the following steps: collecting animal head and neck muscle acceleration value data sets, and preprocessing the data sets; performing feature extraction on the acceleration value by using an FMD module and an FFT module, and fusing the two extracted features; inputting the fused time-frequency features into a multi-scale time-frequency feature analysis network; and classifying by using a classifier. According to the method, the problem that feature extraction and model design still need to be further optimized during cattle behavior classification in an existing method is solved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a cattle behavior deep learning classification method integrating multimodal time-frequency features. Background Art

[0002] Systematic observation and analysis of animal behavior (such as rumination, movement, etc.) can provide key data reflecting their comfort and potential health problems; this data not only helps to detect problems in a timely manner and take effective intervention measures, thereby significantly improving the quality of life and welfare of animals, but also improves the efficiency of pasture management.

[0003] In recent years, deep learning has performed well in processing complex time series data, especially in processing signals with long-term dependencies and nonlinear characteristics, showing great potential;

[0004] The paper Deep learning-based cattle behaviour classification using joint time-frequency data representation discussed the performance of the deep neural network (DNN) model represented by joint time-frequency distribution (TDF) data in cattle behavior classification. The study showed that fusing multimodal time-frequency features can provide more comprehensive information, thereby improving the accuracy of behavior recognition; however, the feature extraction method and model design still need to be further optimized to give full play to the advantages of these two types of features. Summary of the invention

[0005] In view of the shortcomings of the existing methods, the present invention designs the IMTFF-Networks model, constructs multimodal time-frequency fusion and multi-scale time-frequency feature analysis, and fuses the time-frequency domain features through feature modal decomposition and frequency domain processing to comprehensively capture the dynamic characteristics of behavioral signals such as frequency components and time patterns; by fusing time-frequency features, the limitations of single-domain methods in processing complex behavioral signals are overcome; the multi-scale time-frequency feature analysis module can capture long-term dependencies and complex nonlinear relationships between features at different time scales through multi-scale time series processing and attention mechanism; the model of the present invention classifies and identifies rumination and eating behaviors, and can support cattle health management and improvement of production efficiency.

[0006] The technical solution adopted by the present invention is: a cattle behavior deep learning classification method integrating multimodal time-frequency features comprises the following steps:

[0007] Step 1: Collect the animal head and neck muscle acceleration value data set and pre-process the data set;

[0008] As a preferred implementation of the present invention, preprocessing includes: data merging, data cleaning and denoising, data standardization and data windowing.

[0009] As a preferred embodiment of the present invention, a three-axis sensor is used to collect the data set.

[0010] As a preferred implementation of the present invention, data cleaning uses the Z-score method to detect and delete outliers.

[0011] Step 2: Use the FMD module and the FFT module to extract features of the acceleration value respectively, and fuse the two extracted features;

[0012] As a preferred embodiment of the present invention, step 2 specifically includes:

[0013] Step 21, extracting time domain features from the acceleration signal using the FMD module;

[0014] Step 22: Extract frequency domain features from the acceleration signal using FFT;

[0015] Step 23: Use the addition operation to fuse the frequency domain features and the time domain features.

[0016] Step 3: Input the fused time-frequency features into the multi-scale time-frequency feature analysis network;

[0017] As a preferred embodiment of the present invention, the multi-scale time-frequency feature analysis network includes: a TCN network, an adaptive reweighted attention mechanism module and an LSTM network, and the time-frequency features are sequentially input into the TCN network, the adaptive reweighted attention mechanism module and the LSTM network.

[0018] As a preferred embodiment of the present invention, the formula of the adaptive reweighted attention mechanism module is:

[0019]

[0020] s c =σ(W2·δ(W1·z c ))(6)

[0021]

[0022] Where T represents the length of the timing signal, x c (t) represents the input signal of the cth channel at the tth time step, z c represents the compressed features of the cth channel, W represents the weight matrix of the fully connected layer, and s c represents the weight of each channel, Represents the recalibrated timing signal.

[0023] Step 4: Use the classifier to classify;

[0024] As a preferred embodiment of the present invention, MLP is used as a classifier for classification.

[0025] As a preferred embodiment of the present invention, a deep learning classification system for cattle behavior integrating multimodal time-frequency features includes: a memory for storing instructions executable by a processor; and a processor for executing instructions to implement a deep learning classification method for cattle behavior integrating multimodal time-frequency features.

[0026] As a preferred embodiment of the present invention, a computer-readable medium stores a computer program code, and when the computer program code is executed by a processor, a deep learning classification method for cattle behavior that integrates multimodal time-frequency features is implemented.

[0027] Beneficial effects of the present invention:

[0028] 1. Design a multimodal time-frequency fusion module, which can adaptively extract the key modal components of nonlinear and non-stationary signals through characteristic mode decomposition (FMD), and effectively capture short-term fluctuations and instantaneous changes in behavioral signals; the time domain features of FMD are integrated with the frequency domain features extracted by FFT to construct a hybrid feature vector, which comprehensively captures the complex dynamic changes of cattle behavior and overcomes the limitations of traditional methods in dealing with sudden behaviors and complex behavior patterns;

[0029] 2. Design a multi-scale time-frequency feature analysis module. Through multi-scale time series processing and attention mechanism, it can effectively capture long-term dependencies and complex nonlinear relationships between features at different time scales. First, the input signal is processed by dilated causal convolution to efficiently extract local patterns and multi-scale features while maintaining the order of time series data. The attention mechanism is introduced to highlight key features. The gating mechanism and nonlinear activation function are used to capture long-term dependencies and complex nonlinear relationships, thereby successfully coping with the challenges brought by long-term dependencies and nonlinear patterns in behavioral characteristics.

[0030] 3. IMTFF-Networks performs well in the cattle behavior classification task, significantly outperforming methods that rely solely on single time domain or frequency domain features. IMTFF-Networks achieved 86.42% accuracy, 87.53% precision, 86.42% recall, and 86.64% F1 score in the test set. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of the cattle behavior deep learning classification method integrating multi-modal time-frequency features of the present invention;

[0032] Figure 2is a system connection diagram of the measuring device of the present invention;

[0033] Figure 3 The Rumiwatch halter is placed on the cow's mouth, and the Afimilk Silent Herdsman collar is placed around the cow's neck;

[0034] Figure 4 It is a classification diagram of time domain features, frequency domain features and fused multi-scale time-frequency features;

[0035] Figure 5 They are the comparison charts of Confusion matrix, TCN, LSTM, TCN-LSTM, and IMTFF-Networks;

[0036] Figure 6 They are comparison charts of t-SNE, TCN, LSTM, TCN-LSTM, and IMTFF-Networks. DETAILED DESCRIPTION

[0037] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0038] like Figure 1 , 2 As shown, a cattle behavior deep learning classification method integrating multimodal time-frequency features includes the following steps:

[0039] Step 1: Collect the animal head and neck muscle acceleration value data set and preprocess the data set;

[0040] Using the three-axis sensor of the collar on the animal's head and neck, the acceleration values ​​in the x, y and z directions of muscle movement are obtained;

[0041] Preprocessing includes: data merging, data cleaning and denoising, data standardization and data windowing;

[0042] Data Merge:

[0043] Since the dataset contains .CSV files of two sensor systems for each animal, and the SD cards of the two systems have been verified for time alignment, the data of the two systems are merged by sorting and aligning the ID and timestamp of each animal. This ensures that each timestamp corresponds to the data of the collar and halter, and maintains the consistency and correspondence of the data. At the same time, the data of each animal is placed in a .CSV file to simplify the complexity of data management and operation. This method makes data reading, processing and analysis more convenient, reduces the operation and management of multiple files, and thus improves the efficiency of data processing. In addition, the merged dataset provides complete animal behavior and movement information, providing an important data basis for subsequent analysis and modeling. By merging the data of the collar and halter, the information provided by the two sensor systems can be comprehensively utilized to fully understand the behavior patterns and movement trends of animals. This is of great significance for in-depth research on animal behavioral characteristics, identification of potential health problems, and improvement of livestock management strategies.

[0044] Data cleaning and denoising:

[0045] Data cleaning and denoising are one of the important steps in data preprocessing. In the data cleaning stage, the Z-score method is used to detect and delete outliers. Outliers refer to observations that are significantly deviated from most data points. By calculating the Z-score of each data point, its degree of deviation from the overall distribution of the data set can be evaluated. The Z-score threshold is set to 3. According to the statistical principle of normal distribution, about 99.7% of the data points are within three standard deviations of the mean. Therefore, data points outside this range are more likely to be considered outliers. The mathematical expression of the Z-score method is:

[0046]

[0047] Among them, x i is the sample value, u is the mean, and σ is the sample standard deviation.

[0048] The main reason for using the Z-score method for outlier removal is that it is a widely used statistical method that has been proven to have good results in many fields. By removing outliers, the interference of outliers on data analysis and modeling can be reduced, making the results more accurate and reliable. In addition, the Z-score method can also help identify potential data collection or recording errors, further improving the quality of the data.

[0049] The triaxial accelerometer readings include the acceleration generated by animal behavior, the influence of gravity, and noise. In order to solve these problems, a third-order Butterworth bandpass filter is selected. This filter can eliminate the DC component and high-frequency noise of gravity induction. Through experiments, a third-order Butterworth bandpass filter is designed with an upper cutoff frequency of 4 Hz and a lower cutoff frequency of 0.4 Hz.

[0050] Data Standardization:

[0051] In order to ensure the comparability of data between different axes, the three-axis acceleration data was standardized; standardization converts the data into a dimensionless form, eliminating the difference in the numerical range between different axes; this ensures that the data of different axes have the same importance and weight during training and modeling; therefore, all data are scaled to a mean (μ) of 0 and a standard deviation (σ) of 1;

[0052] Its mathematical expression is:

[0053]

[0054] Data Windowing:

[0055] In order to process and analyze the continuous three-axis acceleration data in segments, the sliding window method is used. By windowing the data, the local patterns and trends in the time series can be captured and the relevant feature information can be extracted.

[0056] In the classification task of three-axis acceleration data, it is crucial to choose an appropriate window size; the duration of behaviors of different animals varies, some behaviors occur for a few seconds, while some behaviors usually last for several minutes; for example, the behavior duration of rumination is about 40-60s, while other behaviors last for 2 minutes; in order to explore the impact of different window sizes on the experimental results, the present invention selects different window sizes in the experimental part; since the collected data provides measurements at a frequency of 10Hz, and there may be multiple instances of cattle behaviors at different window sizes, majority voting is applied within each window size to indicate the main behavior.

[0057] Step 2: Use the FMD module and the FFT module to extract features of the acceleration value respectively, and fuse the two extracted features;

[0058] Step 21, extracting time domain features from the acceleration signal using a feature mode decomposition module (FMD);

[0059] FMD is an efficient signal processing method that can adaptively decompose signals into modal components with different frequencies and amplitudes. Each modal component can not only capture specific features in the signal, but also effectively separate nonlinear and non-stationary components, which is particularly suitable for the analysis of complex behavioral signals; in this way, FMD can better characterize short-term fluctuations and instantaneous changes in cattle behavior.

[0060] The experimental parameters of the FMD module are set as follows: mode number n = 4, filter length L = 31, number of filters K = 5, and maximum number of iterations is 10; finally, 4 modal functions IMFs are extracted from each axis, and these IMFs are used as time domain features.

[0061] Step 22, extracting frequency domain features from the acceleration signal using Fast Fourier Transformation (FFT);

[0062] The frequency domain feature extraction process is also carried out in each sliding window, and different window functions are introduced to control the window shape to avoid spectrum leakage, so as to obtain a more accurate frequency domain feature representation; different window functions may have different effects on the extraction of frequency domain features, so the effects of different window functions are usually compared through experiments to determine the window function selection that is most suitable for a specific application. Subsequently, FFT is applied to each window. FFT is an efficient frequency domain analysis method. Its advantage is that it can quickly convert time domain signals into frequency domain representations. It has the characteristics of low computational complexity and fast processing speed, and is particularly suitable for processing large-scale time series data; extracting square amplitude as frequency domain features through FFT helps to reveal periodic patterns and frequency components in the signal. The formula of FFT is as follows:

[0063]

[0064] Among them, X(k) represents the frequency domain signal, x(n) represents the time domain signal, N represents the length of the signal, and k represents the frequency index.

[0065] Step 23: Use the addition operation to fuse the frequency domain features and the time domain features.

[0066] After feature extraction is completed, the time domain features and frequency domain features are fused to form a hybrid feature vector; this process aims to make full use of the complementary advantages of the two features to improve the accuracy and robustness of cattle behavior classification; specifically, the time domain features obtained by FMD are first spliced ​​with the frequency domain features obtained by FFT; this splicing not only retains the short-term fluctuations and instantaneous changes in the time domain features, but also combines the periodic patterns and frequency components revealed by the frequency domain features; this fusion method enables the model to more comprehensively capture the complex dynamic changes of cattle behavior and improves the classification performance under diversified behavior patterns; the hybrid feature vector finally generated provides richer input for subsequent deep learning models, which helps to further improve the accuracy and effectiveness of behavior recognition.

[0067] The fusion process not only integrates the feature information under different modes, but also lays the foundation for subsequent modules to further explore the long-term dependencies and nonlinear characteristics in the behavior patterns.

[0068] In order to comprehensively characterize the behavioral characteristics, the multimodal time-frequency features of the preprocessed sensor signals are fused; this process includes extracting time domain and frequency domain features through time domain decomposition and frequency domain conversion respectively; extracting time domain features using characteristic mode decomposition (FMD), and obtaining frequency domain features through fast Fourier transform (FFT); the time domain and frequency domain features are then effectively fused to form a mixed feature vector; this fusion process not only integrates the feature information under different modes, but also lays the foundation for subsequent modules to further explore the long-term dependencies and nonlinear characteristics in the behavioral patterns.

[0069] Step 3: Input the fused time-frequency features into the multi-scale time-frequency feature analysis network;

[0070] The present invention constructs a multi-scale time-frequency feature analysis network, which can effectively capture long-term dependencies and complex nonlinear relationships between features at different time scales through multi-scale time series processing and attention mechanism, and improve the accuracy and reliability of classification.

[0071] The multi-scale time-frequency feature analysis network includes: TCN network, adaptive re-weighted attention mechanism module and LSTM network. The time-frequency features are sequentially input into the TCN network, adaptive re-weighted attention mechanism module and LSTM network.

[0072] The TCN network processes the input signal through dilated causal convolution. This convolution method can efficiently extract the local pattern and multi-scale features of the signal while maintaining the order of the time series data, especially the features such as instantaneous changes and short-term fluctuations. In addition, by using dilated convolution, the network can expand the receptive field and capture the changes of the signal at different time scales without increasing the computational complexity. The formula of dilated convolution is as follows:

[0073]

[0074] Among them, f:{1,2,...,k-1} represents the filter, x∈R n represents a one-dimensional input sequence, d represents the dilation coefficient, which is used to control the interval size, k is the filter size, which represents the number of convolution kernels, and sd·i represents the past direction.

[0075] The relative importance of different modal features may be different. In order to enhance the expressiveness of key features, the present invention constructs an adaptive reweighted attention mechanism module, which dynamically adjusts the weight of each feature channel through "Squeeze" and "Excitation" operations; it not only highlights the key features, but also improves the overall feature representation ability of the network. The formula of the adaptive reweighted attention mechanism module is:

[0076]

[0077] s c =σ(W2·δ(W1·z c )) (6)

[0078]

[0079] Where T represents the length of the timing signal, x c (t) represents the input signal of the cth channel at the tth time step, z c represents the compressed features of the cth channel, W represents the weight matrix of the fully connected layer, and s c represents the weight of each channel, Represents the recalibrated timing signal.

[0080] In order to further deal with the complex nonlinear relationships and long-term dependencies in animal behavior data, the network finally uses the gating mechanism of the LSTM network, which is embedded in the recurrent unit and can capture and retain long-term dependency information, effectively dealing with long-term patterns and trends in behavioral characteristics; in addition, nonlinear activation functions are used to transform inputs and states, so that the network can learn and represent complex nonlinear relationships, thereby improving classification accuracy; the internal update state of the gating unit and the recurrent unit is calculated as follows:

[0081] i t =σ(W i x t +U i h t-1 +b i ) (8)

[0082] f t =σ(W f x t +U f h t-1 +b f ) (9)

[0083] o t =σ(W o x t +U o h t-1 +b o ) (10)

[0084]

[0085] h t =o t ⊙tanh(c t ) (13)

[0086] Among them, i t represents the input gate, f t represents the forget gate, o t represents the output gate, σ represents the sigmoid activation function, x t represents the input sequence at the current time step t, h t-1 represents the hidden state of the previous time step, W, U and b are the weight matrix and bias vector parameters that need to be learned during training, ⊙ represents the element-by-element product of the vector, represents the candidate memory unit, c t represents the memory unit of the current time step t, h t represents the hidden state at the current time step t.

[0087] By designing a multi-scale time-frequency feature analysis network, we can fully utilize the multimodal time-frequency features and solve the problem that traditional methods cannot simultaneously capture long-term dependencies and complex nonlinear relationships between features; ultimately, the network showed significant advantages and reliability in animal behavior classification tasks.

[0088] Step 4: Use the classifier to classify;

[0089] The output of the multi-scale time-frequency feature analysis module is classified using a multilayer perceptron (MLP) as a classifier; MLP consists of multiple fully connected layers and can effectively capture the high-dimensional interaction between features, thereby improving classification performance; specifically, MLP first receives the output features of LSTM, and then gradually maps them to the target classification space through a series of fully connected layers; after each fully connected layer, a nonlinear activation function (such as ReLU) is applied to increase the expressive power of the model; finally, MLP outputs the classification result to complete the discrimination of the input data.

[0090] Model optimization:

[0091] In order to effectively optimize the model parameters and improve the training efficiency and classification performance of the model, a combination of the Adam optimizer and the cross entropy loss function is adopted. After defining the model network, fine-tuning of the hyperparameters is an indispensable step. To this end, a random search method is used for hyperparameter optimization. First, an optimized hyperparameter space is defined, and then hyperparameter combinations are randomly sampled from the space for evaluation. The hyperparameters include: the number and kernel size of TCN, the number of network layers and neurons of LSTM, the Dropout value, the hidden units of MLP, the learning rate, the batch size, and the epoch. During the random search process, each randomly selected hyperparameter combination is used for model training, and the optimal hyperparameter combination is selected as the hyperparameter setting for the final animal behavior classification model.

[0092] Dataset:

[0093] A dataset of 18 Limousin cross bulls from three farm trials in the UK (Easter Hogate Farm, Edinburgh, UK) was used; each bull was equipped with an Afimilk Silent Herdsman collar, which was mounted on the neck, and a Rumiwatch halter, which was mounted on the muzzle, to collect animal behavior data, such as Figure 3 As shown; the Afimilk SilentHerdsman collar consists of a triaxial accelerometer, an SD card for data storage, and a real-time clock (RTC), sampling at a frequency of 10Hz, with a range of ±2g and a resolution of 12 bits; while the Rumiwatch bridle consists of a pressure sensor, SD card, and RTC, providing behavioral classification at a frequency of 10Hz; a total of 3460 hours of recording data were collected by time alignment verification of the SD cards of the two systems.

[0094] The Afimilk Silent Herdsman collar provides acceleration values ​​in the x, y, and z directions of the animal's head and neck muscle movements, while the Rumiwatch halter captures pressure changes caused by jaw movement, providing the following behavioral categories:

[0095] 1. Other - The animal is engaged in an activity that is neither ruminating nor eating.

[0096] 2. Rumination - Animals chew cud to further break down ingested food and improve nutrient absorption.

[0097] 3. Eating - the animal is taking in food.

[0098] This dataset has been made publicly available at https: / / www.doi.org / 10.5281 / zenodo.4064802.

[0099] Dividing the original data set into subsets such as training set, validation set and test set is the basis for model training and evaluation; using the random partitioning method, the data set is first randomly divided into a training set (80%) and a test set (20%); then the training set is randomly divided into a training set (80%) and a validation set (20%), which can ensure the generalization ability and accuracy of the model on different data sets and avoid overfitting and underfitting problems; using data set stratification technology to solve the key problem of category imbalance in the data set.

[0100] Evaluation indicators:

[0101] In order to evaluate the overall performance of the IMTFF-Networks model, a confusion matrix is ​​used to map the true values ​​and predicted values ​​to determine the true positive (TP), true negative (TN), false positive (FP), and false negative (FN) results of the classification. Based on these values, the following performance indicators can be calculated: including accuracy, precision, recall, and F1-Score.

[0102] Parameter configuration:

[0103] The selection of model hyperparameters has a significant impact on cattle behavior classification. In order to optimize the hyperparameters in the IMTFF-Networks network, a random search method was used. Through this method, an effective search can be performed in the hyperparameter space to find the optimal hyperparameter combination. The hyperparameters of the prediction network model are finally determined as shown in Table 1. Through this optimization method, the classification performance of the model can be maximized to ensure the efficiency and accuracy of the cattle behavior classification task.

[0104] Table 1 Model hyperparameter selection

[0105]

[0106] Experimental results:

[0107] 1. Effects of different sliding windows:

[0108] In the classification task of three-axis acceleration data, it is crucial to choose an appropriate sliding window size. The experimental results are shown in Table 2. When the window size is set to 120s, 86.42% accuracy, 87.53% precision, 86.42% recall and 86.64% F1 score are obtained in the test set. This shows that a larger window size helps to capture more complete behavior features, especially for behaviors with longer durations. However, for shorter behaviors, a finer-grained window setting may be required to avoid information loss. Therefore, in different application scenarios, the appropriate window size should be selected according to the duration of the behavior.

[0109] 2. Effects of different window functions:

[0110] In the process of frequency domain feature extraction, it is common to introduce different window functions to control the window shape. This processing is intended to avoid spectrum leakage and to obtain a more accurate frequency domain feature representation. However, different window functions may have different effects on the extraction of frequency domain features. Therefore, experiments are conducted to compare the effects of different window functions on feature extraction in order to determine the most suitable window function for a specific application. Specifically, the effects of rectangular window, Hanning window, Hamming window and triangular window are compared. The experimental results are shown in Table 3. Using a rectangular window, 86.42% accuracy, 87.53% precision, 86.42% recall and 86.64% F1 score are obtained in the test set.

[0111] Table 2 Experimental results of different sliding windows

[0112]

[0113] Table 3 Experimental results of different window functions

[0114]

[0115] 3. Verification of the effectiveness of feature extraction:

[0116] A key challenge of feature extraction is how to extract the most representative features; therefore, it is necessary to verify the effectiveness of feature extraction; to this end, experiments were conducted to compare the effects of different features on the experimental results; specifically, the performance of time domain features, frequency domain features and fused multi-scale time-frequency features in the classification task was evaluated; the experimental results are shown in Table 4, the fused multi-scale time-frequency features achieved 86.42% accuracy, 87.53% precision, 86.42% recall and 86.64% F1 score in the test set; the fused multi-scale time-frequency features performed well in all indicators; in addition, the classification of time domain features, frequency domain features and fused multi-scale time-frequency features Figure 4; Compared with other feature types, the fused multi-scale time-frequency features perform well in classification tasks and can better capture the information in the signal, thereby improving the classification performance of the model; These results verify the effectiveness of time domain features in this task and further prove the importance of feature extraction in improving model performance.

[0117] Table 4 Experimental results of feature extraction

[0118]

[0119] Ablation experiment:

[0120] In order to verify the superiority of the IMTFF-Networks of the present invention, a systematic ablation experiment was conducted. This experiment included a performance comparison of the following models: TCN, LSTM, TCN-LSTM and IMTFF-Networks. By comparing the performance of these models, we can deeply understand the contribution of each component to the overall model performance. The experimental results are shown in Table 5. IMTFF-Networks performs best in all indicators. In addition, the confusion matrix and t-SNE visualization technology are used to conduct an in-depth analysis of the performance of the model. The confusion matrix shows the accuracy and misclassification of the model classification results, as shown in Table 5. Figure 5 As shown, the confusion matrix clearly shows that the classification effect of the IMTFF-Networks model on the test set is more significant; Figure 6 The visualization results after dimensionality reduction of feature vectors by t-SNE technology are shown. It can be seen that the features extracted by the IMTFF-Networks model are evenly distributed in two-dimensional space, which further proves the effectiveness of the model in feature extraction and classification tasks. The ablation experiment verifies the effectiveness of the IMTFF-Networks model in capturing time series features and improving classification performance. The experimental results show that the combination of TCN and LSTM significantly improves the model performance. After the introduction of SE mechanism, the model performance is further improved and the best effect is achieved. This shows that the IMTFF-Networks model has significant advantages in processing complex time series data. This experiment not only proves the effectiveness of the combination of TCN and LSTM, but also highlights the important role of SE mechanism in enhancing model performance.

[0121] Table 5 Ablation experiment results

[0122]

[0123] Comparison with baseline:

[0124] In the data set, the models in Reference 1: Classification of cattle behaviours using neck-mounted accelerometer-equipped collars and convolutional neural networks and Reference 2: Behavioural classification of cattle using neck-mounted accelerometer-equipped collars are used as baselines to test the architecture of the present invention; specifically, the baseline model includes: the framework proposed in Reference 1 includes a feature extractor and a head; the feature extraction part consists of 4 convolutional layers, Dropout, BatchNorm and ReLU layers; the head consists of a fully connected layer (512 inputs and 3 outputs) and a Softmax layer to produce classification outputs. 42 features were extracted from Reference 2 and evaluated using Hidden Markov Model (HMM), Linear Discriminant Analysis (LDA) and Partial Least Squares Discriminant Analysis (PLSDA); Experiments show that the LDA model works best; The experimental results are shown in Table 6. Compared with the baseline model, the IMTFF-Networks model performs well in multiple indicators: For the method proposed in Reference 1, although the CNN deep learning method is used, it is lower than the IMTFF-Networks model in all evaluation indicators. This shows that a more complex model structure can perform better classification. Although the accuracy of the method in Reference 2 is slightly higher than that of the IMTFF-Networks model, due to the use of more features in Reference 2, the IMTFF-Networks model performs better in other performance indicators. The results show that appropriate feature complexity can improve the recognition rate of cattle behavior.

[0125] Table 6 Results of different neural network models

[0126]

[0127]

[0128] The present invention proposes a Deep Learning Model for Cattle Behavior Classification Integrating Multimodal Time-Frequency Features, IMTFF-Networ. IMTFF-Networks consists of modules such as multimodal time-frequency fusion of sensor signals and multi-scale time-frequency feature analysis. The multimodal time-frequency fusion module fuses time-frequency domain features through feature modal decomposition and frequency domain processing, and comprehensively captures dynamic characteristics such as frequency components and time patterns of behavior signals, thereby improving the accuracy and robustness of behavior recognition; the multi-scale time-frequency feature analysis module can capture long-term dependencies and complex nonlinear relationships between features at different time scales mainly through multi-scale time series processing and attention mechanism.

[0129] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A deep learning classification method for cattle behavior integrating multimodal time-frequency features, characterized in that: The following steps are involved: Step 1: Collect the animal head and neck muscle acceleration value data set and preprocess the data set; Step 2: Use the FMD module and the FFT module to extract features of the acceleration value respectively, and fuse the two extracted features; Step 3: Input the fused time-frequency features into the multi-scale time-frequency feature analysis network; Step 4: Use the classifier for classification.

2. The cattle behavior deep learning classification method integrating multimodal time-frequency features according to claim 1 is characterized in that: Step 2 specifically includes: Step 21, extracting time domain features from the acceleration signal using the FMD module; Step 22: Extract frequency domain features from the acceleration signal using FFT; Step 23: Use the addition operation to fuse the frequency domain features and the time domain features.

3. The cattle behavior deep learning classification method integrating multimodal time-frequency features according to claim 1 is characterized in that: The multi-scale time-frequency feature analysis network includes: TCN network, adaptive re-weighted attention mechanism module and LSTM network. The time-frequency features pass through the TCN network, adaptive re-weighted attention mechanism module and LSTM network in sequence.

4. The cattle behavior deep learning classification method integrating multimodal time-frequency features according to claim 3 is characterized in that: The formula of the adaptive reweighted attention mechanism module is: s c =σ(W2·δ(W1·z c )) (6) Where T represents the length of the timing signal, x c (t) represents the input signal of the cth channel at the tth time step, z c represents the compressed features of the cth channel, W represents the weight matrix of the fully connected layer, and s c represents the weight of each channel, Represents the recalibrated timing signal.

5. The cattle behavior deep learning classification method integrating multimodal time-frequency features according to claim 1 is characterized in that: Use MLP as a classifier for classification.

6. The cattle behavior deep learning classification method integrating multimodal time-frequency features according to claim 1 is characterized in that: Preprocessing includes: Data merging, data cleaning and denoising, data standardization and data windowing.

7. The cattle behavior deep learning classification method integrating multimodal time-frequency features according to claim 1 is characterized in that: The data set was acquired using a three-axis sensor.

8. The cattle behavior deep learning classification method integrating multimodal time-frequency features according to claim 1 is characterized in that: The Z-score method was used for data cleaning to detect and delete outliers.

9. A cattle behavior deep learning classification system integrating multimodal time-frequency features, characterized by: include: a memory for storing instructions executable by a processor; A processor, used to execute instructions to implement the cattle behavior deep learning classification method integrating multimodal time-frequency features as described in any one of claims 1-8.

10. A computer readable medium storing computer program code, characterized in that: When the computer program code is executed by a processor, the computer program code implements the cattle behavior deep learning classification method integrating multimodal time-frequency features as described in any one of claims 1 to 8.

Images