Cow oestrus state prediction method based on multi-source data fusion and neural network enhancement

Abstract

The invention discloses a dairy cow oestrus state prediction method based on multi-source data fusion and neural network enhancement, and the method comprises the following steps: (1), carrying out the precise definition of key oestrus features based on the quantitative analysis of the oestrus behaviors of dairy cows, and building oestrus recognition features with remarkable indication; (2) performing multi-modal processing on the multivariate oestrus characteristic data of the dairy cow, and capturing a characteristic complex mapping relation; (3) capturing a time sequence relation between the features by using serial fusion, capturing an incidence relation of the features in the same time step by using parallel fusion, and outputting a fusion feature vector by combining serial and parallel; (4) inputting the feature vectors into a hidden Markov model improved by a neural network, and outputting estrus probability distribution; and (5) introducing an estrus probability averaging mechanism, avoiding single-time-step misjudgment, enhancing judgment accuracy, and finally outputting a dairy cow estrus state result. According to the method, a multi-modal feature data processing mode is innovatively introduced, feature data information is comprehensively captured by utilizing a fusion framework, and the accuracy and robustness of final oestrus recognition can be ensured by utilizing a neural network improved hidden Markov model and combining an oestrus probability averaging mechanism.

Description

Technical Field

[0001] The present invention defines innovative features, provides effective quantitative indicators for estrus detection, constructs multi-modal feature fusion based on multi-source estrus data of dairy cows, and predicts the starting moment of dairy cow estrus through an improved hidden Markov estrus recognition model based on a neural network. It involves multi-modal fusion strategies of data, the state transition matrix and emission probability matrix of the improved hidden Markov model based on a neural network, and an average framework for estrus probability. Background Art

[0002] The estrus detection of dairy cows is a key link in breeding management and is of great significance for improving reproductive efficiency and optimizing production management. Traditional estrus detection methods mainly rely on manual observation or monitoring of physiological characteristics. However, manual observation has problems such as strong subjectivity, dependence on experience, and high labor intensity, making the automation of estrus detection an urgent problem to be solved.

[0003] Currently, automated estrus detection technologies mainly rely on single features, such as behavioral features, acoustic features, or physiological data. Although these methods have improved the detection efficiency to a certain extent, due to factors such as individual differences of dairy cows and environmental interference, single features often cannot provide stable and accurate estrus prediction, resulting in insufficient accuracy of detection results, and there is a lack of effective methods for quantitative analysis of estrus characteristics, and existing features are difficult to accurately describe the estrus degree of dairy cows.

[0004] Existing research is still relatively limited in multi-feature fusion. Different types of data often have complementarity. For example, behavioral features can reflect the activity status of dairy cows, acoustic features can capture their emotional changes, and physiological data can provide internal physiological indicators. However, current estrus detection methods lack effective multi-modal fusion strategies and cannot make full use of multi-source information to improve detection performance.

[0005] In addition, traditional machine learning models have problems of insufficient accuracy and robustness in the modeling process and are difficult to adapt to complex multi-modal data. Although deep learning methods can extract features, they often lack time series modeling capabilities, resulting in insufficient utilization of time series information for estrus recognition. Summary of the Invention

[0006] Aiming at the above defects or improvement requirements of the existing technology, the present invention provides innovative characteristic indicators for quantifying the estrus degree of dairy cows and a method for predicting the estrus state of dairy cows based on multi-source data fusion and neural network enhancement. The purpose is to fuse multi-source estrus features, fully capture the information association between different estrus feature data, and improve the accuracy and robustness of the existing estrus detection model, so as to accurately predict the starting moment of dairy cow estrus and solve the problem of the lack of effective and reliable estrus detection means in existing modern pastures.

[0007] To solve the problem of the lack of effective quantitative indicators for dairy cow estrus characteristics, the present invention provides two estrus indicators (social behavior index and rumination variation index) based on the high correlation of dairy cow estrus behaviors, including the following steps:

[0008] (1) Social behavior index

[0009] By analyzing the active contact behaviors (such as licking each other, butting, touching) of dairy cows during estrus, defining the contact intensity (light, moderate, severe), and combining the contact time and strength, the activity degree of social behavior is quantified. For the first time, the strength and time of contact behaviors are combined to form a comprehensive scoring mechanism, which can more accurately reflect the behavioral changes of dairy cows during estrus.

[0010] (2) Rumination variation index

[0011] By analyzing the fluctuation of dairy cow rumination behaviors, extracting the time series characteristics of rumination behaviors (such as signal amplitude vector and signal amplitude area), and combining the clustering analysis method, the changes of rumination behaviors are quantified. Introducing the dynamic change index of rumination behaviors can capture the subtle changes of rumination behaviors during estrus and provide a more detailed quantitative analysis for estrus detection.

[0012] To solve the problem of insufficient integration of multi-dimensional estrus data, the present invention proposes a brand-new multi-modal fusion strategy to integrate multi-source data from sound, acceleration, position, and video, ensuring the accurate capture of estrus characteristics in temporal changes and the effective extraction of information at the same time step.

[0013] (1) Multi-modal framework

[0014] Through standardization, principal component analysis, Fourier transform, statistical feature extraction, and semi-tensor product processing, multi-modal processing is performed on multi-source data to extract complex interaction information. A variety of different processing methods are constructed into multi-modal data, and at the same time, semi-tensor product is introduced to extract complex non-linear feature relationships, further enhancing the feature expression ability.

[0015] (2) Serial and parallel fusion

[0016] Serial fusion: Sequentially concatenate multi-modal features and use the LSTM network to process time series information to capture temporal changes.

[0017] Parallel fusion: Through the self-attention mechanism, calculate the weights of each modal feature, and generate a unified feature vector after weighted summation to ensure the effective integration of information at the same time step.

[0018] Combining serial fusion and parallel fusion not only retains the continuity of temporal information but also enhances the interaction ability of multi-modal features.

[0019] To improve the accuracy and robustness of estrus detection, the present invention proposes an improved Hidden Markov Model based on neural network, and combines an estrus probability averaging mechanism to accurately capture the starting moment of estrus.

[0020] (1) State transition matrix: Generate a dynamic state transition matrix through a feedforward neural network to accurately model the transition law of dairy cow estrus states at different times.

[0021] (2) Emission probability matrix: Generate an emission probability matrix through a feedforward neural network to describe the complex correlation between observed features and latent states.

[0022] (3) Estrus probability averaging mechanism: Introduce the average value of estrus probability with weighted time steps to smooth short-term fluctuations, avoid misjudgment at a single time step, and improve the stability and accuracy of detection.

[0023] Combining the neural network with the Hidden Markov Model enhances the model's ability to model complex temporal changes. Through the estrus probability averaging mechanism, combined with exponential decay weights, the importance of the most recent time steps is increased, improving the sensitivity to capture estrus states.

[0024] The present invention solves the problems of lack of quantitative indicators, insufficient data integration, and low detection accuracy in the prior art by constructing strong estrus-indicative feature indicators, proposing a multi-modal fusion strategy, and improving the Hidden Markov Model and the estrus probability averaging mechanism. The new method has the following advantages:

[0025] 1. Quantify the degree of estrus: Provide objective quantitative indicators through the social behavior index and the rumination variation index.

[0026] 2. Multi-dimensional data integration: Make full use of multi-source data through the multi-modal fusion strategy to improve the feature expression ability.

[0027] 3. Accurately detect the starting moment of estrus: Improve the accuracy and robustness of detection through the improved Hidden Markov Model and the estrus probability averaging mechanism. Description of the Drawings

[0028] Figure 1 is the flowchart for predicting the estrus state of dairy cows

[0029] Figure 2 is the social behavior index graph

[0030] Figure 3 is the rumination variation index graph

[0031] Figure 4 is the multi-modal fusion framework graph

[0032] Figure 5 is the graph of the improved Hidden Markov Model based on neural network Detailed implementation manners

[0033] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0034] Method embodiment:

[0035] A method for predicting the estrus state of dairy cows based on multi-source data fusion and neural network enhancement of the present invention is divided into three parts. First, using the typical estrus behaviors of dairy cows, estrus strongly indicative characteristic indexes are constructed. Secondly, a multi-modal fusion strategy is proposed, and multi-dimensional feature data including the newly constructed estrus indexes is processed through multi-modal processing to obtain a fusion feature expression containing complex interaction information of the features. Finally, the improved hidden Markov estrus detection model based on the neural network processes and the estrus probability averaging method calculates to obtain the accurate starting time of the estrus of the dairy cows. Thereby solving the three problems that the modern dairy cow estrus detection lacks effective quantitative estrus degree indexes, lacks multi-dimensional estrus data integration means, and is difficult to accurately detect the starting time of estrus.

[0036] In order to solve the problem of lack of effective quantitative indexes for the estrus degree of dairy cows, the present invention proposes two estrus indexes (social behavior index and rumination change index) based on the high correlation of dairy cow estrus behaviors, including the following steps:

[0037] (1) Social behavior index:

[0038] Define social behavior as active contact behavior and introduce a classification standard for contact intensity. Contact types such as mutual licking, butting and touching all belong to one contact, and the contact intensity is divided into three levels of mild, moderate and severe according to the contact force and contact duration ( Figure 1 ).

[0039] Contact force:

[0040] Define the mild contact force as a small contact force, without obvious pushing or pressure; define the moderate contact force as a medium contact force, accompanied by a small amount of pushing or pressure; define the severe contact force as a large contact force, accompanied by obvious pushing or pressure, resulting in the movement or strong reaction of the other dairy cow.

[0041] Contact time:

[0042] Define the mild contact time as the time t≤2 seconds; define the moderate contact time as the time 2 seconds < t≤5 seconds; define the severe contact time as the time t>5 seconds

[0043] Comprehensive judgment rule:

[0044] Mild contact: The contact force is mild and the duration t≤2 seconds

[0045] Moderate contact: The contact force is moderate, or the duration is 2 s < t ≤ 5 s

[0046] Severe contact: The contact force is severe, or the duration t > 5 s

[0047] Mild is scored 1 point, moderate is scored 2 points, and severe is scored 3 points. The calculation formula for the social behavior index is:

[0048] Social behavior index = total weighted score / time window length (hours)

[0049] (2) Rumination variation index:

[0050] Develop a rumination variation index to capture the fluctuations of rumination behavior and provide a more detailed quantitative analysis for estrus detection ( Figure 2 ).

[0051] First, extract the time series features of rumination behavior from the acceleration data, including the signal magnitude vector (SMV) and signal magnitude area (SMA) of rumination behavior. SMV is used to quantify the intensity of rumination behavior at each moment, and the calculation formula is as follows:

[0052]

[0053] SMA is used to measure the overall activity level of rumination behavior, and the calculation formula is as follows:

[0054] SMA = |accx| + |accy| + |accz|

[0055] Segment the data using a 5-second non-overlapping window and extract 50 features from five time series (accx, accy, accz, SMV, SMA). These features include time domain features (maximum, minimum, standard deviation, amplitude, and signal energy) and frequency domain features (the amplitudes of the top five frequency components extracted by fast Fourier transform).

[0056] Next, through the k-means clustering algorithm, divide the rumination behavior into three activity levels: high rumination level, medium rumination level, and low rumination level, and calculate the rumination activity level per hour. The calculation formula is as follows:

[0057] RuminatingActivity Level for hourt = (HighLevel × 0.9) + (MidLevel × 0.1)

[0058] To further quantify the changes in rumination behavior, we propose a rumination variation index (RVI), and the calculation formula is:

[0059]

[0060] Among them, rt represents the rumination activity level at the current hour, r t-72 , r t-48 and r t-24 are the rumination activity levels in the previous 72 hours, 48 hours, and 24 hours respectively. By comparing the rumination activity level at the current hour with that in the same time period of the past few days, the RVI effectively quantifies the change in rumination behavior.

[0061] To address the lack of integration means for multi-dimensional estrus characteristics in modern automated dairy cow estrus detection, the present invention provides a multi-modal fusion strategy, including the following steps:

[0062] To improve the model's ability to integrate multi-dimensional data and extract feature information, the model processes multi-source data from sound, acceleration, position, and video through five modalities (normalization, principal component analysis, Fourier transform, statistical feature extraction, semi-tensor product), and fuses the multi-modal features through serial and parallel fusion strategies to ensure the precise capture of dairy cow estrus characteristics in temporal changes and the effective extraction of information at the same time step ( Figure 3 ).

[0063] (1) Multi-modal framework

[0064] Normalization processing, which converts the data into a data transformation with a mean of zero and a variance of one, is expressed by the following formula:

[0065]

[0066] Principal component analysis, which reduces the data dimension and extracts the main components, is expressed by the following formula:

[0067] X PCA = X·W

[0068] Fourier transform, which converts the time-domain signal to the frequency domain and extracts spectral features, is expressed by the following formula:

[0069] X FFT = FFT(X)

[0070] Statistical feature extraction, which calculates statistical features based on the data and describes the distribution characteristics of the data, is expressed by the following formula:

[0071]

[0072] Semi-tensor product processing, which processes the data through the semi-tensor product and extracts complex non-linear feature relationships, is expressed by the following formula:

[0073] X semi-tensor = X*W

[0074] Among them, X represents the specific value of the i-th data; μ represents the average value of all data, which is used to describe the central tendency of the data; σ represents the standard deviation of the data, which measures the degree of data dispersion, that is, the distribution of data points around the mean; W represents the weight matrix, which is used for linear transformation; FFT(·) represents the fast Fourier transform.

[0075] (2) Multimodal serial fusion

[0076] After the multi-dimensional features are processed by the modality, they are integrated using the serial-parallel fusion strategy. Serial fusion concatenates the features after multimodal processing in sequence and uses the LSTM network to process the time series information.

[0077] ① Concatenation of modality features:

[0078]

[0079] Each modality feature is T is the time step 1, d i is the feature dimension

[0080] ② Processing by the LSTM network:

[0081] H LSTM = LSTM(X concat )

[0082] The output of LSTM is LSTM(X concat ), h is the hidden layer dimension of LSTM, which is set to 2048.

[0083] ③ Output of time step:

[0084]

[0085] (3) Multimodal parallel fusion:

[0086] Each modality feature is mapped to a unified dimension through a linear projection layer, which represents the importance of the modality at the current moment. By calculating the self-attention weights, each modality feature is weighted and summed to be fused into a unified feature vector.

[0087] ① Linear transformation of the features of each modality:

[0088]

[0089] is the representation of each modality feature, each modality projection matrix, d o is the output dimension, which is 1024.

[0090] ② Calculate the attention weights, and use a linear layer to calculate the attention weights of each modality:

[0091]

[0092] are the learned attention parameters.

[0093] ③ Perform weighted summation on the projection results of each modality to obtain the final fusion representation:

[0094]

[0095] α i is the attention weight of each modality (range 0-1), and Z i is the projected modality feature.

[0096] (4) Serial-parallel result fusion:

[0097] The serial fusion and parallel fusion outputs are concatenated in series to generate a feature vector as the input to the state transition matrix and emission probability matrix modules.

[0098] H combined = W s H final + W p H parallel

[0099] where W s and W p are linear transformation matrices used to adjust the outputs of serial and parallel fusions. Finally, the serial-parallel result fusion outputs a 3072-dimensional feature vector.

[0100] To improve the accuracy and robustness of dairy cow estrus detection, the present invention provides an improved hidden Markov model based on a neural network and an estrus probability averaging mechanism, and the specific steps are as follows:

[0101] (1) State transition matrix:

[0102] The state transition probability matrix is a key module in the algorithm for capturing the changing characteristics of dairy cow's sequential estrus. The feature vector after feature fusion is directly passed to the feedforward neural network, and the dynamic state transition matrix generated through hierarchical mapping and non-linear transformation can accurately model the transition law between hidden variable states ( Figure 4 ).

[0103] A t = softmax(W2·σ(W1·x t + b1)+ b2)

[0104] A t,ij = P(s t+1 = j|s t = i)

[0105] W1 and W2 are weight matrices, N is the number of hidden layer states, b1 and b2 are bias terms, and σ(·) represents the activation function. This matrix describes the probability distribution of the state transitioning from I at the current time step t to state j at the next time step, capturing the complex dynamic change patterns of the dairy cow estrus state between different moments.

[0106] (2) Emission probability matrix:

[0107] The emission probability matrix is a key module in the algorithm for describing the relationship between observed features and latent states. Based on the state transition, the fused feature vector is further passed to another feedforward neural network to generate the emission probability matrix.

[0108] B t = softmax(W4·σ(W3·h t + b3)+ b4)

[0109] B t,ij = P(o t = j|s t = i)

[0110] h t is the hidden state, W3 and W4 are weight matrices, and b3 and b4 are bias terms. This matrix characterizes the probability distribution of observing the current estrus feature j when the state is i at time step t, and is used to accurately model the complex association between the dairy cow estrus state and multi-dimensional features.

[0111] Use the fused features to generate the estrus state at the next moment through the state transition matrix, and then calculate the probability distribution of the current observed features through the emission probability matrix. After iterative calculations over multiple time steps, the model outputs the state probability distribution at each time step.

[0112] z = softmax(W z ·h T + b z )

[0113] W z and b z are the weights and biases, and h T is the last hidden state.

[0114] (3) Estrus probability averaging mechanism:

[0115] At each time step, the model calculates and outputs the current estrus state probability distribution z, representing the probability of the dairy cow being in different estrus states at that time step. To more accurately identify the estrus period and determine the starting moment of the estrus period, the model introduces a weighted time step estrus probability averaging mechanism to judge whether the dairy cow has truly entered the estrus period.

[0116] ① At each time step t, according to the output state probability distribution P_t, the probability value representing the estrus state at the current time step.

[0117] ② In this study, N = 5 is set, that is, the data of the nearest 5 time steps are considered. To increase the emphasis on the current time step, an exponential decay function is used to assign weights, and the weights are set such that w5 > w4 > w3 > w2 > w1.

[0118]

[0119] Among them, α controls the decay rate, α > 0, w i is the weight of the i-th time step, which is used to ensure that the sum of weights is 1.

[0120] ③ For the weighted average estrus probability P of the nearest N time steps avg If it exceeds the set threshold, it is judged to be in the estrus period.

[0121]

[0122] ④ The formula for setting the threshold is:

[0123] Threshold = P mean + k × P std

[0124] Among them: P mean is the average estrus probability of historical data; P std is the standard deviation of the estrus probability of historical data; k is a constant used to adjust the sensitivity of the threshold.

[0125] This judgment mechanism can increase the emphasis on the nearest time step, thereby enhancing the ability to sensitively capture the estrus state. In addition, by weighting the estrus probability, short-term fluctuations are smoothed, avoiding misjudgments in a single time step, making the overall judgment more stable and accurate.

Claims

1. A method for predicting the estrus status of dairy cows based on multi-source data fusion and neural network enhancement, characterized in that, The steps include: (1) Based on the interaction and rumination between dairy cows, two behaviors that have significant differences in performance during the estrus period, two new indicators for defining the estrus characteristics of dairy cows were constructed, namely the social behavior index and the rumination change index. (2) The estrus data of dairy cows containing multivariate features are processed through different modalities to obtain a multimodal feature representation after five modal processing, where the multimodality includes: standardization processing, principal component analysis, Fourier transform, statistical feature extraction and semi-tensor product processing. (3) The feature representations after multimodal processing are first connected in series and in parallel to capture the temporal relationship between features and the correlation between features at the same time step, respectively. Finally, the parallel hybrid fusion is performed to output the final feature vector. (4) The improved hidden Markov model based on neural network is used to process the mixed fusion feature vectors to capture the temporal relationship and state transition rules of estrus data. (5) The probability distribution output by the hidden Markov model is processed using the estrus probability averaging mechanism to improve the robustness and accuracy of estrus recognition.

2. The method for predicting the estrus state of dairy cows based on multi-source data fusion and neural network enhancement according to claim 1, wherein, The step (1) standardizes the multi-source feature data respectively to ensure that the scales of different features are consistent; PCA principal component analysis is performed to reduce redundant features and reduce dimensionality; Fourier transform, which reveals the periodicity and frequency characteristics of a signal; Statistical feature extraction, extracting the core statistical indicators such as the mean, standard deviation, maximum value, minimum value, etc. of the feature extraction data; semi-tensor product processing, matrix operation expansion, and mining the hidden nonlinear relationship between estrus characteristics.

3. The index social behavior index for constructing the dairy cow estrus characteristic data as described in claim 1, wherein, Social behavior is defined as interaction between cows through body or head contact. Each time a cow actively touches another cow with its trunk or head, it is considered a social behavior. This interaction includes behaviors such as touching, licking, or head contact, and usually occurs in a group. By counting the number of social behaviors that occur per hour, it is possible to quantify the level of social activity in dairy cows and use it as one of the potential indicators of estrus status.

4. The index of rumination variation for constructing the data defining the estrus characteristics of dairy cows according to claim 1, wherein, The rumination change rate extracts the time series characteristics of rumination behavior from the acceleration data, including the signal magnitude vector (SMV) and signal magnitude area (SMA) of rumination behavior. Through segmentation processing and feature extraction, combined with clustering algorithms, rumination behavior is divided into different activity levels, and the hourly rumination activity level is calculated. The change of rumination behavior is further quantified by the rumination change index (RVI), and the calculation formula is as follows: where, r t represents the rumination activity level of the current hour, r t-72 , r t-48 and r t-24 are the rumination activity levels of the previous 72 hours, 48 hours, and 24 hours respectively. By comparing the rumination activity level of the current hour with that of the same time period in the past few days, the rumination change index effectively quantifies the change in rumination behavior.

5. The multimodal serial fusion and parallel fusion according to claim 1, characterized in that, The specific steps of the method are as follows: after the multi-dimensional features are processed modally, they are integrated using a serial-parallel fusion strategy. Serial fusion connects the multi-modal processed features in series and processes the time series information with the help of an LSTM network; parallel fusion calculates the weight of each modal feature and performs weighted summation through linear projection and self-attention mechanism to generate a unified feature vector, which can ensure the capture of the long-distance effect of the multi-dimensional features and the feature correlation relationship at the same time step.

6. The multimodal hybrid fusion method according to claim 1, wherein The method has the following specific steps: fusing the output results of serial fusion and parallel fusion to generate a high-dimensional feature vector, and adjusting the output weights of serial and parallel fusion through linear transformation, and finally outputting a unified feature representation.

7. The neural network enhanced hidden Markov method according to claim 1, wherein The feature vector is passed to a feedforward neural network, and a state transition probability matrix and an emission probability matrix are generated through hierarchical mapping and non-linear transformation. The state transition matrix is used to describe the dynamic change pattern of the estrus state of dairy cows at different times, and the emission probability matrix is used to model the complex association between the estrus state of dairy cows and multi-dimensional features.

8. The estrus probability averaging method according to claim 1, wherein The specific steps of the method are as follows: At each time step, according to the state probability distribution output by the hidden Markov model, the weighted average estrus probability of the last N time steps is calculated. Weights are assigned through an exponential decay function, and the calculation formula is as follows:

9. Threshold setting for the estrus probability averaging method according to claim 7, characterized in that, Specific steps: ①Collect historical data and count the estrus probability P at each time step t . ② Calculate the average value P of the estrus probability of historical data mean and the standard deviation P std . ③According to the statistical distribution, set the threshold as: Threshold = P mean + k + P std .

Images