Sow estrus behavior identification method and system based on multi-modal feature fusion

Abstract

The invention discloses a sow oestrus behavior recognition method and system based on multi-modal feature fusion, and the method comprises the steps: S1, collecting sow monitoring video and audio data, extracting sow oestrus video segment information, matching audio information of a corresponding time segment according to the sow oestrus video segment information, and carrying out the matching of the audio information of the corresponding time segment; preprocessing the data of the video segment information and the audio information to construct a data set; s2, constructing a deep learning algorithm model for sow oestrus behavior recognition by adopting a multi-modal data fusion method; and S3, obtaining a final state prediction result from a to-be-identified data set through the deep learning algorithm model for sow oestrus behavior identification, and obtaining whether the sow is in an oestrus state or not according to the prediction result. According to the invention, accurate identification of the oestrus behavior of the sow is realized.

Description

Technical Field

[0001] The present invention relates to the field of smart breeding technology, and in particular to sow estrus behavior recognition based on multimodal feature fusion. Background Art

[0002] In modern pig production, sows play a major role in breeding and are the source of the entire pig production. The reproductive capacity of sows is also an important indicator of production efficiency. "Estrous identification" is a technology to determine the timing of sow mating. It is the first key link in the entire production cycle of sows and is directly related to whether the sow can be bred normally, whether it can conceive after mating, and the final number of piglets after conception.

[0003] The existing sow estrus monitoring systems are mainly divided into manual monitoring and intelligent monitoring methods. For artificial intelligence monitoring methods, statistical analysis models and machine learning models are used to use multi-source heterogeneous detection data to reveal the changing patterns of various states from the perspective of data. Although deep learning models represented by long short-term memory networks (LSTM) have overcome the problems of traditional machine learning that cannot make long-term predictions and have low prediction accuracy due to gradient disappearance, they have extremely high requirements for the time series and integrity of data. The missed inspections and irregular inspections by operation and maintenance personnel make it impossible for the detection data to be strictly distributed in the time series at fixed intervals. These fault-like fluctuating data will damage its "memory cells", thereby reducing the generalization ability, making it difficult to guarantee the accuracy of the prediction of internal rules.

[0004] Another example: Publication number CN 118097560 A, a sow estrus monitoring method and system, collects sow color video data and sow infrared video data, uses sow estrus prediction results and sow actual estrus results to construct a training data set, uses the training data set to fine-tune the weight parameters in the multi-value weighted prediction, and uses the multi-value weighted prediction after the weight parameters are fine-tuned to predict and monitor the estrus of the sow. However, this monitoring method only focuses on image monitoring, does not effectively utilize audio data, and the monitoring results are inaccurate. Summary of the invention

[0005] The purpose of the present invention is to provide a sow estrus behavior recognition method and system based on multimodal feature fusion, so as to achieve accurate recognition of sow estrus behavior and provide an effective and reliable basis for sow estrus identification.

[0006] In order to achieve one of the above purposes, the present invention adopts the following technical solution:

[0007] A sow estrus behavior recognition method based on multimodal feature fusion comprises the following steps:

[0008] S1: Collect sow monitoring video and audio data, extract sow estrus video segment information, match the audio information of the corresponding time period according to the sow estrus video segment information, pre-process the data of the video segment information and the audio information, and construct a data set;

[0009] S2: A multimodal data fusion method was used to construct a deep learning algorithm model for identifying sow estrus behavior, including:

[0010] S201: Build a DBiLSTM+Attention neural network model;

[0011] S202: Integrate the video and audio data in the data set, and extract the video data feature vector and the audio data feature vector based on the DBiLSTM+Attention neural network model;

[0012] S203: splicing the video data feature vector and the audio data feature vector to form a fused feature vector, and using an LSTM network to learn the complex relationship between the fused feature vectors during fusion, so that the LSTM network can capture the long-term dependency relationship in the sequence data;

[0013] S204:: dynamically adjust the fused feature vector using a gating mechanism, dynamically assign weights of different modal features according to the importance of current input data, and achieve multimodal data fusion to identify sow estrus behavior in different scenarios;

[0014] S3: The data set to be identified is subjected to the deep learning algorithm model for identifying the sow estrus behavior to obtain a final state prediction result, and whether the sow is in estrus is determined based on the prediction result.

[0015] Furthermore, the method further includes S4: using an AP (average precision) method to evaluate the state prediction accuracy of the sow estrus behavior recognition algorithm according to the prediction results, and the AP (average precision) method counts the following indicators in the DBiLSTM+Attention neural network model:

[0016] A. Samples predicted by the model as positive (True)

[0017] B. Samples predicted by the model as negative (False)

[0018] C. True Positives (TP)

[0019] D. Samples that are actually negative but predicted to be positive (False Positives, FP)

[0020] E. True Negatives (TN)

[0021] F. Samples that are actually positive but predicted to be negative (False Negatives, FN);

[0022] On this basis, the recall rate Recall (True Positives Rate, TPR) of the DBiLSTM+Attention neural network model is calculated, and the formula is:

[0023]

[0024] Then calculate the precision Precision, the formula is:

[0025]

[0026] Finally, AP is calculated as:

[0027]

[0028] Among them, r is the threshold, and P(r) is the accuracy under different thresholds.

[0029] Furthermore, the S1: collecting sow monitoring video and audio data, extracting sow estrus video segment information, matching audio information of a corresponding time period according to the sow estrus video segment information, preprocessing the video segment information and the audio information data, and constructing a data set; specifically includes:

[0030] S101: Monitor sows around the clock and clearly capture the detailed video and audio data of sows’ behaviors;

[0031] S102: selecting a video segment during estrus according to the estrus behavior of the sow, and having an expert perform annotation and screening, extracting the estrus start and end time periods and key image frames of estrus behavior from the video segment, removing irrelevant background information from the key image frames, and performing normalization processing;

[0032] S103: performing denoising on the audio data, extracting the denoised audio, extracting corresponding audio information by matching the key image frames with the corresponding time periods, and obtaining a data set of multimodal information;

[0033] S104: Divide the data set into a training set, a validation set and a test set.

[0034] Further, the S102: selecting a video segment of the estrus period according to the estrus behavior of the sow, annotating and screening by an expert, extracting the estrus start and end time periods and key image frames of the estrus behavior from the video segment, removing irrelevant background information from the key image frames, and performing normalization processing, specifically includes:

[0035] First, crop and scale the captured videos to ensure that all videos have the same resolution and frame rate;

[0036] Secondly, the image frames are cropped to remove background information irrelevant to the sow's estrus behavior;

[0037] Then, use image enhancement technology to enhance the data;

[0038] Finally, the cropped image frame is normalized, that is, the image pixel values ​​are adjusted to an appropriate range.

[0039] Furthermore, the test set contains some estrus behavior types that have not appeared in the training set and the validation set to test the generalization ability of the model.

[0040] Furthermore, the S201: constructing a DBiLSTM+Attention neural network model specifically includes:

[0041] The Attention mechanism is introduced into the input layer of the DBiLSTM model, and weights are assigned to the input data through the Attention mechanism. After being processed by multiple BiLSTM layers, the results of multiple feature vectors are integrated through the convolution layer to form a multi-layer bidirectional long short-term memory network DBiLSTM+Attention model based on the attention mechanism. The DBiLSTM+Attention model consists of an input layer, an Attention layer, multiple BiLSTM layers, a convolution layer, and an output layer.

[0042] Furthermore, the S202: integrating the video and audio data in the data set, and extracting the video data feature vector and the audio data feature vector based on the DBiLSTM+Attention neural network model, specifically includes:

[0043] A bidirectional long short-term memory network (DBiLSTM) is used to extract feature vectors from the video and audio data. DBiLSTM can capture the bidirectional dependencies of sequence data, and the Attention mechanism can focus on the important parts of the sequence, thereby improving the accuracy of feature extraction.

[0044] Furthermore, the S202 also includes: using a genetic algorithm (GA) to optimize the video data feature vector and the audio data feature vector extracted by the DBiLSTM+Attention neural network model to form a DBiLSTM+Attention+GA network model feature, and finding the optimal network parameter initialization value through iterative search to improve the prediction performance of the DBiLSTM+Attention neural network model.

[0045] In order to achieve the second of the above objectives, the present invention adopts the following technical solution:

[0046] A sow estrus behavior recognition system based on multimodal feature fusion, including a data acquisition module, a model training module and an analysis and processing module.

[0047] The data acquisition module is used to collect sow monitoring video and audio data, extract sow estrus video segment information, match the corresponding audio information according to the sow estrus video segment information, pre-process the data of the video segment information and the audio information, and construct a data set;

[0048] The model training module is used to construct a deep learning algorithm model for identifying sow estrus behavior using a multimodal data fusion method, specifically including: constructing a DBiLSTM+Attention neural network model; integrating the video and audio data in the data set, and extracting video data feature vectors and audio data feature vectors based on the DBiLSTM+Attention neural network model; splicing the video data feature vectors and audio data feature vectors to form a fused feature vector, and using an LSTM network to learn the complex relationship between each fused feature vector during fusion, and using the LSTM network to capture long-term dependencies in sequence data; using a gating mechanism to dynamically adjust the fused feature vector, dynamically assigning weights of different modal features according to the importance of the current input data, realizing multimodal data fusion, and identifying sow estrus behavior in different scenarios;

[0049] The analysis and processing module is used to obtain a final state prediction result of the data set to be identified through the deep learning algorithm model for identifying sow estrus behavior, and determine whether the sow is in estrus based on the prediction result.

[0050] Furthermore, an evaluation module is included, which is used to evaluate the state prediction accuracy of the sow estrus behavior recognition algorithm according to the prediction results using the AP (average precision) method. The AP (average precision) method counts the following indicators in the DBiLSTM+Attention neural network model:

[0051] A. Samples predicted by the model as positive (True)

[0052] B. Samples predicted by the model as negative (False)

[0053] C. True Positives (TP)

[0054] D. Samples that are actually negative but predicted to be positive (False Positives, FP)

[0055] E. True Negatives (TN)

[0056] F. Samples that are actually positive but predicted to be negative (False Negatives, FN);

[0057] On this basis, the recall rate Recall (True Positives Rate, TPR) of the DBiLSTM+Attention neural network model is calculated, and the formula is:

[0058]

[0059] Then calculate the precision Precision, the formula is:

[0060]

[0061] Finally, AP is calculated as:

[0062]

[0063] Among them, r is the threshold, and P(r) is the accuracy under different thresholds.

[0064] Beneficial effects of the present invention:

[0065] The present invention integrates the collected multimodal data about sows, including sow sound data and video data collected by high-definition cameras, and uses a multi-layer bidirectional long short-term memory network (D-Bidirectional long short-term memory, DBiLSTM) combined with an attention mechanism (Attention) to realize a network model and an image + audio mode. This method is applied to the prediction of sow estrus behavior, which reduces the loss of output content of neuron intermediate nodes, and at the same time obtains the importance of different indicator data for model training. After training, a highly accurate sow estrus recognition model is obtained. The model can accurately determine whether a sow is in estrus and provide detailed estrus probability information, providing strong support for sow breeding management.

[0066] Due to the large amount of computation and long training time caused by large-scale parameter tuning of the DBiLSTM neural network model, a genetic algorithm (GA) was added to initialize the parameters of the DBiLSTM neural network to form the DBiLSTM+Attention+GA network model characteristics. By using the DBiLSTM+Attention+GA network model characteristics to predict sow estrus behavior, the accuracy of sow estrus behavior assessment was further improved.

[0067] The AP (average precision) method was used to evaluate the recognition method, and the effectiveness of the recognition method was verified through experiments, ensuring the accuracy of model monitoring.

[0068] The method of the present invention is also applicable to estrus monitoring of other animals. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flow chart of a specific embodiment 1 of the present invention;

[0070] Figure 2 It is a principle block diagram of specific embodiment 2 of the present invention;

[0071] Figure 3 This is a principle block diagram of an additional evaluation module in specific embodiment 2 of the present invention. DETAILED DESCRIPTION

[0072] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0073] Embodiment 1:

[0074] See also Figure 1 As shown, a sow estrus behavior recognition method based on multimodal feature fusion includes the following steps:

[0075] Step S1: Collect sow monitoring video and audio data, extract sow estrus video segment information, match audio information of the corresponding time period according to the sow estrus video segment information, pre-process the data of the video segment information and the audio information, and construct a data set;

[0076] Specifically include:

[0077] S101: Use a high-definition camera to monitor the sows around the clock, and clearly capture the behavior details of the sows in the video; use a sound collection sensor to accurately capture the sound of the sows, and ensure that the sound data matches the recording time of the high-definition video data to obtain audio;

[0078] S102: Select video segments during estrus according to the estrus behavior of the sow, and have experts perform annotation and screening, extract key image frames of the estrus start and end time periods and estrus behaviors (such as standing estrus and mounting) from the video segments, remove irrelevant background information from the key image frames, and perform normalization processing;

[0079] Compared with the traditional labeling method of manually observing and recording the estrus behavior of sows, this method is not only extremely time-consuming and laborious, but also easily affected by the subjective judgment of the observer, resulting in significant inaccuracies and inconsistencies in the labeling results. In contrast, the method of labeling high-definition video clips by experienced experts has shown obvious advantages. Expert labeling can accurately determine the start and end time of estrus behavior, and accurately identify and classify various estrus behaviors, such as standing estrus, climbing on other sows, etc., thus ensuring high accuracy and reliability of the labeling results. In addition, expert labeling can comprehensively cover the various types of behaviors that may occur during estrus in sows, which provides richer and more detailed feature information for the training of machine learning models.

[0080] Specifically include:

[0081] First, crop and scale the captured videos to ensure that all videos have the same resolution and frame rate;

[0082] Secondly, the image frames are cropped to remove background information irrelevant to the sow's estrus behavior in the picture, so as to reduce the complexity of data processing and improve recognition accuracy;

[0083] Then, use image enhancement technology (rotation, flipping, brightness adjustment) to enhance data and improve the generalization ability of the model;

[0084] Finally, the cropped image frames are normalized, that is, the image pixel values ​​are adjusted to an appropriate range to improve the efficiency and accuracy of subsequent image processing and model training.

[0085] S103: performing denoising on the audio data, extracting the denoised audio, extracting corresponding audio information by matching the key image frames with the corresponding time periods, and obtaining a data set of multimodal information;

[0086] The audio data is denoised to eliminate the potential interference of environmental noise on the recognition results and ensure the clarity and accuracy of the audio data. This helps to more accurately locate and analyze the sound features related to sow estrus behavior, and facilitates subsequent feature extraction and model training. Finally, the audio data is aligned with the video frame according to the timestamp to ensure that the sound and image information can match each other when analyzing the estrus behavior of sows, so as to provide complete and accurate multimodal information, thereby enhancing the accuracy and reliability of estrus behavior recognition.

[0087] S104: Divide the data set into a training set, a validation set and a test set.

[0088] In this specific embodiment: Training set: most of the data in the data set (such as 70%) is selected as the training set for training the model, and it is ensured that the training set contains samples of various estrus behavior types.

[0089] Validation set: A small portion of the data (such as 15%) is selected as the validation set to evaluate the performance of the model during the training process. The validation set should maintain a similar data distribution to the training set.

[0090] Test set: Select the remaining data (such as 15%) as the test set to evaluate the final performance of the model. The test set should contain some estrus behavior types or special cases that have not appeared in the training set and validation set to test the generalization ability of the model.

[0091] The training set is used to train the model and optimize the model parameters through the back propagation algorithm. During the training process, data enhancement techniques (random cropping and rotation) are used to enhance the training data to improve the generalization ability of the model. The trained model is evaluated using the test set, and the accuracy and recall rate indicators are calculated to evaluate the performance of the model.

[0092] S2: A multimodal data fusion method was used to construct a deep learning algorithm model for identifying sow estrus behavior, including:

[0093] Step S201: construct a DBiLSTM+Attention neural network model;

[0094] The Attention mechanism is introduced into the input layer of the DBiLSTM model, and the input data is weighted by the Attention mechanism. After being processed by the multi-layer BiLSTM layer, the results of various feature vectors are integrated by the convolution layer to form a multi-layer bidirectional long short-term memory network DBiLSTM+Attention model based on the attention mechanism. The DBiLSTM+Attention model consists of an input layer, an Attention layer, multiple BiLSTM layers, a convolution layer, and an output layer. See the following table for a multi-layer bidirectional long short-term memory network structure based on the attention mechanism:

[0095]

[0096] The bidirectional long short-term memory network BILSTM is a time-based bidirectional recurrent network that is trained through forward time series and reverse time series. The output data contains information on the entire time series. BILSTM solves the problem of LSTM network lacking context.

[0097] For a time series data, different indicators of the data contribute to model training to different degrees. For a time series data, not every indicator data has the same contribution to the judgment of the sow estrus state. In the traditional sow estrus state detection project without the Attention mechanism, it can only be assumed that each dimension in the data at each moment makes the same contribution to the final state detection judgment; when the Attention mechanism is added to the input layer of the DBiLSTM network model, since the Attention mechanism is a probability weight allocation mechanism that imitates the attention of the human brain. When observing things, the human brain will focus on specific places and ignore other places. With the Attention mechanism, each indicator data in the time series is assigned a weight. Attention highlights more important factors by assigning different probability weights to the input, thereby improving the accuracy of the DBiLSTM+Attention neural network model. In the learning process, the contribution rate of each data indicator to the evaluation of the sow estrus state can be obtained. In addition, introducing the Attention mechanism into the BiLSTM model can improve the prediction effect of the model under a certain probability.

[0098] The DBiLSTM+Attention neural network model is a deep learning algorithm model based on multimodal data fusion. It is designed to process and analyze the audio and video multimodal data of sow estrus behavior. The DBiLSTM+Attention neural network model uses the collected sow audio and video data as the core recognition basis and efficiently processes the fused multimodal data through a deep neural network.

[0099] Step S202: Integrate the video and audio data in the data set, and extract the video data feature vector and the audio data feature vector based on the DBiLSTM+Attention neural network model;

[0100] Specifically include:

[0101] A bidirectional long short-term memory network (DBiLSTM) is used to extract feature vectors from the video and audio data. DBiLSTM can capture the bidirectional dependencies of time series data, and the Attention mechanism can focus on the important parts of the sequence, thereby improving the accuracy of feature vector extraction.

[0102] Possibly, the S202 also includes: initializing the network parameters of the DBiLSTM+Attention neural network model using a genetic algorithm (GA), finding the optimal initialization value of the network through iterative search, and thereby improving the prediction performance of the DBiLSTM+Attention neural network model.

[0103] The advantages of this genetic algorithm include the following three aspects: first, it has good global search capabilities, and through crossover, mutation and other means, it can prevent falling into the local optimal solution, thereby obtaining the global optimal solution; second, it has good compatibility and can be combined with other algorithms for optimization; third, compared with general optimization problems, it has lower requirements on mathematics, and does not need to establish conditions such as objective functions, but only needs to solve the problem. Based on the above three advantages, the genetic algorithm can effectively optimize the recurrent neural network (RNN, LSTM, BiLSTM and DBiLSTM), thereby improving the model prediction performance.

[0104] Step S203: splicing the video data feature vector and the audio data feature vector to form a fused feature vector, integrating data information of different modalities together to provide rich input for subsequent model training, and using the LSTM network to learn the complex relationship between the fused feature vectors during fusion. The LSTM network can capture the long-term dependency in the sequence data, which helps the model to understand the estrus behavior of the sow more accurately;

[0105] Step S204: dynamically adjust the fused feature vector using a gating mechanism, dynamically assign weights of different modal features according to the importance of current input data, and achieve multimodal data fusion to identify sow estrus behavior in different scenarios.

[0106] The deep learning algorithm model for identifying sow estrus behavior is implemented using Python programming under the Pytorch framework. Other important parameters in the DBiLSTM+Attention neural network model include:

[0107] (1) The learning rate is mainly used to control the learning progress of the model, which is equivalent to the step size of the network weight adjustment. If the learning rate is too small, the network learning time will be greatly increased. However, if the learning rate is increased blindly in order to shorten the training time of the network model, it may cause the network to oscillate near the optimal value, or even fail to converge and find the optimal result. Therefore, the adjustment of the learning rate is crucial. Generally, the learning rate of 0.01-0.001 is appropriate. The learning rate needs to be continuously adjusted during the training process, and the best prediction result of the model is obtained by observing the loss value of the prediction result. The final learning rate of the model is 0.005.

[0108] (2) The number of iterations is the number of times the network weights are adjusted according to the learning rate. The optimal solution for the neural network parameters is found through continuous iterative adjustment. If the number of iterations is too small, the model may stop training before it reaches the best training effect. However, if the number of iterations is increased blindly, the model training time will be too long, resulting in a waste of resources. During the training process, the number of model iterations should also be continuously adjusted according to the model loss value in order to obtain the best prediction results in the shortest time. The final number of model iterations is 1000.

[0109] (3) The loss value is used to quantify whether the model's prediction results are accurate. The loss value function in the model uses the mean square error function, which compares the state results predicted by the model with the actual state results of the corresponding parts. The results are used to measure the prediction effect of the model. The training process also adjusts the key parameters of the model by continuously observing the model loss value until acceptable results are obtained at the training point.

[0110] S3: The data set to be identified is passed through the trained deep learning algorithm model for identifying the sow estrus behavior to obtain the final state prediction result, and whether the sow is in estrus is determined according to the prediction result.

[0111] Possibly, it also includes S4: using an AP (average precision) method to evaluate the state prediction accuracy of the sow estrus behavior recognition algorithm according to the prediction results, wherein the AP (average precision) method counts the following indicators in the DBiLSTM+Attention neural network model:

[0112] A. Samples predicted by the model as positive (True)

[0113] B. Samples predicted by the model as negative (False)

[0114] C. True Positives (TP)

[0115] D. Samples that are actually negative but predicted to be positive (False Positives, FP)

[0116] E. True Negatives (TN)

[0117] F. Samples that are actually positive but predicted to be negative (False Negatives, FN);

[0118] On this basis, the recall rate Recall (True Positives Rate, TPR) of the DBiLSTM+Attention neural network model is calculated, and the formula is:

[0119]

[0120] Then calculate the precision Precision, the formula is:

[0121]

[0122] Finally, AP is calculated as:

[0123]

[0124] Among them, r is the threshold, and P(r) is the accuracy under different thresholds. Specific embodiment 2:

[0126] See also Figure 2 As shown, a sow estrus behavior recognition system based on multimodal feature fusion includes a data acquisition module 1, a model training module 2 and an analysis and processing module 3.

[0127] The data acquisition module 1 is used to collect sow monitoring video and audio data, extract sow estrus video segment information, match the corresponding audio information according to the sow estrus video segment information, pre-process the data of the video segment information and the audio information, and construct a data set;

[0128] The model training module 2 is used to construct a deep learning algorithm model for identifying sow estrus behavior using a multimodal data fusion method, specifically including: constructing a DBiLSTM+Attention neural network model; integrating the video and audio data in the data set, and extracting video data feature vectors and audio data feature vectors based on the DBiLSTM+Attention neural network model; splicing the video data feature vector and the audio data feature vector to form a fused feature vector, integrating data information of different modalities together, providing rich input for subsequent model training, and using the LSTM network to learn the complex relationship between each fused feature vector during fusion. The LSTM network can capture long-term dependencies in sequence data, which helps the model to more accurately understand the estrus behavior of sows; using a gating mechanism to dynamically adjust the fused feature vector, dynamically assigning weights of different modal features according to the importance of the current input data, realizing multimodal data fusion, and identifying the estrus behavior of sows in different scenarios;

[0129] The analysis and processing module 3 is used to obtain a final state prediction result of the data set to be identified through the deep learning algorithm model of sow estrus behavior identification, and determine whether the sow is in estrus according to the prediction result.

[0130] See also Figure 3 As shown, it is possible to further include an evaluation module 4, which is used to use an AP (average precision) method to evaluate the state prediction accuracy of the sow estrus behavior recognition algorithm according to the prediction results. The AP (average precision) method counts the following indicators in the DBiLSTM+Attention neural network model:

[0131] A. Samples predicted by the model as positive (True)

[0132] B. Samples predicted by the model as negative (False)

[0133] C. True Positives (TP)

[0134] D. Samples that are actually negative but predicted to be positive (False Positives, FP)

[0135] E. True Negatives (TN)

[0136] F. Samples that are actually positive but predicted to be negative (False Negatives, FN);

[0137] On this basis, the recall rate Recall (True Positives Rate, TPR) of the DBiLSTM+Attention neural network model is calculated, and the formula is:

[0138]

[0139] Then calculate the precision Precision, the formula is:

[0140]

[0141] Finally, AP is calculated as:

[0142]

[0143] Among them, r is the threshold, and P(r) is the accuracy under different thresholds.

[0144] The functions of the data acquisition module 1, the model training module 2 and the analysis and processing module 3 in the system are similar to the sow estrus behavior recognition method based on multimodal feature fusion in specific embodiment 1, so this specific embodiment is omitted here.

[0145] The technical solution provided by the present invention is described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A sow estrus behavior recognition method based on multimodal feature fusion, characterized in that: The steps include: S1: Collect sow monitoring video and audio data, extract sow estrus video segment information, match the audio information of the corresponding time period according to the sow estrus video segment information, pre-process the data of the video segment information and the audio information, and construct a data set; S2: A multimodal data fusion method was used to construct a deep learning algorithm model for identifying sow estrus behavior, including: S201: Build a DBiLSTM+Attention neural network model; S202: Integrate the video and audio data in the data set, and extract the video data feature vector and the audio data feature vector based on the DBiLSTM+Attention neural network model; S203: splicing the video data feature vector and the audio data feature vector to form a fused feature vector, and using an LSTM network to learn the complex relationship between the fused feature vectors during fusion, so that the LSTM network can capture the long-term dependency relationship in the sequence data; S204:: dynamically adjust the fused feature vector using a gating mechanism, dynamically assign weights of different modal features according to the importance of current input data, and achieve multimodal data fusion to identify sow estrus behavior in different scenarios; S3: The data set to be identified is subjected to the deep learning algorithm model for identifying the sow estrus behavior to obtain a final state prediction result, and whether the sow is in estrus is determined based on the prediction result.

2. The sow estrus behavior recognition method based on multimodal feature fusion according to claim 1, characterized in that: The method further includes S4: using an AP (average precision) method to evaluate the state prediction accuracy of the sow estrus behavior recognition algorithm according to the prediction results, wherein the AP (average precision) method counts the following indicators in the DBiLSTM+Attention neural network model: A. Samples predicted by the model as positive (True) B. Samples predicted by the model as negative (False) C. True Positives (TP) D. Samples that are actually negative but predicted to be positive (False Positives, FP) E. True Negatives (TN) F. Samples that are actually positive but predicted to be negative (False Negatives, FN); On this basis, the recall rate Recal l (True Positives Rate, TPR) of the DBiLSTM+Attention neural network model is calculated, and its formula is: Then calculate the precision Precision, the formula is: Finally, AP is calculated as: Among them, r is the threshold, and P(r) is the accuracy under different thresholds.

3. The sow estrus behavior recognition method based on multimodal feature fusion according to claim 1, characterized in that: The S1: collecting sow monitoring video and audio data, extracting sow estrus video segment information, matching audio information of the corresponding time period according to the sow estrus video segment information, pre-processing the video segment information and the audio information data, and constructing a data set; specifically includes: S101: Monitor sows around the clock and clearly capture the detailed video and audio data of sows’ behaviors; S102: selecting a video segment during estrus according to the estrus behavior of the sow, and having an expert perform annotation and screening, extracting the estrus start and end time periods and key image frames of estrus behavior from the video segment, removing irrelevant background information from the key image frames, and performing normalization processing; S103: performing denoising on the audio data, extracting the denoised audio, extracting corresponding audio information by matching the key image frames with the corresponding time periods, and obtaining a data set of multimodal information; S104: Divide the data set into a training set, a validation set and a test set.

4. The method for identifying sow estrus behavior based on multimodal feature fusion according to claim 3, characterized in that: S102: selecting a video segment during estrus according to the estrus behavior of the sow, annotating and screening it by an expert, extracting the estrus start and end time periods and key image frames of estrus behavior from the video segment, removing irrelevant background information from the key image frames, and performing normalization processing, specifically including: First, crop and scale the captured videos to ensure that all videos have the same resolution and frame rate; Secondly, the image frames are cropped to remove background information irrelevant to the sow's estrus behavior; Then, use image enhancement technology to enhance the data; Finally, the cropped image frame is normalized, that is, the image pixel values ​​are adjusted to an appropriate range.

5. The sow estrus behavior recognition method based on multimodal feature fusion according to claim 3 is characterized in that: The test set includes some estrus behavior types that have not appeared in the training set and the validation set to test the generalization ability of the model.

6. The sow estrus behavior recognition method based on multimodal feature fusion according to claim 1, characterized in that: The S201: constructing a DBiLSTM+Attention neural network model specifically includes: The Attention mechanism is introduced into the input layer of the DBiLSTM model, and weights are assigned to the input data through the Attention mechanism. After being processed by multiple BiLSTM layers, the results of multiple feature vectors are integrated through the convolution layer to form a multi-layer bidirectional long short-term memory network DBiLSTM+Attention model based on the attention mechanism. The DBiLSTM+Attention model consists of an input layer, an Attention layer, multiple BiLSTM layers, a convolution layer, and an output layer.

7. The sow estrus behavior recognition method based on multimodal feature fusion according to claim 1, characterized in that: The S202: integrating the video and audio data in the data set, and extracting the video data feature vector and the audio data feature vector based on the DBiLSTM+Attention neural network model, specifically includes: A bidirectional long short-term memory network (DBiLSTM) is used to extract feature vectors from the video and audio data. DBiLSTM can capture the bidirectional dependencies of sequence data, and the Attention mechanism can focus on the important parts of the sequence, thereby improving the accuracy of feature extraction.

8. The sow estrus behavior recognition method based on multimodal feature fusion according to claim 1 or 7, characterized in that: The S202 also includes: using a genetic algorithm (GA) to optimize the video data feature vector and the audio data feature vector extracted by the DBiLSTM+Attention neural network model to form a DBiLSTM+Attention+GA network model feature, and finding the optimal network parameter initialization value through iterative search to improve the prediction performance of the DBiLSTM+Attention neural network model.

9. A sow estrus behavior recognition system based on multimodal feature fusion, characterized in that: It includes data acquisition module, model training module and analysis and processing module. The data acquisition module is used to collect sow monitoring video and audio data, extract sow estrus video segment information, match the corresponding audio information according to the sow estrus video segment information, pre-process the data of the video segment information and the audio information, and construct a data set; The model training module is used to construct a deep learning algorithm model for identifying sow estrus behavior using a multimodal data fusion method, specifically including: constructing a DBiLSTM+Attention neural network model; integrating the video and audio data in the data set, and extracting video data feature vectors and audio data feature vectors based on the DBiLSTM+Attention neural network model; splicing the video data feature vectors and audio data feature vectors to form a fused feature vector, and using an LSTM network to learn the complex relationship between each fused feature vector during fusion, and using the LSTM network to capture long-term dependencies in sequence data; using a gating mechanism to dynamically adjust the fused feature vector, dynamically assigning weights of different modal features according to the importance of the current input data, realizing multimodal data fusion, and identifying sow estrus behavior in different scenarios; The analysis and processing module is used to obtain a final state prediction result of the data set to be identified through the deep learning algorithm model for identifying sow estrus behavior, and determine whether the sow is in estrus based on the prediction result.

10. The sow estrus behavior recognition system based on multimodal feature fusion according to claim 9, characterized in that: It also includes an evaluation module, which is used to evaluate the state prediction accuracy of the sow estrus behavior recognition algorithm according to the prediction results using the AP (average precision) method. The AP (average precision) method counts the following indicators in the DBiLSTM+Attention neural network model: A. Samples predicted by the model as positive (True) B. Samples predicted by the model as negative (False) C. True Positives (TP) D. Samples that are actually negative but predicted to be positive (False Positives, FP) E. True Negatives (TN) F. Samples that are actually positive but predicted to be negative (False Negatives, FN); On this basis, the recall rate Recal l (True Positives Rate, TPR) of the DBiLSTM+Attention neural network model is calculated, and its formula is: Then calculate the precision Precision, the formula is: Finally, AP is calculated as: Among them, r is the threshold, and P(r) is the accuracy under different thresholds.

Images