Nondestructive testing method for estrus of breeding pigs based on cross-modal feature fusion
Through the detection method based on cross-modal feature fusion, combined with thermal infrared images and audio data, the problems of strong subjectivity and low accuracy of sow estrus detection are solved, and high-precision estrus detection is achieved, which significantly improves the efficiency of fertility rate and breeding management.
Patent Information
- Application Number
- CN202411684080.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-03-18
AI Technical Summary
The existing sow estrus detection methods have problems with strong subjectivity and low accuracy, resulting in a low pregnancy rate.
The non-destructive detection method of estrus in breeding pigs based on cross-modal feature fusion is adopted. By obtaining the thermal infrared image and audio data of breeding pigs, data augmentation, frequency domain filtering and wavelet denoising technology are carried out, and detection is combined with a multimodal neural network detection model.
High-precision estrus detection was achieved, with the detection accuracy reaching 98.92%, the recall rate was 98.92%, and the F1-score was 97.35%, which significantly improved the pregnancy rate and the efficiency of breeding management.
Smart Images
Figure CN119649403A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of pig breeding and farming, and in particular to a non-destructive detection method for estrus of breeding pigs based on cross-modal feature fusion. Background Art
[0002] Estrus detection of sows is an important part of farm production management. Timely and accurate detection of estrus of sows helps determine the best time for mating, thereby significantly improving conception rate and litter size and optimizing reproductive efficiency. Effective estrus detection can shorten the empty period of sows, reduce feed and management costs, and avoid production losses caused by missing the estrus period. In addition, estrus detection can also help identify reproductive problems, take timely measures, reduce the risk of disease transmission, and improve the health level of the pig herd. Accurate estrus detection is an important means to achieve scientific and standardized management of farms, which helps to improve overall economic and production benefits.
[0003] The main methods of traditional sow estrus identification are external observation, boar estrus testing and artificial estrus testing. However, traditional sow estrus identification methods have several obvious limitations. First, these methods mainly rely on the subjective observation and judgment of technicians, which are easily affected by individual experience and subjective consciousness, resulting in inconsistent judgment standards and misjudgment of estrus status. Secondly, these methods mainly rely on observing the behavioral performance of sows, such as standing still, climbing, etc., and these behaviors may be affected by factors such as environment and individual differences, lacking objectivity and reliability. Therefore, sow estrus identification based on traditional methods often has the problem of insufficient accuracy, resulting in a lower conception rate.
[0004] Method content
[0005] The present application aims to solve the problems of strong subjectivity and low accuracy in the existing sow estrus detection, and to provide a non-destructive detection method for sow estrus based on cross-modal feature fusion.
[0006] The technical solution adopted to achieve the purpose of this application is:
[0007] A non-destructive detection method for estrus of breeding pigs based on cross-modal feature fusion, comprising the following steps:
[0008] Step 1, obtaining thermal infrared images and audio of breeding pigs;
[0009] Step 2, performing data enhancement on the thermal infrared image and audio of the breeding pig obtained in step 1 respectively;
[0010] Step 3, performing frequency domain filtering on the audio data after data enhancement, normalizing and standardizing the thermal infrared image after data enhancement, and using wavelet denoising technology to process the standardized thermal infrared image;
[0011] Step 4, inputting the thermal infrared image and audio of the sow processed in step 3 into a pre-trained multimodal neural network detection model for detection to determine whether the sow is in estrus.
[0012] In the above technical solution, the step 2 comprises the following steps:
[0013] S2.1: Rotate, scale, translate, horizontally flip and adjust the contrast of the acquired thermal infrared image of the breeding pig;
[0014] S2.2: Perform time scaling, noise addition, volume adjustment and spectrum transformation on the acquired sow audio.
[0015] In the above technical solution, the process of processing the standardized thermal infrared image using wavelet denoising technology includes the following steps:
[0016] S3.1.1: Perform a two-dimensional discrete wavelet transform on the standardized thermal infrared image to decompose it into wavelet coefficients of different scales and directions;
[0017] S3.1.2: Perform threshold processing on the decomposed wavelet coefficients to remove noise;
[0018] S3.1.3: Perform inverse wavelet transform on the wavelet coefficients after threshold processing to reconstruct the denoised thermal infrared image.
[0019] In the above technical solution, the expression of the two-dimensional discrete wavelet transform is as follows:
[0020]
[0021] Where W j (x,y) represents the wavelet coefficient of the jth layer, ψ j (x, y) is the two-dimensional wavelet basis function with scale j, and f(m, n) is the pixel value of the original image;
[0022] The expression of the threshold processing is as follows:
[0023]
[0024] Where λ is the threshold, W j,k are the original wavelet coefficients, is the processed wavelet coefficient;
[0025] The expression of the inverse wavelet transform is as follows:
[0026]
[0027] Where f(x,y) represents the processed wavelet coefficients, Represents the two-dimensional wavelet basis function of scale j, j is the scale level, and m and n represent the position index of the wavelet coefficients respectively.
[0028] In the above technical solution, the frequency domain filtering of the audio data after data enhancement includes the following steps:
[0029] S3.2.1: Use fast Fourier transform to convert the time domain signal of the sow audio into the frequency domain signal;
[0030] S3.2.2: construct a Chebyshev type I bandpass filter, and apply the Chebyshev type I bandpass filter to the frequency domain signal for filtering;
[0031] S3.2.3: Convert the filtered frequency domain signal back to the time domain signal by inverse fast Fourier transform.
[0032] In the above technical solution, the expression of the fast Fourier transform is as follows:
[0033]
[0034] Where X(f) is the frequency domain signal, x(n) is the time domain signal, N is the number of samples of the signal, and j is the imaginary unit;
[0035] The construction formula of the Chebyshev type I bandpass filter is as follows:
[0036]
[0037] Where H(s) is the filter transfer function, the Chebyshev filter fluctuation factor ∈ and the Chebyshev polynomial T N Determines the characteristics of the filter. Frequency ω and passband frequency ω p Determines the frequency response of the filter;
[0038] The specific expression of the IFFT is as follows:
[0039]
[0040] Where Y(k) represents the frequency domain signal after filtering, y(n) represents the corresponding time domain signal, N represents the number of samples of the signal, and j is the imaginary unit.
[0041] In the above technical solution, the cutoff frequency of the Chebyshev I-type bandpass filter is 150 Hz to 2 kHz, the order of the filter is 5th order, the sampling frequency is 800 Hz, and the allowable ripple size is 0.5 db.
[0042] In the above technical solution, the construction of the Chebyshev type I bandpass filter includes the following steps:
[0043] S3.3.1: Use Python's scipy.signal library to calculate the coefficients of a Chebyshev type I bandpass filter.
[0044] S3.3.2: Generate the frequency response of a Chebyshev type I bandpass filter using the sosfreqz function.
[0045] S3.3.3: Use the sosfilt function to apply the calculated Chebyshev type I bandpass filter coefficients to the audio data for filtering.
[0046] In the above technical solution, the step 4 specifically includes the following steps:
[0047] S4.1: construct a multimodal neural network detection model, input the thermal infrared image and audio of the breeding pig after data enhancement into the multimodal learning model for pre-training and testing, and obtain a pre-trained multimodal neural network detection model;
[0048] S4.2: Use the pre-trained multimodal neural network detection model to detect estrus of the breeding pigs and determine whether the breeding pigs are in estrus.
[0049] In the above technical solution, the pre-training process of the multimodal neural network detection model includes:
[0050] S4.1.1: Implement thermal infrared image feature extraction through the custom VisionTransformerFeatureExtractor class;
[0051] S4.1.2: Implement audio feature extraction through a custom convolutional neural network model for audio feature extraction;
[0052] S4.1.3: Implement adaptive cross-attention feature fusion of thermal infrared image features and audio features through a custom CrossAttentionLayer class:
[0053] S4.1.4: Introduce the Squeeze-and-Excitation Block module in DenseNet, and train the fused features by improving the DenseNet backbone network.
[0054] The beneficial effects of the present invention are as follows:
[0055] 1. This application proposes a method for detecting estrus in sows based on cross-modal feature fusion. It combines audio and thermal infrared image data and performs detection through the designed APO-CVIT model. The model integrates Vision Transformer and convolutional neural network, which can efficiently extract multimodal data features, with a detection accuracy of 98.92%, a recall rate of 98.92%, and an F1-score of 97.35%. Compared with traditional detection methods, this model can more accurately identify the estrus status of sows.
[0056] 2. This application realizes the feature fusion of audio and thermal infrared images through an adaptive cross-attention mechanism, improves the model's ability to learn multimodal information, makes estrus detection more objective and reliable, reduces the errors caused by human judgment, and provides an efficient and non-destructive means of estrus detection, greatly improving the efficiency and accuracy of breeding management. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0058] Figure 1 Shown is a flow chart of the non-destructive detection method for estrus of breeding pigs based on cross-modal feature fusion of the present invention.
[0059] Figure 2 Shown is a schematic diagram of a thermal infrared image of a pig according to the present invention.
[0060] Figure 3 Shown is a schematic diagram of audio data of the present invention.
[0061] Figure 4 Shown is a schematic diagram of thermal infrared image feature extraction of the present invention.
[0062] Figure 5 Shown is a schematic diagram of audio data feature extraction according to the present invention.
[0063] Figure 6 Shown is a schematic diagram of the adaptive cross-attention feature fusion of the present invention.
[0064] Figure 7 Shown is a schematic diagram of the (SE) Block module introduced in the present invention.
[0065] Figure 8 Shown is a schematic diagram of the detection results of the multimodal neural network detection model (APO-CVIT model) of the present invention. DETAILED DESCRIPTION
[0066] In order to enable those skilled in the art to better understand the present method, the technical solution of the present method is further described below in conjunction with specific embodiments.
[0067] See also Figure 1 A non-destructive detection method for estrus of sows based on cross-modal feature fusion includes the following steps:
[0068] Step 1, obtain the thermal infrared image and audio of the sow; wherein the thermal infrared image and audio of the sow can be found in Figure 2 and Figure 3 .
[0069] Furthermore, the thermal infrared image in step 1 is obtained by using an infrared detector to detect the intensity of infrared rays emitted from the buttocks area of the breeding pig, and converting it into an electrical signal, and then processing the electrical signal to generate a thermal infrared image; the audio is obtained by collecting the calls of the breeding pig.
[0070] Step 2: perform data enhancement on the thermal infrared image and audio of the breeding pig obtained in step 1.
[0071] Furthermore, the step 2 comprises the following steps:
[0072] S2.1: Rotate, scale, translate, horizontally flip and adjust the contrast of the acquired thermal infrared image of the breeding pig.
[0073] S2.2: Perform time scaling, noise addition, volume adjustment and spectrum transformation on the acquired sow audio.
[0074] Step 3, frequency domain filtering is performed on the audio data after data enhancement to remove audio background noise, retain the characteristic frequency of the estrus sound of the sow, normalize and standardize the thermal infrared image after data enhancement, and use wavelet denoising technology to process the standardized thermal infrared image to reduce the interference of hair and stains and make the image clearer. The use of wavelet denoising technology to process the standardized thermal infrared image includes the following steps:
[0075] S3.1.1: Perform a two-dimensional discrete wavelet transform on the standardized thermal infrared image to decompose it into wavelet coefficients of different scales and directions. The expression of the two-dimensional discrete wavelet transform is as follows:
[0076]
[0077] Where W j (x,y) represents the wavelet coefficient of the jth layer, ψ j (x, y) is the two-dimensional wavelet basis function of scale j, and f(m, n) is the pixel value of the original image.
[0078] S3.1.2: Perform threshold processing on the decomposed wavelet coefficients (high-frequency noise (such as hair and stains) of the thermal infrared image) to remove noise, thereby removing these interference factors while retaining the main features and contours of the thermal infrared image. The expression of the threshold processing is as follows:
[0079]
[0080] Where λ is the threshold, W j,k are the original wavelet coefficients, are the processed wavelet coefficients.
[0081] S3.1.3: Perform inverse wavelet transform on the wavelet coefficients after threshold processing to reconstruct the denoised thermal infrared image. The expression of the inverse wavelet transform is as follows:
[0082]
[0083] Where f(x,y) represents the processed wavelet coefficients, Represents the two-dimensional wavelet basis function of scale j, j is the scale level, and m and n represent the position index of the wavelet coefficients respectively.
[0084] The frequency domain filtering of the audio data after data enhancement comprises the following steps:
[0085] S3.2.1: Use Fast Fourier Transform (FFT) to convert the time domain signal of the sow audio into a frequency domain signal. The expression of the Fast Fourier Transform is as follows:
[0086]
[0087] Where X(f) is the frequency domain signal, x(n) is the time domain signal, N is the number of samples of the signal, and j is the imaginary unit.
[0088] S3.2.2: Construct a Chebyshev I-type bandpass filter, and apply the Chebyshev I-type bandpass filter to the frequency domain signal for filtering. The cutoff frequency of the Chebyshev I-type bandpass filter is 150 Hz to 2 kHz, the order of the filter is 5th order, the sampling frequency is 800 Hz, and the allowed ripple size is 0.5 db; the construction formula of the Chebyshev I-type bandpass filter is as follows:
[0089]
[0090] Where H(s) is the filter transfer function, the Chebyshev filter fluctuation factor ∈ and the Chebyshev polynomial T N Determines the characteristics of the filter. Frequency ω and passband frequency ω pDetermines the frequency response of the filter.
[0091] Specifically, the construction of the Chebyshev type I bandpass filter includes the following steps:
[0092] S3.3.1: Use Python’s scipy.signal library to compute the coefficients of a Chebyshev type I bandpass filter.
[0093] S3.3.2: Generate the frequency response of a Chebyshev type I bandpass filter using the sosfreqz function.
[0094] S3.3.3: Use the sosfilt function to apply the calculated Chebyshev type I bandpass filter coefficients to the audio data for filtering.
[0095] S3.2.3: Convert the filtered frequency domain signal back to the time domain signal through inverse fast Fourier transform (IFFT). The specific expression of the IFFT is as follows:
[0096]
[0097] Where Y(k) represents the filtered frequency domain signal, y(n) represents the corresponding time domain signal, N represents the number of samples of the signal, and j is the imaginary unit.
[0098] Step 4, inputting the thermal infrared image and audio of the sow processed in step 3 into a pre-trained multimodal neural network detection model for detection to obtain the result of the sow estrus.
[0099] The step 4 specifically comprises the following steps:
[0100] S4.1: Construct a multimodal neural network detection model (APO-CVIT model), input the thermal infrared image and audio of the breeding pig after data enhancement into the multimodal learning model for pre-training and testing, and obtain a pre-trained multimodal neural network detection model;
[0101] The pre-training process of the multimodal neural network detection model includes:
[0102] S4.1.1: Implement thermal infrared image feature extraction through the customized VisionTransformerFeatureExtractor class.
[0103] For details, see Figure 4, since the Transformer model itself has no position information, the position information needs to be explicitly added to the patch embedding. To this end, positional embeddings are introduced, and these embedding vectors are added to the patch embeddings to preserve the position information of the patch in the original image. The processed patch embedding sequence is input into the Transformer encoder. The Transformer encoder consists of multiple Transformer encoder layers, which contain a self-attention mechanism and a feed-forward neural network.
[0104] Self-Attention Mechanism: Calculate the attention weight of each patch with other patches to capture global information. This process is implemented through the query, key, and value matrices, and the representation of each patch will be updated to the weighted sum of all patches. The self-attention calculation formula is as follows:
[0105]
[0106] Where Q represents the query matrix, K represents the key matrix, V represents the value matrix, and d k Represents the dimension of the key matrix, which is used to scale the dot product attention to prevent the dot product result from being too large and causing the softmax function to saturate.
[0107] Feed-Forward Neural Network (FFN): After each self-attention layer, it contains two linear transformation layers and a ReLU activation function for further processing and transforming the patch representation. The specific formula of the feed-forward neural network is as follows:
[0108] FFN(x)=ReLU(xW1+b1)W2+b2
[0109] In the formula, x represents the input vector, W1 and W2 represent the weight matrices of linear transformation, b1 and b2 represent bias terms, and ReLU represents the activation function.
[0110] S4.1.2: See Figure 5 , audio feature extraction is achieved through a custom convolutional neural network (CNN) model for audio feature extraction;
[0111] S4.1.3: See Figure 6 , the adaptive cross-attention feature fusion of thermal infrared image features and audio features is realized through the custom CrossAttentionLayer class:
[0112] S4.1.4: Introducing the Squeeze-and-Excitation (SE) Block module in DenseNet (Densely Connected Network) (see Figure 7 ), and train the fused features by improving the DenseNet backbone network.
[0113] S4.2: Use the pre-trained multimodal neural network detection model to detect estrus of the breeding pigs and determine whether the breeding pigs are in estrus.
[0114] See also Figure 8 It can be seen that the detection effect of the non-destructive detection method of estrus in breeding pigs based on cross-modal feature fusion is good.
[0115] The above is a preferred implementation mode of the present invention and should not be regarded as any form of limitation on the present invention. The preferred embodiments and related technologies in the present invention have been described above, but they do not limit the present invention. For relevant technical personnel familiar with the technical field, without departing from the scope of the technical principles of the present invention, the technical principles described above can be used to make some improvements or substitutions, including modifications and equivalent changes based on the core content of the present invention, all of which are within the scope of protection of the present invention.
Claims
1. A non-destructive detection method for estrus of sows based on cross-modal feature fusion, characterized in that: The following steps are involved: Step 1, obtaining thermal infrared images and audio of breeding pigs; Step 2, performing data enhancement on the thermal infrared image and audio of the breeding pig obtained in step 1 respectively; Step 3, performing frequency domain filtering on the audio data after data enhancement, normalizing and standardizing the thermal infrared image after data enhancement, and using wavelet denoising technology to process the standardized thermal infrared image; Step 4, inputting the thermal infrared image and audio of the sow processed in step 3 into a pre-trained multimodal neural network detection model for detection to determine whether the sow is in estrus.
2. The nondestructive detection method for estrus of sows according to claim 1, characterized in that: The step 2 comprises the following steps: S2.1: Rotate, scale, translate, horizontally flip and adjust the contrast of the acquired thermal infrared image of the breeding pig; S2.2: Perform time scaling, noise addition, volume adjustment and spectrum transformation on the acquired sow audio.
3. The nondestructive detection method for estrus of sows according to claim 1, characterized in that: The process of processing the standardized thermal infrared image using wavelet denoising technology comprises the following steps: S3.1.1: Perform a two-dimensional discrete wavelet transform on the standardized thermal infrared image to decompose it into wavelet coefficients of different scales and directions; S3.1.2: Perform threshold processing on the decomposed wavelet coefficients to remove noise; S3.1.3: Perform inverse wavelet transform on the wavelet coefficients after threshold processing to reconstruct the denoised thermal infrared image.
4. The nondestructive detection method for estrus of sows according to claim 3, characterized in that: The expression of the two-dimensional discrete wavelet transform is as follows: Where W j (x,y) represents the wavelet coefficient of the jth layer, ψ j (x, y) is the two-dimensional wavelet basis function with scale j, and f(m, n) is the pixel value of the original image; The expression of the threshold processing is as follows: Where λ is the threshold, W j,k are the original wavelet coefficients, is the processed wavelet coefficient; The expression of the inverse wavelet transform is as follows: Where f(x,y) represents the processed wavelet coefficients, Represents the two-dimensional wavelet basis function of scale j, j is the scale level, and m and n represent the position index of the wavelet coefficients respectively.
5. The nondestructive detection method for estrus of sows according to claim 1, characterized in that: The frequency domain filtering of the audio data after data enhancement comprises the following steps: S3.2.1: Use fast Fourier transform to convert the time domain signal of the sow audio into the frequency domain signal; S3.2.2: construct a Chebyshev type I bandpass filter, and apply the Chebyshev type I bandpass filter to the frequency domain signal for filtering; S3.2.3: Convert the filtered frequency domain signal back to the time domain signal by inverse fast Fourier transform.
6. The non-destructive detection method for estrus of sows according to claim 5, characterized in that: The expression of the fast Fourier transform is as follows: Where X(f) is the frequency domain signal, x(n) is the time domain signal, N is the number of samples of the signal, and j is the imaginary unit; The construction formula of the Chebyshev type I bandpass filter is as follows: Where H(s) is the filter transfer function, the Chebyshev filter fluctuation factor ∈ and the Chebyshev polynomial T N Determines the characteristics of the filter. Frequency ω and passband frequency ω p Determines the frequency response of the filter; The specific expression of the IFFT is as follows: Where Y(k) represents the filtered frequency domain signal, y(n) represents the corresponding time domain signal, N represents the number of samples of the signal, and j is the imaginary unit.
7. The nondestructive detection method for estrus of sows according to claim 5, characterized in that: The cutoff frequency of the Chebyshev type I bandpass filter is 150 Hz to 2 kHz, the order of the filter is 5th order, the sampling frequency is 800 Hz, and the allowed ripple size is 0.5 db.
8. The non-destructive detection method for estrus of sows according to claim 5, characterized in that: The construction of the Chebyshev type I bandpass filter comprises the following steps: S3.3.1: Use Python's scipy.signal library to calculate the coefficients of a Chebyshev type I bandpass filter. S3.3.2: Generate the frequency response of a Chebyshev type I bandpass filter using the sosfreqz function. S3.3.3: Use the sosfilt function to apply the calculated Chebyshev type I bandpass filter coefficients to the audio data for filtering.
9. The nondestructive detection method for estrus of sows according to claim 1, characterized in that: The step 4 specifically comprises the following steps: S4.1: construct a multimodal neural network detection model, input the thermal infrared image and audio of the breeding pig after data enhancement into the multimodal learning model for pre-training and testing, and obtain a pre-trained multimodal neural network detection model; S4.2: Use the pre-trained multimodal neural network detection model to detect the estrus of the breeding pigs and obtain the result of whether the breeding pigs are in estrus.
10. The non-destructive detection method for estrus of sows according to claim 9, characterized in that: The pre-training process of the multimodal neural network detection model includes: S4.1.1: Implement thermal infrared image feature extraction through the custom VisionTransformerFeatureExtractor class; S4.1.2: Implement audio feature extraction through a custom convolutional neural network model for audio feature extraction; S4.1.3: Implement adaptive cross-attention feature fusion of thermal infrared image features and audio features through a custom CrossAttentionLayer class: S4.1.4: Introduce the Squeeze-and-Excitation Block module in DenseNet, and train the fused features by improving the DenseNet backbone network.