Intelligent live pig behavior recognition system and method based on multi-modal data fusion

Abstract

The invention relates to the technical field of animal husbandry, and discloses an intelligent live pig behavior recognition system based on multi-modal data fusion, which comprises a data acquisition module, a data preprocessing module, a feature extraction module, a multi-modal fusion module, a behavior recognition module, a data storage and management module and a user interaction module. According to the intelligent live pig behavior recognition system and method based on multi-modal data fusion, the limitation of a single-modal recognition method and the defect that audio information can supplement visual information are overcome by combining data of two modals of video and audio, the recognition accuracy is improved, and meanwhile, by using an improved AVSlowFast model and a GCT channel attention mechanism, the recognition accuracy is improved. The system can effectively fuse visual and auditory features, and further enhance the recognition capability in a complex environment, so that the system can accurately recognize and classify different behaviors of live pigs under various breeding conditions, and the recognition reliability is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of animal husbandry, and in particular to a system and method for intelligently identifying pig behaviors based on multimodal data fusion. Background Art

[0002] Pig farming is an important part of the agricultural industry. The health status and behavior patterns of pigs have a direct impact on the breeding benefits. In traditional pig farming management, farmers mainly judge the behavior and health status of pigs through manual observation. This method not only consumes a lot of manpower and time, but also has low accuracy and is easily affected by subjective factors.

[0003] With the continuous development of science and technology, computer vision and audio processing technologies are gradually applied to the field of pig behavior recognition. However, most of the existing technologies have certain limitations. For example, the recognition method based on a single visual modality is difficult to accurately identify pig behavior under poor lighting conditions and when pigs occlude each other. The method based on audio analysis can only identify some behaviors with obvious auditory characteristics, and the monitoring range is limited. In addition, the current multimodal fusion method is not accurate enough in fusion strategy, and fails to give full play to the complementary advantages of visual and auditory information, resulting in the accuracy and comprehensiveness of behavior recognition to be improved. Therefore, it is of great practical significance to develop an efficient and accurate pig behavior intelligent recognition system and method based on multimodal data fusion. Therefore, we propose a pig behavior intelligent recognition system and method based on multimodal data fusion. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the existing technology, the present invention provides a system and method for intelligent identification of pig behavior based on multimodal data fusion, which has the advantages of integrating video and audio data, deeply mining visual and auditory features, and realizing accurate identification and comprehensive monitoring of pig behavior, solving the problems of low recognition accuracy and limited monitoring range of the existing technology in complex breeding environments.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose of integrating video and audio data, deeply mining visual and auditory features, and realizing accurate identification and comprehensive monitoring of pig behavior, the present invention provides the following technical solutions: a pig behavior intelligent identification system based on multimodal data fusion, comprising: a data acquisition module, a data preprocessing module, a feature extraction module, a multimodal fusion module, a behavior recognition module, a data storage and management module, and a user interaction module, the data acquisition module comprises a high-definition camera and a high-sensitivity microphone, the data preprocessing module comprises a video preprocessing submodule and an audio preprocessing submodule, and the feature extraction module comprises a visual feature extraction submodule and an auditory feature extraction submodule.

[0008] Preferably, the high-definition camera has a frame rate of 30 frames per second and a resolution of 1920×1080 pixels. It is installed in a pig house with a horizontal range of 0-360 degrees and a vertical range of -45 degrees to 45 degrees, and is arranged at 120-150 degrees with the high-sensitivity microphone; the high-sensitivity microphone has a sampling rate of 44.1kHz and is installed no more than 2 meters away from the pig activity area.

[0009] Preferably, the video preprocessing submodule uses a two-stage denoising algorithm, first median filtering and then wavelet transform threshold denoising, and crops the video based on the preset area of ​​interest according to the pig body shape distribution probability model. The cropped area can be adjusted within 30%-80% of the original picture, and the adjustment time interval does not exceed 1 hour, with an accuracy of 5% of the picture; the audio preprocessing submodule uses an adaptive bandpass filter to reduce noise, and converts the audio signal into a Log-mel spectrum diagram with each frame length of 25ms.

[0010] Preferably, the visual feature extraction submodule uses the improved AVSlowFast model video fast and slow channels, the slow channel collects static features at 4 frames / second, and the fast channel collects dynamic features at 32 frames / second, and introduces a spatial pyramid pooling structure. The auditory feature extraction submodule uses the AVSlowFast model audio channel, and adopts a multi-scale convolution kernel combination of [3×1, 1×3], [5×1, 1×5] and [7×1, 1×7] and a specific time step to extract audio features.

[0011] Preferably, the multimodal fusion module adopts an adaptive weighted fusion strategy combined with the GCT channel attention mechanism, and the GCT calculation includes the GCA calculation of global context information, and the formula is:

[0012]

[0013] It also includes normalization processing and GCE calculation of channel weights, fusion time and space calibration, time synchronization error does not exceed 0.05 seconds, and spatial alignment accuracy is pixel level.

[0014] Preferably, the data storage and management module uses a distributed relational database, data sharding storage, primary key and foreign key association, incremental backup once an hour, full backup once every 7 days, off-site redundant storage, and an optimization period of no more than 7 days.

[0015] Preferably, the feature extraction module uses the PCA algorithm to reduce the dimension after extracting the features, and the dimension retention rate is not less than 80%. The system has self-learning and adaptive capabilities, optimizes the model every 15 days, and detects and repairs hardware faults within 5 minutes during operation.

[0016] A method for intelligently identifying pig behaviors based on multimodal data fusion, comprising an intelligently identifying pig behaviors based on multimodal data fusion, and further comprising the following steps:

[0017] S1. Data collection: Use high-definition cameras and high-sensitivity microphones to collect video and audio data from pig houses;

[0018] S2, data preprocessing: video data is subjected to two-stage denoising and cropping of the region of interest preset according to the body model; audio data is subjected to adaptive bandpass filtering and denoising and then converted into a Log-mel spectrum graph;

[0019] S3, Feature Extraction: Extract features from the preprocessed video and audio data respectively. For video, the improved AVSlowFast model is used, combined with the spatial pyramid pooling structure to obtain static and dynamic features; for audio, the audio channel of the AVSlowFast model is used, and the multi-scale convolution kernel combination and specific time step are used to extract audio features;

[0020] S4, multimodal fusion: the visual and auditory features are fused through an adaptive weighted fusion strategy combined with the GCT channel attention mechanism. This process includes calculating global context information, normalization processing, and calculating channel weights to ensure the accuracy of spatiotemporal calibration;

[0021] S5. Behavior recognition: Use a deep neural network classification model with a multi-layer residual structure to recognize pig behavior;

[0022] S6. Data storage and management: Use a distributed relational database, shard storage, primary key and foreign key association, incremental backup once an hour, full backup once every 7 days, off-site redundant storage, optimization cycle no more than 7 days, system self-learning and adaptive optimization every 15 days, and hardware failure detection and repair within 5 minutes during operation;

[0023] S7. User interaction: The identified behavior results are displayed to the user in a visual manner and corresponding management suggestions are provided. At the same time, the system can automatically record abnormal behaviors and trigger alarms to notify management personnel to respond in a timely manner.

[0024] (III) Beneficial effects

[0025] Compared with the prior art, the present invention provides a pig behavior intelligent recognition system and method based on multimodal data fusion, which has the following beneficial effects:

[0026] 1. The intelligent pig behavior recognition system and method based on multimodal data fusion overcomes the limitations of single-modality recognition methods by combining data from both video and audio modalities. In particular, in complex situations such as poor lighting conditions and pigs occluding each other, audio information can supplement the lack of visual information and improve recognition accuracy. At the same time, by using the improved AVSlowFast model and GCT channel attention mechanism, the system can effectively integrate visual and auditory features, further enhancing recognition capabilities in complex environments. This enables the system to accurately identify and classify different behaviors of pigs under various breeding conditions, greatly improving recognition reliability.

[0027] 2. The intelligent pig behavior recognition system and method based on multimodal data fusion, through the use of high-definition cameras and high-sensitivity microphones, can not only cover a larger monitoring range, but also capture more types of pig behaviors. For example, audio data can be used to identify subtle behavioral characteristics such as the screams of piglets competing for milk and the humming of sows when they let milk, which may not be easy to detect in single visual monitoring. In addition, the system can monitor the behavior of pigs 24 hours a day, defining daily activities as behaviors other than the five categories of breastfeeding, eating, arching the fence, drinking water, and sleeping, thereby realizing comprehensive monitoring of pig behavior and effectively making up for the shortcomings of traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a top view of the installation of the data acquisition device of the present invention;

[0029] Figure 2 A three-dimensional schematic diagram of the installation of the data acquisition device of the present invention;

[0030] Figure 3 A schematic diagram of the audio-visual characteristics of pig behavior of the present invention;

[0031] Figure 4 It is a schematic diagram of the behavior recognition network architecture of the present invention;

[0032] Figure 5 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the embodiments of the present invention and the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Embodiment 1:

[0035] A pig behavior intelligent recognition system based on multimodal data fusion, comprising: a data acquisition module, a data preprocessing module, a feature extraction module, a multimodal fusion module, a behavior recognition module, a data storage and management module and a user interaction module, wherein the data acquisition module comprises a high-definition camera and a high-sensitivity microphone, the data preprocessing module comprises a video preprocessing submodule and an audio preprocessing submodule, and the feature extraction module comprises a visual feature extraction submodule and an auditory feature extraction submodule.

[0036] A method for intelligently identifying pig behaviors based on multimodal data fusion, comprising an intelligently identifying pig behaviors system based on multimodal data fusion, and further comprising the following steps:

[0037] S1. Data collection: Use high-definition cameras and high-sensitivity microphones to collect video and audio data from pig houses;

[0038] S2, data preprocessing: video data is subjected to two-stage denoising and cropping of the region of interest preset according to the body model; audio data is subjected to adaptive bandpass filtering and denoising and then converted into a Log-mel spectrum graph;

[0039] S3, Feature Extraction: Extract features from the preprocessed video and audio data respectively. For video, the improved AVSlowFast model is used, combined with the spatial pyramid pooling structure to obtain static and dynamic features; for audio, the audio channel of the AVSlowFast model is used, and the multi-scale convolution kernel combination and specific time step are used to extract audio features;

[0040] S4, multimodal fusion: the visual and auditory features are fused through an adaptive weighted fusion strategy combined with the GCT channel attention mechanism. This process includes calculating global context information, normalization processing, and calculating channel weights to ensure the accuracy of spatiotemporal calibration;

[0041] S5. Behavior recognition: Use a deep neural network classification model with a multi-layer residual structure to recognize pig behavior;

[0042] S6. Data storage and management: Use a distributed relational database, shard storage, primary key and foreign key association, incremental backup once an hour, full backup once every 7 days, off-site redundant storage, optimization cycle no more than 7 days, system self-learning and adaptive optimization every 15 days, and hardware failure detection and repair within 5 minutes during operation;

[0043] S7. User interaction: The identified behavior results are displayed to the user in a visual manner and corresponding management suggestions are provided. At the same time, the system can automatically record abnormal behaviors and trigger alarms to notify management personnel to respond in a timely manner.

[0044] Embodiment 2:

[0045] As shown in the attached figure of the specification Figure 1 and Figure 2 As shown in the figure, the number of pigs in the pig house is 8 to 12. Hikvision 2CD3386FWDV2-I camera and Hikvision DS-2FP021 microphone are used to collect on-site video and audio data respectively. The camera is installed in the pig house at a height of 1.8m from the ground to shoot the scene in the pig house from above. The microphone is connected to the camera. The test environment and equipment installation position are as shown in the figure. Figure 2 As shown; during the behavior recognition process, the camera and the microphone record the pig's activities for 24 hours, and the DS-7804N-K1 hard disk recorder is used to store the video and audio; the stored video frame rate is 30 frames / s, the resolution is 1920×1080 pixels, the audio bit rate is 128kbps, and the sampling rate is 44.1kHz.

[0046] According to the behavioral characteristics of pigs, when constructing the data set, their behaviors were divided into six categories: breastfeeding, eating, drinking, sleeping, roosting, and daily activities. In order to ensure that the model can continuously and uninterruptedly monitor the behavior of pigs for 24 hours, daily activities are defined as behaviors other than breastfeeding, eating, roosting, drinking, and sleeping. The visual and auditory features of the six behaviors are defined as shown in Table 1. Figure 3 Examples of Log-mel spectrograms of images and audio are given for each behavior.

[0047] Table 1 Key behavioral categories of sows and their visual and auditory characteristics

[0048]

[0049]

[0050] In the video preprocessing stage, a two-stage denoising algorithm is used to perform median filtering first. The median filtering algorithm selects the median of the pixel values ​​in its neighborhood as the pixel value after filtering for each pixel point in the video frame, thereby effectively removing pulse noise such as salt and pepper noise. Then, wavelet transform threshold denoising is performed. The principle is to process the high-frequency coefficients after wavelet decomposition according to the set threshold, retain the coefficients greater than the threshold, and suppress the coefficients less than the threshold, further reduce Gaussian noise and other noise interference, and significantly improve the quality of video data.

[0051] The video is cropped based on the preset area of ​​interest according to the pig body size distribution probability model. The specific operation is to collect a large amount of pig body size data in advance, use statistical analysis methods to build a body size distribution probability model, and use this model to predict the area where pigs may appear in different scenarios, and then set the area of ​​interest. The cropped area can be flexibly adjusted within 30%-80% of the original picture, and the adjustment time interval does not exceed 1 hour, with an accuracy of 5% of the picture, which effectively reduces the redundancy of background information and improves the efficiency of subsequent processing and the accuracy of feature extraction.

[0052] In the audio preprocessing stage, an adaptive bandpass filter is used for noise reduction. Its filter parameters can be automatically adjusted according to the characteristics of the environmental noise, effectively filtering out interference such as environmental background noise. The audio signal is converted into a Log-mel spectrogram with each frame length of 25ms. The conversion process involves short-time Fourier transform of the audio signal, and then weighting and logarithm of the Mel filter group are performed to obtain a spectrogram representation suitable for audio feature extraction.

[0053] Embodiment 3: The present invention introduces and improves the AVSlowFast model, and realizes the recognition and classification of key behaviors of pigs by integrating the data of the two modalities of audio and video. The main model structure of the algorithm of the present invention is shown in the attached figure of the specification. Figure 4 As shown, the deep neural network AVSlowFast adopts a three-branch structure to extract the spatial, temporal and auditory features of audio-video respectively, and finally splices the three features to form a fused feature map with a high number of channels; considering the limited computing resources in practical applications, in order to make the model more focused on key information without increasing too much computing burden, the present invention introduces the GCT attention mechanism at the three-channel fused feature map of the AVSlowFast network, thereby enhancing the effect of the three-branch feature fusion and improving the overall performance of the model.

[0054] Among them, H and W are the height and width of the feature map, T and C are the time dimension and the number of feature channels, X and Y are the input and output of the GCT attention mechanism, and α F and α A Respectively represent the sampling rate multiple constants of the video fast channel and the audio channel, β F and βA Represents the channel quantity multiple constants of the video fast channel and audio channel respectively.

[0055] AVSlowFast is a three-branch neural network with an audio channel added to the SlowFast neural network model. The network can fuse visual and auditory features at multiple stages. SlowFast is a two-stream deep neural network that completes behavior recognition based on video information. Referring to the structure of the two-stream convolutional network, the two network branches use 3D-ResNet50 as the backbone network to extract spatial distribution features and dynamic timing features from the input image sequence respectively. The slow channel inputs a low-frame-rate video frame sequence to collect static spatial distribution features in the video, and the fast channel inputs a high-frame-rate video frame sequence to collect dynamic timing features in the video.

[0056] In the pig key behavior recognition task of the present invention, both static space and dynamic time series features can provide effective identification information. For example, the posture of a sleeping pig has unique static features, and the recognition process needs to pay more attention to the details of the picture; while the movement features such as the rapid shaking of the pig's breast during lactation and the violent breast arching behavior of the piglet need to pay more attention to the dynamic changes in the recognition process; in the implementation process, the two channels set different sampling rates for the input frame sequence to distinguish the extracted feature types, and the relationship between the two is as shown below:

[0057] F f =αF s

[0058] Where F f is the fast channel sampling rate, frame / s; F s is the sampling rate of the slow channel, frames / s; α is F f With F s A multiple constant of .

[0059] The audio channel of AVSlowFast is also based on the ResNet network structure; its input is the Log-mel spectrogram of the audio clip, that is, the time-frequency two-dimensional graph obtained through Fourier transform, Mel filter bank weighting, logarithmic transform and other operations; the time structure of this channel is finer than the two channels of SlowFast, and the time step is τ / F a , where the audio channel sampling rate is F a Much larger than the sampling rate F of the video fast channel f; Compared with video signals, the dimension and data volume of audio signals are relatively low, so the computational cost of audio channels is low, and the impact on the computational efficiency of the entire network is small; The overall network structure and parameter settings of AVSlowFast are shown in Table 2; When fusing video and audio features, AVSlowFast establishes horizontal connections between the three channels; Since audio features only correspond to semantically rich mid- and high-level visual features such as regions and contours in images, and low-level visual features such as edges and corners of images do not match specific auditory features, this model only establishes horizontal connections in the middle of the three channels to fuse audio and video information; Among them, the fusion of fast and slow video channels is the direct splicing of feature maps, while the fusion of audio to video channels splices and superimposes the features of the two; This fusion method does not strictly require the time alignment of auditory and visual features. When the audio features and visual features of the behavior cannot be completely synchronized, such as in some cases, the sound of pigs calling piglets appears before the breastfeeding behavior, this method can improve the robustness of the model and effectively reduce the impact of data acquisition equipment deviation; Finally, the three-branch feature map is spliced ​​into a fused feature map with a high number of channels on the channel axis;

[0060] Since the channel fusion strategy of AVSlowfast gives equal weight to audio and video signals, for some behaviors with unclear audio features, the audio and video fusion method may reduce the accuracy of behavior recognition compared to the model using single video information. Therefore, it is necessary to introduce an attention mechanism to adjust the importance of audio and video channels. The present invention proposes to introduce the GCT channel attention mechanism on the basis of AVSlowFast to improve the model and optimize the fusion method of multimodal features. GCT is different from the classic SENet. GCT uses Gaussian function to directly represent the importance of channels, does not require a learning process, can achieve zero parameterization, and its attention activation part shows better stability.

[0061] The main operations of GCT can be summarized into three stages, namely global context aggregation, normalization, and Gaussian context activation; GCA obtains channel statistics by aggregating the global context information of the feature map in space, thereby helping the network to determine long-range dependencies. Considering that the feature map processed is a 4-dimensional feature after multimodal fusion, the present invention modifies the calculation method of GCA in the original text, replacing the average pooling of the global context with an adaptive 3D average pooling to adapt to the feature map after 3D convolution processing. The calculation of the global context information is as follows:

[0062]

[0063] Where X∈R CF×TF×HF×WF is the input feature map; WF, HF, CF are the width, height, and number of channels of the feature map respectively; T Fis the number of frames; avg(.) means averaging the feature map; Z is the global context information.

[0064] Normalization is to adjust the data to a suitable range to facilitate model training and optimization. The specific calculation is as follows:

[0065]

[0066] In the formula is the normalized result of the global context information Z; μ and σ are the mean and standard deviation of the global context respectively; is a constant.

[0067] The final GCE realizes context feature activation through a Gaussian function that satisfies the preset relationship. The specific calculation is as follows:

[0068]

[0069] Where Y is the output of GCT; G(.) is the Gaussian function; c is the standard deviation of the Gaussian function.

[0070] The present invention adds the GCT attention mechanism on the basis of AVSlowFast to improve the original network model; compared with the feature map at a single branch of the network, the feature map at the fusion of three branches has the largest number of channels, so this paper embeds the GCT attention mechanism into the fusion of three branches of the network, and uses the three-channel fusion feature map output by AVSlowFast as the input of GCT; the improved AVSlowFast model uses GCT to adjust the weights of all feature channels, thereby improving the overall perception ability of the model, and this improvement adds almost no additional parameters; therefore, the model is suitable for environments with limited computing resources such as pig farms, and can be transplanted to edge computing devices for rapid deployment and application.

[0071] In summary, the intelligent pig behavior recognition system and method based on multimodal data fusion overcomes the limitations of single-modality recognition methods by combining data from both video and audio modalities. In particular, in complex situations such as poor lighting conditions and pigs occluding each other, audio information can supplement the lack of visual information and improve recognition accuracy. At the same time, by using the improved AVSlowFast model and GCT channel attention mechanism, the system can effectively fuse visual and auditory features, further enhancing the recognition capability in complex environments. This enables the system to accurately identify and classify different behaviors of pigs under various breeding conditions, greatly improving the reliability of recognition.

[0072] Moreover, the intelligent pig behavior recognition system and method based on multimodal data fusion, through the coordinated use of high-definition cameras and high-sensitivity microphones, can not only cover a larger monitoring range, but also capture more types of pig behaviors. For example, subtle behavioral characteristics such as the screams of piglets competing for milk and the humming of sows letting milk can be identified through audio data, which may not be easy to detect in single visual monitoring. In addition, the system can monitor the behavior of pigs 24 hours a day, defining daily activities as behaviors other than the five categories of breastfeeding, eating, arching the fence, drinking water, and sleeping, thereby realizing comprehensive monitoring of pig behavior, effectively making up for the shortcomings of traditional methods, and solving the problems of low recognition accuracy and limited monitoring range of existing technologies in complex breeding environments.

[0073] The relevant modules involved in this system are all hardware system modules or functional modules that combine computer software programs or protocols with hardware in the prior art. The computer software programs or protocols involved in the functional modules are themselves technologies that are well known to those skilled in the art and are not improvements of this system. The improvements of this system are the interaction or connection relationships between the modules, that is, improvements to the overall structure of the system to solve the corresponding technical problems to be solved by this system.

[0074] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A pig behavior intelligent recognition system based on multimodal data fusion, characterized in that: include: A data acquisition module, a data preprocessing module, a feature extraction module, a multimodal fusion module, a behavior recognition module, a data storage and management module, and a user interaction module. The data acquisition module includes a high-definition camera and a high-sensitivity microphone. The data preprocessing module includes a video preprocessing submodule and an audio preprocessing submodule. The feature extraction module includes a visual feature extraction submodule and an auditory feature extraction submodule.

2. The pig behavior intelligent recognition system based on multimodal data fusion according to claim 1 is characterized in that: The high-definition camera has a frame rate of 30 frames per second and a resolution of 1920×1080 pixels. It is installed in a pig house with a horizontal range of 0-360 degrees and a vertical range of -45 degrees to 45 degrees, and is arranged at 120-150 degrees with the high-sensitivity microphone; the high-sensitivity microphone has a sampling rate of 44.1kHz and is installed no more than 2 meters away from the pig activity area.

3. The pig behavior intelligent recognition system based on multimodal data fusion according to claim 1 is characterized in that: The video preprocessing submodule uses a two-stage denoising algorithm, first median filtering and then wavelet transform threshold denoising, and crops the video based on the preset area of ​​interest according to the pig body distribution probability model. The cropped area can be adjusted to 30%-80% of the original picture, and the adjustment time interval does not exceed 1 hour, with an accuracy of 5% of the picture; the audio preprocessing submodule uses an adaptive bandpass filter to reduce noise, and converts the audio signal into a Log-mel spectrum diagram with each frame length of 25ms.

4. The pig behavior intelligent recognition system based on multimodal data fusion according to claim 1 is characterized in that: The behavior recognition module and visual feature extraction submodule use the improved AVSlowFast model video fast and slow channels, the slow channel collects static features at 4 frames / second, and the fast channel collects dynamic features at 32 frames / second, and introduces a spatial pyramid pooling structure. The auditory feature extraction submodule uses the AVSlowFast model audio channel, and adopts a multi-scale convolution kernel combination of [3×1, 1×3], [5×1, 1×5] and [7×1, 1×7] and a specific time step to extract audio features.

5. The pig behavior intelligent recognition system based on multimodal data fusion according to claim 1 is characterized in that: The multimodal fusion module adopts an adaptive weighted fusion strategy combined with the GCT channel attention mechanism. The GCT calculation contains GCA calculation of global context information. The formula is: It also includes normalization processing and GCE calculation of channel weights, fusion time and space calibration, time synchronization error does not exceed 0.05 seconds, and spatial alignment accuracy is pixel level.

6. The pig behavior intelligent recognition system based on multimodal data fusion according to claim 1 is characterized in that: The data storage and management module uses a distributed relational database, data sharding storage, primary key and foreign key association, incremental backup once an hour, full backup once every 7 days, off-site redundant storage, and an optimization cycle of no more than 7 days.

7. The pig behavior intelligent recognition system based on multimodal data fusion according to claim 1 is characterized in that: The feature extraction module uses the PCA algorithm to reduce the dimension after extracting the features, and the dimension retention rate is not less than 80%. The system has self-learning and adaptive capabilities, optimizes the model every 15 days, and detects and repairs hardware faults within 5 minutes during operation.

8. A method for intelligent identification of pig behavior based on multimodal data fusion, characterized in that: The system for intelligently identifying pig behaviors based on multimodal data fusion according to claims 1 to 7 further comprises the following steps: S1. Data collection: Use high-definition cameras and high-sensitivity microphones to collect video and audio data from pig houses; S2, data preprocessing: video data is subjected to two-stage denoising and cropping of the region of interest preset according to the body model; audio data is subjected to adaptive bandpass filtering and denoising and then converted into a Log-mel spectrum graph; S3, Feature Extraction: Extract features from the preprocessed video and audio data respectively. For video, the improved AVSlowFast model is used, combined with the spatial pyramid pooling structure to obtain static and dynamic features; for audio, the audio channel of the AVSlowFast model is used, and the multi-scale convolution kernel combination and specific time step are used to extract audio features; S4, multimodal fusion: the visual and auditory features are fused through an adaptive weighted fusion strategy combined with the GCT channel attention mechanism. This process includes calculating global context information, normalization processing, and calculating channel weights to ensure the accuracy of spatiotemporal calibration; S5. Behavior recognition: Use a deep neural network classification model with a multi-layer residual structure to recognize pig behavior; S6. Data storage and management: Use a distributed relational database, shard storage, primary key and foreign key association, incremental backup once an hour, full backup once every 7 days, off-site redundant storage, optimization cycle no more than 7 days, system self-learning and adaptive optimization every 15 days, and hardware failure detection and repair within 5 minutes during operation; S7. User interaction: The identified behavior results are displayed to the user in a visual manner and corresponding management suggestions are provided. At the same time, the system can automatically record abnormal behaviors and trigger alarms to notify management personnel to respond in a timely manner.

Images