Animal husbandry multi-modal data analysis and decision-making method based on deep transfer learning

Abstract

According to the animal husbandry multi-modal data analysis and decision-making method based on deep transfer learning provided by the invention, the multi-modal data of the breeding environment is comprehensively collected through the data acquisition sensor, and a rich and comprehensive data basis is provided. Through the pre-trained transfer learning model, the existing knowledge and experience can be effectively utilized, the method can quickly adapt to the characteristics of livestock breeding data, the key characteristics of each modal data are effectively extracted, and the efficiency and accuracy of characteristic extraction are improved. The multi-modal data features are fused according to the preset data fusion strategy, mutually complementary information in the multi-modal data is integrated, more complete fusion features with better characterization force are formed, the data can more comprehensively reflect the actual situation of the breeding environment, and the analysis result is more reliable. The fusion features are analyzed based on the pre-trained decision model to determine the target processing decision, a more accurate decision is made based on the fusion features, and the analysis and decision accuracy of animal husbandry multi-modal data is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent monitoring and management technology, and in particular to a multimodal data analysis and decision-making method for animal husbandry based on deep transfer learning. Background Art

[0002] As the livestock industry modernizes, the analysis and decision-making of on-farm environmental and manure treatment data have become increasingly critical. With the rise of technologies like the Internet of Things, big data, and artificial intelligence, multimodal data collection systems are becoming widely used on farms. By deploying various sensors, they can capture multi-dimensional information such as environmental parameters, manure characteristics, and animal behavior in real time.

[0003] Existing livestock industry decision-making technologies mostly rely on rule-based thresholding and simple statistical analysis. Rule-based thresholding, based on domain knowledge and experience, sets fixed thresholds for key indicators. Exceeding these thresholds triggers alerts or decisions. This approach is simple to operate, understand, and implement, and can, to a certain extent, standardize the decision-making process and improve management efficiency.

[0004] However, simple statistical analysis can only process small-scale, low-dimensional data. It is prone to errors when faced with multimodal, massive aquaculture data, and is unable to deeply explore the potential complex patterns and relationships in the data, making the decision-making basis insufficient and difficult to meet the efficient and precise needs of modern aquaculture.

[0005] Therefore, how to improve the analysis and decision-making accuracy of multimodal data in animal husbandry has become a technical problem that needs to be solved urgently. Summary of the Invention

[0006] This application provides a method, device, equipment and storage medium for livestock industry multimodal data analysis and decision-making based on deep transfer learning, aiming to improve the accuracy of analysis and decision-making of livestock industry multimodal data.

[0007] In a first aspect, the present application provides a multimodal data analysis and decision-making method for animal husbandry based on deep transfer learning, the method comprising:

[0008] Based on the data acquisition sensor, multimodal data in the breeding environment is collected, wherein the multimodal data includes time series data, audio data and image data;

[0009] Based on the pre-trained transfer learning model, the multimodal data is analyzed to obtain data features corresponding to each modal data;

[0010] Based on a preset data fusion strategy, the data features corresponding to each modal data are fused to obtain fused features;

[0011] Based on the pre-trained decision model, feature analysis is performed on the fusion features to determine the target processing decision for the breeding environment.

[0012] In a second aspect, the present application further provides a multimodal data analysis and decision-making device for animal husbandry based on deep transfer learning, the multimodal data analysis and decision-making device for animal husbandry based on deep transfer learning comprising:

[0013] A data acquisition module, configured to collect multimodal data in the aquaculture environment based on a data acquisition sensor, wherein the multimodal data includes time series data, audio data, and image data;

[0014] A data analysis module is used to analyze the multimodal data based on a pre-trained transfer learning model to obtain data features corresponding to each modal data;

[0015] A feature fusion module is used to fuse the data features corresponding to each modal data based on a preset data fusion strategy to obtain a fusion feature;

[0016] The decision determination module is used to perform feature analysis on the fusion features based on a pre-trained decision model to determine the target processing decision for the breeding environment.

[0017] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the livestock multimodal data analysis and decision-making method based on deep transfer learning as described above are implemented.

[0018] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the livestock multimodal data analysis and decision-making method based on deep transfer learning as described above are implemented.

[0019] This application provides a multimodal data analysis and decision-making method for livestock farming based on deep transfer learning. This method uses data acquisition sensors to comprehensively collect multimodal data (including time series data, audio data, and image data) from the farming environment, providing a rich and comprehensive data foundation. A pre-trained transfer learning model can effectively leverage existing knowledge and experience, quickly adapt to the characteristics of livestock farming data, and effectively extract key features from each modal data. This reduces over-reliance on domain data, avoids the large-scale data and time costs required to train the model from scratch, and improves the efficiency and accuracy of feature extraction. Multimodal data features are fused according to a preset data fusion strategy, integrating complementary information within the multimodal data to form more complete and representative fused features. This allows the data to more comprehensively reflect the actual conditions of the farming environment, overcomes the limitations of single-modal data, and makes the analysis results more reliable. The fused features are analyzed based on a pre-trained decision model to determine target processing decisions. More accurate decisions can be made based on the fused, high-quality features, thereby effectively improving the overall accuracy of analysis and decision-making for multimodal data in the livestock industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A flowchart of a first embodiment of a multimodal data analysis and decision-making method for animal husbandry based on deep transfer learning provided in an embodiment of the present application;

[0022] Figure 2 A task-oriented decision-making process diagram provided for this application;

[0023] Figure 3 This is a structural diagram of the first embodiment of a multimodal data analysis and decision-making device for animal husbandry based on deep transfer learning provided by the present application;

[0024] Figure 4 This is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0025] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0028] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 A flowchart of the first embodiment of a multimodal data analysis and decision-making method for animal husbandry based on deep transfer learning provided in an embodiment of the present application.

[0030] like Figure 1 As shown, the livestock multimodal data analysis and decision-making method based on deep transfer learning includes steps S101 to S104.

[0031] S101. Collect multimodal data in a farming environment based on a data acquisition sensor, wherein the multimodal data includes time series data, audio data, and image data;

[0032] In one embodiment, the data acquisition sensor may include an image sensor, an audio sensor, a time series data acquisition sensor, and the like.

[0033] For example, image sensors may include cameras, etc., and audio sensors may include microphones, radio equipment, etc.

[0034] The time series data acquisition sensor can be selected according to actual needs.

[0035] For example, to collect characteristic time series data of manure and sewage, physical sensors such as liquid level sensors (manure pits / liquid storage tanks), flow meters (transmission pipelines), pressure sensors (pipelines / tanks), temperature sensors (inside manure and sewage / compost piles / fermentation tanks), and humidity sensors (composts) can be used, or chemical sensors such as ammonia (NH3) sensors, hydrogen sulfide (H2S) sensors, methane (CH4) sensors (which can be used for anaerobic fermentation monitoring or leakage warning), and pH sensors can be used.

[0036] For the environmental time series data of the breeding environment, temperature and humidity sensors, NH3 concentration sensors, H2S concentration sensors, CO2 concentration sensors, particulate matter (PM2.5 / PM10) sensors, light intensity sensors, etc. can be used to collect indoor environmental time series data. For outdoor environmental time series data, sensor equipment such as temperature and humidity sensors, wind speed and direction sensors, light sensors, and rainfall sensors can be used to collect external temperature and humidity, wind speed and direction, light, rainfall, and other data.

[0037] Furthermore, based on the image sensor, images of feces and animal behavior in the breeding environment are collected to obtain the image data; based on the audio sensor, the audio data in the breeding environment is collected; based on the time series data collection sensor, the environmental parameter time series data and feces characteristic time series data in the breeding environment are collected to obtain the time series data.

[0038] In one embodiment, the image data may include feces images and animal behavior images. The feces images can be used to identify feces distribution, accumulation status, and liquid level information. The animal behavior images can be used to analyze the daily behavior of animals in the breeding environment and assist in identifying abnormal areas within the breeding environment. For example, when the ammonia (NH3) concentration in a certain area reaches a certain level, animals will subconsciously avoid the area, thereby helping to determine the degree of feces accumulation or poor ventilation in the area.

[0039] Specifically, cameras and other image sensors are used to monitor and capture images of manure storage areas (such as pits and liquid storage tanks) in real time. Analysis of these images reveals information such as the accumulation status and liquid level, enabling timely cleaning or treatment measures to prevent environmental problems caused by overflow or excessive accumulation.

[0040] Cameras are installed in breeding areas to capture animals' daily activities, such as eating, drinking, resting, and exercising. Analysis of these behavioral images can be used to assess the animals' health and welfare, providing a basis for breeding management. For example, observing animals for unusual activity or changes in their mental state can help identify illness or stress reactions.

[0041] In one embodiment, the audio data may include the sounds of animals in the breeding environment, and audio such as ambient noise in the breeding environment and its surroundings.

[0042] Specifically, microphones and other audio sensors are deployed throughout the breeding environment to collect various animal sounds, such as the grunts of pigs and the crowing of chickens. These sounds can reveal factors such as an animal's health, mood, and stress. By analyzing characteristics such as the frequency, intensity, and duration of these sounds, the health of the animals and the comfort of the breeding environment can be monitored.

[0043] Collect background noise on the farm, including sounds from equipment and personnel activities. Understanding the ambient noise level helps assess its impact on animals, as excessive noise may cause stress in animals, which in turn affects their growth and production performance.

[0044] In addition, environmental noise may also be an indicator of equipment failure or abnormal conditions. By analyzing audio data, potential problems can be discovered and resolved in a timely manner.

[0045] In one embodiment, the time series data can be divided into environmental parameter time series data and feces characteristic time series data.

[0046] Exemplarily, the environmental parameter time series data may include indoor environment time series data and outdoor environment time series data.

[0047] Indoor environmental time-series data can be collected using devices such as temperature and humidity sensors, NH3 concentration sensors, H2S concentration sensors, CO2 concentration sensors, particulate matter (PM2.5 / PM10) sensors, and light intensity sensors. This data reflects the quality and comfort of the indoor environment and provides a basis for automatic control of the breeding environment. For example, ventilation systems and heating or cooling equipment can be adjusted based on temperature and humidity data to maintain a suitable breeding environment. Lighting equipment can also be controlled based on light intensity data to simulate natural light cycles and promote animal growth and reproduction.

[0048] Time-series data on the outdoor environment can be collected using devices such as temperature and humidity sensors, wind speed and direction sensors, light sensors, and rainfall sensors. These factors can also impact the overall farm environment and animal health. For example, wind speed and direction can affect ventilation and air quality within the shed, while excessive rainfall can lead to water accumulation or overflow in manure treatment facilities.

[0049] Exemplarily, the manure characteristic time series data may include liquid level time series data, flow time series data, pressure time series data, temperature time series data, humidity time series data, and gas concentration time series data, etc.

[0050] Specifically, liquid level sensors can be installed in manure storage facilities (such as pits and liquid storage tanks) to measure the changes in the liquid level of manure over time in real time, so as to understand the accumulation rate and discharge of manure, and provide a basis for the formulation of manure treatment plans, such as determining the appropriate cleaning time interval to avoid manure overflow.

[0051] Chemical sensors such as ammonia (NH3) sensors, hydrogen sulfide (H2S) sensors, methane (CH4) sensors and pH sensors can be used to collect changes in gas concentration and pH value over time during the manure treatment process, which is convenient for monitoring chemical reactions in the manure treatment process and assessing environmental risks (such as the greenhouse effect that may be caused by methane leakage, pollution from ammonia and hydrogen sulfide, etc.).

[0052] In addition, flow meters can be installed on manure conveying pipelines to monitor flow changes of manure in the pipelines; pressure sensors can be installed in manure conveying pipelines, tanks and other parts to monitor pressure changes in the system in real time and help determine the operating status of pipelines and equipment; temperature sensors can be installed inside manure (such as compost piles, fermentation tanks, etc.) and in the surrounding environment to monitor temperature changes over time, and the treatment effect can be evaluated by analyzing temperature time series data; humidity sensors can be used to collect humidity data during composting and other treatment processes to understand the progress of composting and adjust composting conditions (such as turning frequency, adding water, etc.) in a timely manner to ensure the quality and effect of composting.

[0053] In one embodiment, after collecting multimodal data, the multimodal data may be preprocessed before feature recognition and extraction. The data preprocessing may include operations such as data cleaning and normalization to eliminate noise and differences in the data and improve data quality.

[0054] This embodiment uses a variety of data acquisition sensors to comprehensively collect multimodal data from the breeding environment, including time series data, audio data, and image data, which can fully reflect the actual situation of the breeding environment. Preprocessing various types of data can improve data quality and lay a good foundation for subsequent analysis. Using this data, it is possible to accurately monitor the status of manure, animal behavior, environmental parameters, etc., thereby promptly identifying potential problems, providing a scientific basis for breeding management, achieving refined management of the breeding environment, improving management efficiency, reducing labor costs, and improving animal welfare and production performance, with significant beneficial effects.

[0055] S102: Analyze the multimodal data based on the pre-trained transfer learning model to obtain data features corresponding to each modality of data;

[0056] In one embodiment, the transfer learning model can be obtained by using multiple sub-models of single-modal data analysis for transfer learning. The transfer learning model is divided into multiple sub-models, which perform single-modal data analysis on data of different modalities respectively.

[0057] In one embodiment, the transfer learning model may include an image analysis sub-model, an audio analysis sub-model, and a time series prediction sub-model.

[0058] For example, a pre-trained vision model (such as ResNet, EfficientNet, ViT) can be used to fine-tune it on a small number of labeled livestock-specific manure images. The goal is to learn feature representations that can distinguish manure states (such as thin / thick, good or bad solid-liquid separation, surface crust, foam amount) or estimate key components (such as DM, N content) as one of the functional components of the image analysis sub-model.

[0059] Pre-trained behavior recognition or posture estimation models (such as those trained on the Kinetics or Animal Kingdom datasets) can also be fine-tuned in specific environmental monitoring areas of the target farm to achieve high-precision, automated recognition of animal behaviors (such as gathering away from defecation areas, frequent head shaking, and open-mouth breathing) as another functional component of the image analysis sub-model.

[0060] For example, pre-trained time series prediction models (such as Transformers and Informers trained based on complex industrial processes or meteorological data) can be used to migrate to livestock environmental series data (environmental parameter time series, equipment operation time series) and a small amount of manure characteristic time series data to predict the peak NH3 concentration, manure pit temperature changes, expected fermentation degree, etc. in the short future.

[0061] Furthermore, based on the image analysis sub-model, data analysis is performed on the image data to obtain image data features; based on the audio analysis sub-model, data analysis is performed on the audio data to obtain audio data features; based on the time series prediction sub-model, data analysis is performed on the time series data to obtain time series data features.

[0062] Specifically, a convolutional neural network (CNN) or its variants (ResNet / VGG / EfficientNet / VisionTransformer) can be trained using general datasets such as ImageNet and COCO, or publicly available agricultural / animal image datasets. The top-level weights of the pre-trained model are fine-tuned on livestock-specific image datasets (such as animal posture, body condition scores, behavioral clips, and manure status images). The bottom layer of the pre-trained model is frozen, and only the newly added classification / regression layers are trained to output high-level feature vectors, completing transfer learning for the image analysis sub-model. Image data captured by the image sensor is then fed into the image analysis sub-model to extract image data features, such as manure distribution, manure status, and animal behavior.

[0063] You can use common audio datasets such as AudioSet and FSD50K to train audio pre-trained models such as 1D-CNN (for processing raw waveforms), 2D-CNN (for processing Mel-spectrograms), and RNN / LSTM / Transformer (for processing time series features). This converts raw audio into Mel-spectrograms or MFCCs (Mel-Frequency Cepstral Coefficients). Then, fine-tune the model on livestock audio datasets (such as animal calls, coughs, and equipment operating noise) to generate an audio analysis sub-model. The audio data collected by the audio acquisition sensor is input into the audio analysis sub-model to extract audio data features, such as animal call characteristics (such as frequency and pitch), cough characteristics, and equipment operating characteristics.

[0064] Pre-trained models such as LSTM, GRU, TCN (Time Convolutional Network), and Transformer can be used to train self-supervised models on public time series datasets (such as power load and meteorological data). The pre-trained model can be fine-tuned using time series data such as sensor data (temperature, humidity, and ammonia concentration), physiological indicators (heart rate and body temperature), and device status (fan speed). The model outputs predicted values ​​for future states (such as ammonia concentration in the next hour), creating a time series prediction sub-model that can be used for different time series data analyses. Based on different time series data analysis tasks, the corresponding time series data is input into the time series prediction sub-model, which then outputs time series data features or time series prediction data features.

[0065] This embodiment uses a pre-trained transfer learning model to efficiently analyze multimodal data and extract the features of each modal data. By using multiple sub-models for single-modal data analysis, data of different modalities can be processed in a targeted manner to improve the accuracy and efficiency of feature extraction. At the same time, the transfer learning method reduces the dependence on large amounts of labeled data, reducing training costs and time. In addition, by fine-tuning the pre-trained model to adapt it to specific scenarios in animal husbandry, key features such as manure status, animal behavior, and environmental parameters can be better captured, providing a high-quality data foundation for subsequent data fusion and decision-making, thereby improving the performance and reliability of the entire system.

[0066] S103: Based on a preset data fusion strategy, perform feature fusion on the data features corresponding to each modality data to obtain fusion features;

[0067] In one embodiment, the data fusion strategy can be selected independently according to the data type and feature fusion requirements. For example, a combination of hybrid fusion and attention mechanism can be used as the data fusion strategy.

[0068] Specifically, high-level features extracted from manure images / spectra (visual), environmental sensor data (numerical time series), animal behavior data (visual / behavioral time series), and measured / predicted manure chemical parameter (such as NH3) vectors can be spliced ​​or passed through a fusion network (such as MLP) after splicing.

[0069] By adding a cross-modal attention module before or during the fusion layer, the model automatically learns which features from which modalities are more important for specific decision-making objectives (such as predicting NH3 concentration, developing ventilation strategies, and adjusting agitation frequency). For example, when predicting NH3 outbreaks, the model might use the attention module to focus more on indoor temperature and humidity, ventilation history, animal density distribution, and visual characteristics of the manure pit surface (such as whether there are signs of strong volatilization).

[0070] For example, Figure 2 As shown in the figure, the fusion features are used to make preliminary decisions for different subtasks (such as ventilation control, spray deodorization, and stirring optimization), and then these decisions are combined to form the final operation instructions.

[0071] Furthermore, based on the time series correspondence, the data features corresponding to each modal data are spliced ​​to obtain the spliced ​​features; based on the cross-modal attention mechanism, the attention weights of the data features corresponding to each modal data under the current decision task are determined; based on the attention weights of the data features corresponding to each modal data under the current decision task, the spliced ​​features are weightedly fused to obtain the fused features.

[0072] In one embodiment, data features of different modalities are spliced ​​according to a temporal correspondence, and a cross-modal attention mechanism is used to determine the importance weight of each modal feature in the current decision-making task, thereby achieving weighted fusion of the spliced ​​features, and finally obtaining fusion features that can comprehensively characterize the breeding environment and manure conditions.

[0073] Specifically, based on temporal correspondence, the extracted image, audio, and time series features are aligned on the timeline to ensure that the features at each time point originate from the same breeding environment. These aligned feature vectors are then simply concatenated to form a concatenated feature vector containing information from all modalities.

[0074] A cross-modal attention mechanism model is constructed. This model takes the concatenated feature vector as input and automatically determines the weight of each modal feature for the current decision task by calculating the correlation and importance between the features of each modality. Specifically, the cross-modal attention mechanism dynamically adjusts the weight distribution by learning the contribution of each modal feature to the decision goal.

[0075] The concatenated feature vectors are weighted and fused according to the weights determined by the cross-modal attention mechanism to produce a fused feature vector. This fused feature vector can highlight the modal information that contributes more to the decision-making goal while suppressing relatively less important information, thereby more accurately reflecting the comprehensive state of the aquaculture environment.

[0076] This embodiment uses a data fusion strategy to fuse features from various modal data to create a fused feature that comprehensively characterizes the aquaculture environment. It combines features based on temporal correspondence to ensure accurate feature fusion. A cross-modal attention mechanism automatically weights each modal feature based on the current decision task, highlighting important information and suppressing less important information, making the fused feature more targeted and decision-making valuable. This fusion approach leverages the strengths of multimodal data to provide more accurate and comprehensive information about the aquaculture environment, thereby providing a more reliable basis for subsequent decision-making.

[0077] S104: Based on the pre-trained decision model, perform feature analysis on the fusion features to determine a target processing decision for the breeding environment.

[0078] In one embodiment, the fused features are fed into a pre-trained decision model to generate decisions for farm environmental control and target processing. The decision model, trained on a large amount of fused feature data, can generate decision recommendations based on the fused feature vectors for the current state of the farming environment. For example, it can make decisions such as adjusting ventilation and cleaning manure based on environmental parameters, animal behavior, and manure status, thereby facilitating intelligent and automated farming management.

[0079] Furthermore, based on the current decision task, a threshold range corresponding to at least one preset processing decision is determined; based on the comparison result between the fusion feature and the threshold range, a target processing decision corresponding to the fusion feature is determined.

[0080] For example, assuming that the current decision-making task is to decide on the timing of manure treatment, more weight can be assigned to the manure monitoring data in the multimodal data (such as manure liquid level, manure distribution and accumulation degree, ammonia concentration, pit temperature, etc.), and the obtained fusion features can be digitized and compared with the preset threshold range.

[0081] For example, if the manure level exceeds 80% of the preset capacity and the ammonia concentration remains above 30 ppm for two hours, the decision model determines that the manure storage capacity is approaching the upper limit and the environmental quality has significantly deteriorated. At this time, the output target processing decision is: immediately start the manure cleaning process and mobilize the cleaning equipment to the designated manure pit to carry out cleaning operations to prevent manure overflow and further ammonia pollution.

[0082] If the manure level is between 60% and 80% of capacity, the ammonia concentration fluctuates between 15 and 30 ppm, and the pit temperature remains above 35°C for three consecutive hours, this indicates that manure treatment efficiency may be reduced, posing a potential risk to environmental quality. The corresponding treatment strategy is to appropriately reduce the source of manure generation (such as adjusting feed intake), increase manure mixing frequency to promote fermentation, and strengthen ventilation systems to reduce temperature and ammonia concentration to prevent further deterioration of environmental indicators.

[0083] For example, assuming that the current decision-making task is to decide on the method of manure treatment, more weight can be assigned to the relevant data in the multimodal data (such as manure dry matter content, pH value, animal behavior characteristics, etc.), and the obtained fusion features can be digitized and compared with the preset threshold range.

[0084] For example, when the dry matter content of manure exceeds 30%, the pH is between 7 and 8, and the proportion of animals congregating away from the excretion area exceeds 30%, the decision model determines that the manure is suitable for composting and that adverse effects on animals are becoming apparent. In this case, the output treatment strategy is to transfer the manure to the composting area for aerobic composting, while also adding fresh bedding to the original excretion area to improve the animal living environment and guide the animals back to their normal activity areas.

[0085] If the dry matter content of manure is less than 20%, the pH is less than 6.5, and the proportion of animals congregating away from the excretion area is between 10% and 30%, it indicates that the manure is more suitable for anaerobic digestion, but the impact on animals is relatively mild. The treatment strategy in this case is to feed the manure into an anaerobic digester for anaerobic digestion, and the generated biogas can be collected for energy utilization. At the same time, appropriate adjustments to the feed formula or stocking density can be made to gradually improve manure quality and animal behavior.

[0086] In another embodiment, the user can also select the feature data that needs to be paid attention to, or the decision direction that needs to be implemented. The output decision can be a single target processing decision or an environmental control decision, or a combination of multiple decisions.

[0087] For example, the target treatment decision can be a combination of environmental parameter control and manure treatment method decisions. The model can focus on environmental parameters such as temperature, humidity, and ammonia concentration, as well as the impact of manure monitoring data on these environmental parameters. However, the aquaculture environment is large, and sensors cannot monitor environmental parameters in all areas of the entire aquaculture environment. Therefore, it is also possible to indirectly monitor environmental parameters within the aquaculture environment based on animal behavior. For example, if an animal subconsciously avoids a certain area, it may be due to manure accumulation or insufficient ventilation, which has led to increased ammonia concentration.

[0088] For example, when the temperature is >30°C, humidity >70%, ammonia concentration >25ppm, the dry matter content of manure is >30%, the pH is between 7-8, and more than 30% of animals are congregating away from the excretion area, this indicates a poor breeding environment and that the manure is suitable for composting. In this case, composting should be initiated, along with refrigeration and dehumidification equipment to reduce ambient temperature and humidity and improve the animal living environment.

[0089] When the temperature is between 25-30°C, the humidity is between 60%-70%, the ammonia concentration is between 15-25 ppm, the dry matter content of the manure is between 20%-30%, the pH is between 6.5-7, and the proportion of animals congregating away from the excretion area is between 10%-30%, it indicates that the breeding environment needs improvement and the manure is suitable for anaerobic digestion. In this case, the manure should be transferred to an anaerobic digester, and the feed formula or stocking density should be adjusted to gradually improve manure quality and animal behavior.

[0090] This embodiment analyzes fused features based on a pre-trained decision model, generating accurate decision recommendations based on the current state of the aquaculture environment and facilitating intelligent and automated aquaculture management. By comparing fused features with preset threshold ranges, the model can output corresponding processing strategies for different decision-making tasks (such as the timing and method of manure treatment), effectively addressing various issues in the aquaculture environment. Furthermore, this method can comprehensively consider multimodal data to achieve comprehensive monitoring and assessment of the aquaculture environment, improving the accuracy and reliability of decisions, thereby optimizing the aquaculture management process and enhancing aquaculture efficiency.

[0091] In another embodiment, the current decision task may be a task specified by the user, and the model may also automatically monitor and make decisions on the current data features based on analyzing historical decisions and data feature thresholds corresponding to the historical decisions.

[0092] In one embodiment, historical decision data is obtained; and based on the historical decision data, feature data and a data feature threshold corresponding to at least one historical decision are determined.

[0093] Specifically, a dedicated database can be established to store multimodal monitoring data and corresponding decision records. In the database, multimodal data at the same decision time point are associated with the decision results and stored as historical decision data.

[0094] In one embodiment, historical decision data can be annotated to record the decision type, underlying feature data, and decision results. Historical decision records can be analyzed to identify the key feature data underlying each decision type. For example, target treatment decisions focus on manure level height and ammonia concentration, while animal health intervention decisions focus on animal body temperature and food intake.

[0095] Filter out cases with significant effects in historical decision-making, analyze the corresponding feature data value range, and determine it as the data feature threshold range of the decision; at the same time, for decision-making cases with poor results, analyze the abnormalities of their feature data and clarify the threshold range that needs to be avoided.

[0096] For example, suppose a farm database stores multimodal monitoring data and decision records from the past year. Analysis reveals that target treatment decisions are primarily based on three key data characteristics: manure level, ammonia concentration, and pit temperature. By screening for successful target treatment decisions, it was found that in these cases, manure level was typically between 75% and 85%, ammonia concentration between 25 and 30 ppm, and pit temperature between 35 and 40°C. These ranges were tentatively designated as decision thresholds.

[0097] Furthermore, feature data and a data feature threshold corresponding to at least one historical decision are obtained; based on the decision model, feature analysis is performed on the fusion feature that matches the feature data to obtain a feature analysis result; when the feature comparison result is that the fusion feature is greater than or equal to the data feature threshold, the historical decision corresponding to the data feature threshold is determined to be the target processing decision.

[0098] In one embodiment, historical data of target processing decisions can be obtained by querying the database, including decision time, characteristic data based on which the decision was made (such as manure liquid level height, ammonia concentration, pit temperature, etc.), and decision effect evaluation.

[0099] In one embodiment, suppose that analysis of historical decision data reveals that target treatment decisions are primarily based on the following characteristic data: manure level, ammonia concentration, and pit temperature. Based on threshold analysis of decision effectiveness, cases with significant historical results are screened and the corresponding characteristic data value ranges are analyzed. For example, for cases with good target treatment decision effectiveness, it was found that manure level was typically between 75% and 85%, ammonia concentration was between 25 and 30 ppm, and pit temperature was between 35 and 40°C.

[0100] Using a pre-trained decision model, feature analysis is performed on the fused features that match the target processing decision feature data. During the feature comparison process, the manure liquid level, ammonia concentration, and pit temperature in the fused features are compared with the determined data feature thresholds. If these indicators in the fused features are greater than or equal to the corresponding data feature thresholds, that is, the manure liquid level is ≥70% of capacity, the ammonia concentration is ≥20ppm, and the pit temperature is ≥30°C, the decision model determines that the current breeding environment is in a state where manure treatment is required. At this point, the system automatically determines that the decision corresponding to the feature data threshold in the historical decision data (such as immediately starting the manure cleaning process and increasing ventilation) is the current target treatment decision and recommends it to the farm staff.

[0101] Specifically, the model can focus on data such as temperature, humidity, and light intensity in the breeding house that directly affect animal growth and production performance. It can also focus on manure monitoring data such as dry matter content, nitrogen content, and phosphorus content in manure. It can also focus on animal health-related data such as respiratory rate, body temperature, and food intake. Based on historical decisions and the data feature thresholds corresponding to historical decisions, the decision model compares the data features of interest with the historical data features. When the data features of the current focus meet the data feature threshold requirements corresponding to the historical decision, the corresponding historical decision can be used as the current target processing decision output.

[0102] For example, when the model detects that the temperature in the breeding environment is higher than 32°C or lower than 15°C, the humidity is greater than 80% or less than 40%, and the light intensity is higher than 1200 lux or lower than 100 lux, it can output decisions such as adjusting the ventilation system, turning on cooling or heating equipment, and adjusting lighting equipment to control environmental parameters within a range suitable for animal growth.

[0103] Alternatively, when it is monitored that the dry matter content in the manure is greater than 35%, the nitrogen content is greater than 2%, and the phosphorus content is greater than 0.5%, decisions such as composting or anaerobic fermentation of the manure can be output to achieve reasonable resource utilization of manure and reduce the pollution risk of manure to the environment.

[0104] Alternatively, when it is monitored that the manure liquid level exceeds 85% of the capacity, the ammonia concentration is greater than 28ppm, and the pit temperature is higher than 37°C, a decision can be output to immediately start the manure cleaning program and increase the operating frequency of the ventilation system to reduce the ammonia concentration and pit temperature.

[0105] Alternatively, when an animal's respiratory rate is monitored to be over 40 times per minute, its body temperature is above 40°C, and its food intake is reduced by more than 20%, decisions such as adjusting feed formula, increasing nutritional supplements, and optimizing the breeding environment can be output to improve the animal's health.

[0106] In this example, by analyzing historical decision data to determine characteristic data and thresholds, the model can automatically monitor current data characteristics and make decisions accordingly. This not only improves the scientific nature and accuracy of decision-making, but also enables intelligent and automated farming management. Furthermore, this method comprehensively considers multiple data features to comprehensively assess the state of the farming environment, promptly identifying potential issues and providing corresponding treatment strategies, effectively reducing environmental risks, improving farming efficiency, and ensuring the healthy growth of animals.

[0107] This embodiment provides a multimodal data analysis and decision-making method for livestock farming based on deep transfer learning. This method uses data acquisition sensors to comprehensively collect multimodal data (including time series data, audio data, and image data) from the farming environment, providing a rich and comprehensive data foundation. A pre-trained transfer learning model effectively leverages existing knowledge and experience, quickly adapts to the characteristics of livestock farming data, and effectively extracts key features from each modal data. This reduces over-reliance on domain data, avoids the large-scale data and time costs required to train the model from scratch, and improves the efficiency and accuracy of feature extraction. Multimodal data features are fused according to a preset data fusion strategy, integrating complementary information from the multimodal data to form more complete and representative fused features. This allows the data to more comprehensively reflect the actual conditions of the farming environment, overcomes the limitations of single-modal data, and makes the analysis results more reliable. The fused features are analyzed based on a pre-trained decision model to determine target processing decisions. More accurate decisions can be made based on the fused, high-quality features, thereby effectively improving the overall accuracy of analysis and decision-making for multimodal data in the livestock industry.

[0108] See also Figure 3 , Figure 3 This is a structural schematic diagram of the first embodiment of a multimodal data analysis and decision-making device for animal husbandry based on deep transfer learning provided in this application. The multimodal data analysis and decision-making device for animal husbandry based on deep transfer learning is used to execute the aforementioned multimodal data analysis and decision-making method for animal husbandry based on deep transfer learning.

[0109] like Figure 3 As shown, the animal husbandry multimodal data analysis and decision-making device 200 based on deep transfer learning includes: a data acquisition module 201, a data analysis module 202, a feature fusion module 203 and a decision determination module 204.

[0110] The data acquisition module 201 is used to collect multimodal data in the farming environment based on a data acquisition sensor, wherein the multimodal data includes time series data, audio data, and image data;

[0111] A data analysis module 202 is configured to perform data analysis on the multimodal data based on a pre-trained transfer learning model to obtain data features corresponding to each modality of data;

[0112] The feature fusion module 203 is used to perform feature fusion on the data features corresponding to each modality data based on a preset data fusion strategy to obtain fused features;

[0113] The decision determination module 204 is used to perform feature analysis on the fusion features based on the pre-trained decision model to determine the target processing decision for the breeding environment.

[0114] In one embodiment, the data acquisition module 201 includes:

[0115] An image data acquisition unit, configured to acquire images of feces and animal behavior in the breeding environment based on an image sensor to obtain the image data;

[0116] An audio data acquisition unit, configured to collect the audio data in the breeding environment based on an audio sensor;

[0117] The time series data acquisition unit is used to collect the time series data of environmental parameters and the time series data of feces characteristics in the breeding environment based on the time series data acquisition sensor to obtain the time series data.

[0118] In one embodiment, the data analysis module 202 includes:

[0119] An image data analysis unit, configured to perform data analysis on the image data based on an image analysis sub-model to obtain image data features;

[0120] An audio data analysis unit, configured to perform data analysis on the audio data based on the audio analysis sub-model to obtain audio data features;

[0121] The time series data analysis unit is used to perform data analysis on the time series data based on the time series prediction sub-model to obtain time series data features.

[0122] In one embodiment, the feature fusion module 203 includes:

[0123] A feature splicing unit, configured to splice the data features corresponding to each modal data based on a time sequence correspondence to obtain a spliced ​​feature;

[0124] An attention weight determination unit, used to determine the attention weights of the data features corresponding to each modal data under the current decision task based on the cross-modal attention mechanism;

[0125] A feature fusion unit is used to perform weighted fusion on the splicing features based on the attention weights of the data features corresponding to each modal data under the current decision task to obtain the fused features.

[0126] In one embodiment, the decision determination module 204 includes:

[0127] A threshold range determining unit, configured to determine a threshold range corresponding to at least one preset processing decision based on a current decision task;

[0128] The first decision determination unit is configured to determine the target processing decision corresponding to the fusion feature based on a comparison result between the fusion feature and the threshold range.

[0129] In one embodiment, the decision determination module 204 further includes:

[0130] A historical feature acquisition unit, configured to acquire feature data and a data feature threshold corresponding to at least one historical decision;

[0131] a feature analysis unit, configured to perform feature analysis on the fusion feature matched with the feature data based on the decision model to obtain a feature analysis result;

[0132] The second decision determination unit is used to determine the historical decision corresponding to the data feature threshold as the target processing decision when the feature comparison result shows that the fusion feature is greater than or equal to the data feature threshold.

[0133] In one embodiment, the decision determination module 204 further includes:

[0134] A historical decision data acquisition unit, used to acquire historical decision data;

[0135] The historical feature determination unit is used to determine feature data and a data feature threshold corresponding to at least one historical decision based on the historical decision data.

[0136] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned embodiment of the livestock multimodal data analysis and decision-making method based on deep transfer learning, and will not be repeated here.

[0137] The apparatus provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 4 Runs on the computer device shown.

[0138] See also Figure 4 , Figure 41 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0139] See Figure 4 The computer device includes a processor, a memory, and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0140] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions that, when executed, enable a processor to perform any of the livestock multimodal data analysis and decision-making methods based on deep transfer learning.

[0141] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0142] The internal memory provides an environment for the operation of computer programs in non-volatile storage media. When the computer program is executed by the processor, the processor can execute any livestock multimodal data analysis and decision-making method based on deep transfer learning.

[0143] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0145] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0146] Based on the data acquisition sensor, multimodal data in the breeding environment is collected, wherein the multimodal data includes time series data, audio data and image data;

[0147] Based on the pre-trained transfer learning model, the multimodal data is analyzed to obtain data features corresponding to each modal data;

[0148] Based on a preset data fusion strategy, the data features corresponding to each modal data are fused to obtain fused features;

[0149] Based on the pre-trained decision model, feature analysis is performed on the fusion features to determine the target processing decision for the breeding environment.

[0150] In one embodiment, when the processor collects multimodal data in the breeding environment based on the data acquisition sensor, it is used to implement:

[0151] Based on the image sensor, images of feces and animal behavior in the breeding environment are collected to obtain the image data;

[0152] Based on the audio sensor, the audio data in the breeding environment is collected;

[0153] Based on the time series data acquisition sensor, the environmental parameter time series data and the feces characteristic time series data in the breeding environment are collected to obtain the time series data.

[0154] In one embodiment, when the processor implements the pre-trained transfer learning model, performs data analysis on the multimodal data, and obtains data features corresponding to the multimodal data, it is configured to implement:

[0155] Based on the image analysis sub-model, performing data analysis on the image data to obtain image data features;

[0156] Based on the audio analysis sub-model, performing data analysis on the audio data to obtain audio data features;

[0157] Based on the time series prediction sub-model, data analysis is performed on the time series data to obtain time series data features.

[0158] In one embodiment, when the processor implements the preset data fusion strategy to perform feature fusion on the data features corresponding to each modality data to obtain the fusion feature, it is configured to implement:

[0159] Based on the time series correspondence, the data features corresponding to each modal data are spliced ​​to obtain splicing features;

[0160] Based on the cross-modal attention mechanism, the attention weights of the data features corresponding to each modality data under the current decision task are determined;

[0161] Based on the attention weights of the data features corresponding to each modal data under the current decision task, the splicing features are weightedly fused to obtain the fused features.

[0162] In one embodiment, when implementing the pre-trained decision model, performing feature analysis on the fusion features, and determining a target processing decision for the breeding environment, the processor is configured to implement:

[0163] Determining a threshold range corresponding to at least one preset processing decision based on a current decision task;

[0164] Based on a comparison result between the fusion feature and the threshold range, the target processing decision corresponding to the fusion feature is determined.

[0165] In one embodiment, when implementing the pre-trained decision model, performing feature analysis on the fusion features, and determining a target processing decision for the breeding environment, the processor is configured to implement:

[0166] Obtaining feature data and data feature thresholds corresponding to at least one historical decision;

[0167] Based on the decision model, performing feature analysis on the fusion feature that matches the feature data to obtain a feature analysis result;

[0168] When the feature comparison result shows that the fusion feature is greater than or equal to the data feature threshold, a historical decision corresponding to the data feature threshold is determined as the target processing decision.

[0169] In one embodiment, before obtaining the feature data and data feature threshold corresponding to at least one historical decision, the processor is further configured to:

[0170] Obtain historical decision data;

[0171] Based on the historical decision data, feature data and a data feature threshold corresponding to at least one historical decision are determined.

[0172] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the livestock multimodal data analysis and decision-making methods based on deep transfer learning provided in the embodiments of the present application.

[0173] The computer-readable storage medium may be an internal storage unit of the computer device described in the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., equipped on the computer device.

[0174] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal data analysis and decision-making method for animal husbandry based on deep transfer learning, characterized in that: The method comprises: Based on the data acquisition sensor, multimodal data in the breeding environment is collected, wherein the multimodal data includes time series data, audio data and image data; Based on the pre-trained transfer learning model, the multimodal data is analyzed to obtain data features corresponding to each modal data; Based on a preset data fusion strategy, the data features corresponding to each modal data are fused to obtain fused features; Based on the pre-trained decision model, feature analysis is performed on the fusion features to determine the target processing decision for the breeding environment; The pre-trained decision model performs feature analysis on the fusion feature to determine the target processing decision for the breeding environment, including: determining a threshold range corresponding to at least one preset processing decision based on the current decision task; and determining the target processing decision corresponding to the fusion feature based on the comparison result between the fusion feature and the threshold range.

2. The livestock multimodal data analysis and decision-making method based on deep transfer learning according to claim 1, characterized in that: The data acquisition sensor is used to collect multimodal data in the breeding environment, including: Based on the image sensor, images of feces and animal behavior in the breeding environment are collected to obtain the image data; Based on the audio sensor, the audio data in the breeding environment is collected; Based on the time series data acquisition sensor, the environmental parameter time series data and the feces characteristic time series data in the breeding environment are collected to obtain the time series data.

3. The livestock multimodal data analysis and decision-making method based on deep transfer learning according to claim 1, characterized in that: The pre-trained transfer learning model is used to perform data analysis on the multimodal data to obtain data features corresponding to the multimodal data, including: Based on the image analysis sub-model, performing data analysis on the image data to obtain image data features; Based on the audio analysis sub-model, performing data analysis on the audio data to obtain audio data features; Based on the time series prediction sub-model, data analysis is performed on the time series data to obtain time series data features.

4. The livestock multimodal data analysis and decision-making method based on deep transfer learning according to claim 1, characterized in that: The step of fusing the data features corresponding to each modality data based on a preset data fusion strategy to obtain fused features includes: Based on the time series correspondence, the data features corresponding to each modal data are spliced ​​to obtain splicing features; Based on the cross-modal attention mechanism, the attention weights of the data features corresponding to each modality data under the current decision task are determined; Based on the attention weights of the data features corresponding to each modal data under the current decision task, the splicing features are weightedly fused to obtain the fused features.

5. The livestock multimodal data analysis and decision-making method based on deep transfer learning according to claim 1, characterized in that: The pre-trained decision model is used to perform feature analysis on the fusion features to determine a target processing decision for the aquaculture environment, including: Obtaining feature data and data feature thresholds corresponding to at least one historical decision; Based on the decision model, performing feature analysis on the fusion feature that matches the feature data to obtain a feature analysis result; When the feature comparison result shows that the fusion feature is greater than or equal to the data feature threshold, a historical decision corresponding to the data feature threshold is determined as the target processing decision.

6. The livestock multimodal data analysis and decision-making method based on deep transfer learning according to claim 5, characterized in that: Before obtaining the feature data and data feature threshold corresponding to at least one historical decision, the method further includes: Obtain historical decision data; Based on the historical decision data, feature data and a data feature threshold corresponding to at least one historical decision are determined.

7. A multimodal data analysis and decision-making device for animal husbandry based on deep transfer learning, characterized in that: The animal husbandry multimodal data analysis and decision-making device based on deep transfer learning includes: A data acquisition module, configured to collect multimodal data in the aquaculture environment based on a data acquisition sensor, wherein the multimodal data includes time series data, audio data, and image data; A data analysis module is used to analyze the multimodal data based on a pre-trained transfer learning model to obtain data features corresponding to each modal data; A feature fusion module is used to fuse the data features corresponding to each modal data based on a preset data fusion strategy to obtain a fusion feature; A decision determination module, configured to perform feature analysis on the fusion features based on a pre-trained decision model to determine a target processing decision for the aquaculture environment; The decision determination module is further used to determine a threshold range corresponding to at least one preset processing decision based on the current decision task; and determine the target processing decision corresponding to the fusion feature based on the comparison result between the fusion feature and the threshold range.

8. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored on the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the livestock multimodal data analysis and decision-making method based on deep transfer learning as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by the processor, the steps of the livestock multimodal data analysis and decision-making method based on deep transfer learning as described in any one of claims 1 to 6 are implemented.

Images