Pasture decision management system and method

Abstract

The embodiment of the invention provides a pasture decision management system and method, and belongs to the technical field of pasture management. The system comprises an equipment end used for collecting multi-modal data and sending the multi-modal data to an edge end; the edge end is used for performing behavior recognition by adopting a behavior recognition model based on the dairy cow image data to obtain dairy cow behavior category data; based on the cow audio data, performing sound classification by adopting a sound classification model to obtain cow barking category data; performing abnormal value filtering and aggregation processing based on the sensor data to obtain aggregated data; the multi-modal data, the dairy cow behavior category data, the dairy cow sound category data and the aggregated data are sent to a cloud end; the cloud is used for storing the knowledge graph; updating the knowledge graph based on the data sent by the edge end; and generating a management decision based on the updated knowledge graph. The system is used for overcoming the defects in existing pasture intelligent decision making.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pasture management, in particular to a pasture decision management system and method. BACKGROUND

[0002] With the development of modern animal husbandry towards intelligence, intensification and refinement, the dependence on dairy cow behavior recognition, health monitoring and environmental regulation in pasture management is increasing. More and more pastures begin to deploy image monitoring systems, sound collection devices, environmental sensors, physiological monitoring devices and other Internet of Things terminals to collect the behavior state, health indicators and environmental information of dairy cows in real time. These multi-modal data are non-structured and difficult to manage and analyze. Therefore, traditional pastures usually use single methods for decision management, such as manual monitoring, deep learning target detection, sensor monitoring, etc. These methods are often single-dimensional and lack comprehensiveness, and cannot provide comprehensive decision support for pastures.

[0003] For image visual monitoring, currently, many pastures use fixed cameras or mobile pan-tilt camera systems combined with convolutional neural networks to realize target detection and behavior classification of dairy cows. For example, YOLOv8, Faster R-CNN, MobileNet and other models are used to distinguish behaviors such as feeding, standing and lying of cattle. Some studies also combine time modeling structures (such as LSTM or 3D-CNN) to identify dynamic behaviors (such as walking and climbing) and analyze the health status of dairy cows.

[0004] For sound analysis technology, some studies model the sound changes of cattle in disease state or estrus period, acquire audio data through audio collection devices to extract cow call signals, use acoustic features such as MFCC and Mel spectrogram, and combine CNN or RNN models to build a sound classification system to identify abnormal calls for non-contact early health warning.

[0005] For sensor monitoring, behavior recognition systems based on wearable devices (such as collars and leg rings) are widely used in modern animal husbandry farms. Individual motion data, rumination frequency, physiological indicators, etc. are collected through acceleration sensors, gyroscopes, thermometers, heart rate meters, etc. and time series modeling is performed with decision trees, support vector machines (SVM), LSTM, etc. to identify whether the animal is in a feeding, moving, rumination, lying, estrus, etc. state.

[0006] However, the above monitoring methods have single perception means and can only acquire single data, lack comprehensive analysis capability for multi-modal data, and have poor adaptability and generalization ability for specific scenarios of pastures, making it difficult to meet the needs of precision management. For the expert system-based decision-making method, the experience and knowledge of veterinarians and animal husbandry experts are converted into rules embedded in the system to provide professional decision-making suggestions for ranches, such as disease diagnosis and feed formula recommendation, which can be applied to the health management field of actual ranches and is helpful for ranches lacking animal husbandry professionals. However, this method relies too much on expert experience, has poor flexibility and adaptability, and is difficult to cope with complex and changing breeding conditions. When the ranch environment and cow breeds change, the system rules need to be adjusted frequently by humans. Moreover, it cannot learn and optimize independently, and the decision-making ability improves slowly.

[0007] In addition, the current real-time early warning system has slow response speed and cannot timely discover the disease risk of cows. The current static knowledge graph suggestion has a lagging nature and lacks multi-modal data-driven knowledge updating, making it difficult to achieve precise and real-time decision support and affecting the production efficiency and economic benefits of ranches.

[0008] In summary, the existing technology has deficiencies in intelligent decision-making in ranches and cannot meet the efficient, precise and intelligent management needs of smart ranches. SUMMARY

[0009] The purpose of the embodiments of the present application is to provide a ranch decision-making management system and method to solve the deficiencies in the existing intelligent decision-making of ranches.

[0010] To achieve the above-mentioned purpose, in a first aspect, the embodiments of the present application provide a ranch decision-making management system, which comprises: a device end for collecting multi-modal data and sending the multi-modal data to an edge end, wherein the multi-modal data comprises cow image data, cow audio data and sensor data; an edge end for performing behavior recognition based on the cow image data using a behavior recognition model to obtain cow behavior category data; performing sound classification based on the cow audio data using a sound classification model to obtain cow call category data; performing outlier filtering and aggregation processing based on the sensor data to obtain aggregated data; and sending the multi-modal data, the cow behavior category data, the cow call category data and the aggregated data to a cloud end; and a cloud end for storing a knowledge graph; updating the knowledge graph based on the data sent by the edge end; and generating management decisions based on the updated knowledge graph.

[0011] Optionally, the device end comprises: a camera for collecting cow image data; a microphone array for collecting cow audio data; a temperature sensor for collecting temperature sensor data; and a humidity sensor for collecting humidity sensor data, wherein each device of the device end performs time synchronization processing before collecting data.

[0012] Optionally, the sending, by the device side, of the multi-modal data to the edge side comprises: encrypting the multi-modal data using a TLS encryption technology; and sending the encrypted multi-modal data to the edge side using an MQTT communication protocol.

[0013] Optionally, before the sending, by the edge side, of the multi-modal data to the cloud side, the method further comprises: spatially calibrating the multi-modal data to obtain spatial coordinates of each of the multi-modal data; uniformly timestamping each of the multi-modal data; and determining whether the time of the multi-modal data is aligned based on the timestamps, if aligned, uniformly packaging all of the multi-modal data to obtain first packaged data; if not aligned, first time-aligning all of the multi-modal data, and then uniformly packaging all of the multi-modal data to obtain first packaged data.

[0014] Optionally, the sending, by the edge side, of the multi-modal data, the dairy cow behavior category data, the dairy cow call category data and the aggregated data to the cloud side comprises: uniformly packaging the dairy cow behavior category data, the dairy cow call category data and the aggregated data to obtain second packaged data; encrypting the first packaged data and the second packaged data using a TLS encryption technology, respectively; and sending the encrypted first packaged data and the second packaged data to the cloud side using an MQTT communication protocol, respectively.

[0015] Optionally, the behavior recognition model adopts a pre-trained YOLOv11n-FCA_SFA model, the YOLOv11n-FCA_SFA model is a model improved based on a YOLOv11n model, and the improvement comprises: replacing a residual block in a C3k2 module of the YOLOv11n model with an FCA_SFA module, the FCA_SFA module comprising an SFA module and a DFFN module connected in sequence.

[0016] Optionally, based on the dairy cow audio data, performing sound classification using a sound classification model to obtain dairy cow call category data comprises: performing noise reduction processing on the dairy cow audio data; extracting the noise-reduced dairy cow audio data based on an MFCC acoustic feature extraction algorithm to obtain dairy cow sound features; and performing sound classification on the dairy cow sound features based on a sound classification model to obtain a dairy cow call category, the dairy cow call category comprising normal chewing call, estrus high-frequency call and abnormal groan.

[0017] Optionally, the cloud end is further configured to: train the behavior recognition model and the sound classification model based on the data sent by the edge end; perform model evaluation on the newly trained behavior recognition model to obtain a first evaluation result; perform model evaluation on the newly trained sound classification model to obtain a second evaluation result; if the first evaluation result is better than the evaluation result of the behavior recognition model of the edge end, send the newly trained behavior recognition model to the edge end for model updating; and if the second evaluation result is better than the evaluation result of the sound classification model of the edge end, send the newly trained sound classification model to the edge end for model updating.

[0018] Optionally, the cloud end is further configured to perform double-chain fusion, and the double-chain fusion comprises: inferring a health status of the dairy cow based on the dairy cow behavior category, the dairy cow sound category and the aggregated data; inferring a management strategy based on the health status of the dairy cow; establishing a causal relationship chain based on the multi-modal data, the health status of the dairy cow and the management strategy; and updating a knowledge graph based on the causal relationship chain.

[0019] In another aspect, the embodiment of the present application provides a pasture decision management method, and the method comprises: collecting multi-modal data, wherein the multi-modal data comprises dairy cow image data, dairy cow audio data and sensor data; performing behavior recognition based on the dairy cow image data by using a behavior recognition model to obtain dairy cow behavior category data; performing sound classification based on the dairy cow audio data by using a sound classification model to obtain dairy cow sound category data; performing outlier filtering and aggregation processing based on the sensor data to obtain aggregated data; updating a knowledge graph based on the dairy cow behavior category data, the dairy cow sound category data and the aggregated data; and generating a management decision based on the updated knowledge graph.

[0020] The pasture decision management system provided by the present application adopts a cloud-edge-end integrated architecture, and the computing device of the edge end can acquire data collected by the device end in real time, and the data can be preprocessed by deploying a deep learning model, so that the amount of data transmitted to the cloud end can be effectively reduced, high-frequency and low-delay data processing and preliminary anomaly detection can be realized, the response speed can be greatly improved, and the disease risk of the dairy cow can be discovered in time. After receiving the data, the cloud end can directly use the data for updating the knowledge graph. With continuous updating of real-time data, the generated management decision can be more refined, so that more real-time and comprehensive decision support can be provided, and the production efficiency and economic benefits of the pasture can be improved.

[0021] Other features and advantages of the embodiments of the present application will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings are included to provide a further understanding of embodiments of the application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the application, and together with the description serve to explain embodiments of the application, but do not limit the application. In the drawings: Figure 1 is a structural schematic diagram of a pasture decision management system provided by an embodiment of the application; Figure 2 is a multi-modal data standardization process schematic diagram provided by an embodiment of the application; Figure 3 is a YOLOv11n-FCA_SFA network structure schematic diagram provided by an embodiment of the application; Figure 4 is a FCA_SFA network structure schematic diagram provided by an embodiment of the application; Figure 5 is a SFA network structure schematic diagram provided by an embodiment of the application; Figure 6 is a MFCC feature extraction process schematic diagram provided by an embodiment of the application; Figure 7 is a CNN network structure schematic diagram provided by an embodiment of the application; Figure 8 is a management decision generation process schematic diagram provided by an embodiment of the application; Figure 9 is a cloud-edge-end architecture technology process schematic diagram provided by an embodiment of the application; Figure 10 is a cloud-edge-end architecture schematic diagram provided by an embodiment of the application; Figure 11 is a process schematic diagram of a pasture decision management method provided by an embodiment of the application. DETAILED DESCRIPTION

[0023] The specific embodiments of the application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the application, and are not intended to limit the application.

[0024] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present application comply with relevant provisions of laws and regulations. In the embodiments of the present application, some industry existing solutions, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solutions.

[0025] Figure 1 is a structural schematic diagram of a pasture decision management system provided by an embodiment of the application, as Figure 1The system comprises a device end, an edge end and a cloud end.

[0026] The device end is configured to collect multi-modal data, and send the multi-modal data to the edge end, wherein the multi-modal data comprises cow image data, cow audio data and sensor data.

[0027] The edge end is configured to perform behavior recognition based on the cow image data by using a behavior recognition model to obtain cow behavior category data, perform sound classification based on the cow audio data by using a sound classification model to obtain cow call category data, perform outlier filtering and aggregation processing based on the sensor data to obtain aggregated data, and send the multi-modal data, the cow behavior category data, the cow call category data and the aggregated data to the cloud end.

[0028] The cloud end is configured to store a knowledge graph, update the knowledge graph based on the data sent by the edge end, and generate a management decision based on the updated knowledge graph.

[0029] Although there are many methods for monitoring the health status of cows at present, and the purpose of providing management suggestions and decision support for the farm is achieved. However, the current method only relies on one type of data for single modal analysis. For example, only one of the cow image data, cow audio data and sensor data is used to make decisions. Among them, the method of relying solely on cow behavior image data to identify cow behavior by using a target detection model can only monitor from the behavior dimension, and when the cow shows abnormal behavior in the video, the warning is usually highly lagging, and the potential health problem cannot be found; the technology of using audio analysis alone is usually difficult to determine the individual of the cow's voice, and is greatly affected by other sounds in the farm, such as wind sound and mechanical noise of equipment, which will bring challenges to audio recognition; the method of using sensors alone, many existing technologies choose to install sensor devices on cows, which has high equipment cost, is easy to lose or damage, and may cause discomfort to cows and produce stress response.

[0030] The system provided by the present application can collect multi-modal data at the device end, and use image, audio and sensor data to provide decision support for the pasture. When the system is working, multi-modal data is first collected at the device end, and then data of different modalities is sent to the edge end. After the edge end receives the data sent by the device end, it completes part of the inference task, that is, behavior recognition is performed by using a behavior recognition model, sound classification is performed by using a sound classification model, and outlier filtering and aggregation processing are performed, and then the processing result and the multi-modal data are sent to the cloud end. The cloud end stores a knowledge graph, and when the cloud end receives the data, the data can be stored in a MySQL database, and the knowledge graph is updated based on the received data. When updating the knowledge graph, the cow behavior category data, the cow sound category data and the aggregation data provided by the edge end are embedded into the semantic knowledge graph in the form of triples to realize the expression of the causal chain between the behavior events, the audio events and the environmental factors. After updating the knowledge graph, reasoning is performed according to the knowledge graph, and finally cow individualized management suggestions and health early warning information are generated.

[0031] The system of the "cloud-edge-end integrated architecture" provided by the present application deploys an edge computing device, that is, an edge end, in the pasture, acquires data collected by the device end in real time through the edge computing device, and pre-processes the data by deploying a deep learning model, so that the amount of data transmitted to the cloud end can be effectively reduced, high-frequency and low-delay data processing and preliminary anomaly detection can be realized, the response speed is greatly improved, and the disease risk of the cow can be discovered in time. After the cloud end receives the data, the data can be directly used for updating the knowledge graph. With the continuous updating of real-time data, the generated management decisions can also be more refined, and higher real-time and more comprehensive decision support can be provided to improve the production efficiency and economic benefits of the pasture.

[0032] In addition, the edge computing device is deployed in a position relatively close to the data source, and the powerful computing capacity and low delay characteristics of edge computing are used to pre-process and preliminarily analyze real-time data acquisition, thereby reducing the load of the cloud server.

[0033] The cloud-edge-end collaborative system architecture provided by the present application is based on multi-modal data and a knowledge graph, can realize data standardization integration, cloud-edge efficient collaborative calculation, precise multi-modal analysis and intelligent decision support, and has important practical significance for promoting the development of animal husbandry. The system comprehensively improves the ability of the intelligent pasture in data fusion, behavior recognition, health management and intelligent decision-making, and has the outstanding advantages of precision and real-time.

[0034] Further, the device end comprises: a camera for collecting cow image data; a microphone array for collecting cow audio data; a temperature sensor for collecting temperature sensor data; and a humidity sensor for collecting humidity sensor data.

[0035] The device end of the system is connected with the cloud end through the edge end.

[0036] Unlike existing sensor devices installed on cows, the system only needs to set temperature sensors and humidity sensors to collect environmental factors, thereby avoiding the problem of causing discomfort and stress reactions to cows.

[0037] Further, the device end sends the multi-modal data to the edge end, including: encrypting the multi-modal data using TLS encryption technology; and sending the encrypted multi-modal data to the edge end using the MQTT communication protocol.

[0038] Through the lightweight MQTT communication protocol combined with the TLS encryption technology for data transmission, it can prevent data from being stolen or tampered with during transmission, ensure safe and efficient transmission of data, and protect the safety of core data of the ranch.

[0039] Further, before the edge end sends the multi-modal data to the cloud end, it also includes: spatial calibration of the multi-modal data to obtain the spatial coordinates of each multi-modal data; time stamp unification of each multi-modal data; and based on the time stamp, determining whether the time of the multi-modal data is aligned, if aligned, then uniformly packaging all the multi-modal data to obtain first packaging data; if not aligned, first aligning the time of all the multi-modal data, and then uniformly packaging all the multi-modal data to obtain first packaging data.

[0040] Because the multi-modal data obtained by each device has different formats and lacks a unified standard, it is difficult to effectively integrate; different modal data has poor synchronization in time and space, and cannot establish precise space-time correlation, which is not conducive to multi-modal data analysis, therefore, the system proposed in the present application first uniformly processes and fuses the multi-modal data after obtaining the multi-modal data at the edge end, and then converts the processed multi-modal data into a unified format before sending it to the cloud for storage in the MySQL database. Figure 2 The collection and standardization of multi-modal data are carried out through the following steps S101 to S108.

[0041] Step S101: synchronize the time of all devices.

[0042] Specifically, each device at the device end is executed: device time <- synchronize_NTP (NTP time source); and the device system time is set to device time.

[0043] Step S102: multi-modal data collection.

[0044] Specifically, this includes: cameras collecting image data of dairy cows; microphone arrays collecting audio data of dairy cows; temperature sensors collecting temperature data; and humidity sensors collecting humidity data.

[0045] Step S103: Image data spatial calibration.

[0046] Specifically, for each image in the cow image data, the following steps are performed: obtain intrinsic and extrinsic parameters based on the camera ID and camera parameters corresponding to the image; calculate the spatial position of the image based on the image, intrinsic and extrinsic parameters; and obtain the spatial coordinates of the image based on the spatial position of the image.

[0047] Step S104: Audio data spatial calibration (TDOA sound source localization).

[0048] Specifically, for each audio segment in the cow audio data, the following steps are performed: obtain the microphone position based on the microphone array ID and microphone parameters corresponding to the audio; perform TDOA positioning based on the audio data and microphone position to obtain the sound source coordinates of the audio; and obtain the spatial coordinates of the audio based on the sound source coordinates.

[0049] Step S105: Sensor data spatial calibration (direct mapping).

[0050] Specifically, for each piece of sensor data, the following steps are performed: obtain the spatial coordinates of the sensor data based on the preset deployment location coordinates.

[0051] Step S106: Timestamp unification (using synchronized timestamps).

[0052] Specifically, perform the following for all multimodal data: obtain the timestamp after multimodal data synchronization.

[0053] Step S107: Time alignment (DTW dynamic time warping).

[0054] Specifically, this includes: first, determining whether the timestamps are aligned; if aligned, uniformly encapsulating all the multimodal data; if not aligned, performing dynamic time warping to align the timestamps of all the multimodal data, and then uniformly encapsulating all the multimodal data again. Dynamic time warping includes: sorting image data by timestamps to obtain image sequences; sorting audio data by timestamps to obtain audio sequences; sorting sensor data by timestamps to obtain sensor sequences; and using DTW (Dynamic Time Warping) to align the timestamps of each sequence.

[0055] Step S108: Encapsulate into a standard format (JSON).

[0056] Specifically, for each time point in the alignment index, the following is performed: obtaining a standard entry, and adding the standard entry to the standardized data. The standard entry includes: "timestamp": time point, "image": obtaining image data (image sequence, time point), "audio": obtaining audio data (audio sequence, time point), "sensor": obtaining sensor data (sensor sequence, time point), "space": { "image": image. spatial coordinates, "audio": audio. spatial coordinates, "sensor": sensor. spatial coordinates } Through the above steps S101 to S108, the system clock of each collection terminal is unified through the network time protocol (NTP), so that all devices are time-synchronized, and all data have high-precision time stamps. Then, the collected multi-modal data is space-time calibrated. For image data, the spatial position of the image shooting area is calculated using camera internal and external parameter data through camera position and orientation information; for microphone array audio data, the spatial position of the sound source is inferred using a sound source positioning algorithm (TDOA); for sensor data, the coordinates are directly labeled through the deployment position. Finally, the data of different modalities have sampling frequency differences and time axis offsets, so a dynamic time warping algorithm (DTW) is used to fuse the time dimension, and the data of different modalities are calibrated in the time dimension to align the data with time misalignment. Different formats and frequencies of raw information such as image, audio, and sensor data are uniformly packaged into JSON format data to complete data standardization. The data after standardization and packaging can be uniformly parsed, facilitating deep integration with the knowledge graph, realizing data-to-knowledge conversion and real-time dynamic updating.

[0057] Further, the edge end sending the multi-modal data, the dairy cow behavior category data, the dairy cow call category data, and the aggregated data to the cloud end includes: uniformly packaging the dairy cow behavior category data, the dairy cow call category data, and the aggregated data to obtain second packaging data; encrypting the first packaging data and the second packaging data using TLS encryption technology respectively; and sending the encrypted first packaging data and the second packaging data to the cloud end using the MQTT communication protocol respectively.

[0058] Further, the behavior recognition model adopts a pre-trained YOLOv11n-FCA_SFA model, the YOLOv11n-FCA_SFA model is a model improved based on a YOLOv11n model, and the improvement includes: replacing a residual block in a C3k2 module of the YOLOv11n model with an FCA_SFA module, and the FCA_SFA module includes an SFA module and a DFFN module connected in sequence.

[0059] The behavior recognition model provided in the application labels six different behavior categories (foraging, drinking, standing, lying, walking and climbing) of the dairy cow with Labelimg software during training to construct a dairy cow behavior data set, and the dairy cow behavior description is shown in Table 1.

[0060] Table 1 Basic behavior description of dairy cow

[0061] To ensure the consistency and accuracy of the data set, the dairy cow behavior data set in different scenes is divided into a training set, a validation set and a test set according to a ratio of 7:2:1, then data augmentation is used to expand the data set, the new dairy cow behavior data set has a total of 6264 images, the training set of the new data set has 4364 images, the validation set has 1235 images, and the test set has 665 images, then YOLOv11n-FCA_SFA is used for dairy cow behavior recognition training to construct a dairy cow behavior recognition detection model.

[0062] In order to realize efficient and accurate detection of dairy cow individuals and key behaviors, the behavior recognition model provided in the application adopts a deep learning recognition framework based on a YOLOv11n target detection algorithm, and uses an FCA_SFA module (or a C3k2-FCA_SFA module) to improve the model, and the improved model is YOLOv11n-FCA_SFA. Figure 3 is a YOLOv11n-FCA_SFA network structure schematic diagram provided by an embodiment of the application, Figure 4 is a FCA_SFA network structure schematic diagram provided by an embodiment of the application, Figure 5 is a SFA network structure schematic diagram provided by an embodiment of the application. As shown in Figure 3 The application replaces the third and fourth C3k2 modules in the Backbone of YOLOv11n-FCA_SFA with FCA_SFA modules. FCA_SFA is a redesign of the C3k2 module in the YOLOv11 model, including replacing the residual block in the C3k2 module with an FCA_SFA module. As shown in Figure 4As shown, the FCA_SFA module fuses a space-frequency attention mechanism (SFA) and a dynamic feedforward network (DFFN). The network structure of the SFA is as shown in FIG. 2. Figure 5 As shown, the space-frequency attention (SFA) can simultaneously capture local features and global features of the cow behavior, and is particularly suitable for recognition of dynamic behaviors (walking, climbing) and spatial structure analysis of static behaviors (standing, lying). The deep feedforward network (DFFN) enhances the feature expression capability through nonlinear transformation, fuses the behavior features of different channels, and improves the recognition ability of confused behaviors. Therefore, the FCA_SFA module can capture subtle posture changes, local dynamic regions, and complex interactive behaviors between individuals, thereby improving the recognition accuracy of the model in complex scenarios with small action amplitude or subtle behavior differences. Replacing the C3k2 module with the FCA_SFA module can significantly enhance the perception ability of the cow behavior, thereby improving the overall recognition effect.

[0063] In order to verify the effectiveness of the improved cow behavior recognition model YOLOv11n-FCA_SFA, in the present application, four evaluation indexes are selected to evaluate the network model, and the four evaluation indexes are precision (Precision), recall (Recall), mAP50 and F1. The experiments before and after improvement are carried out in the test set, and the evaluation results are shown in Table 2.

[0064] Table 2 Comparison and analysis of evaluation results

[0065] From the above comparison results, it can be seen that although the recall rate (Recall) of the improved model as a whole has decreased, the precision (Precision), mAP50 and F1 values have been greatly improved, among which the precision (Precision) has been improved by 3.5%, the mAP50 has been improved by 1%, and the F1 value has been improved by 1.3%. The improved model can accurately recognize the cow behavior, and the overall detection performance of the model is better, which can meet the demand for cow behavior recognition.

[0066] Further, based on the audio data of the dairy cow, a sound classification model is used for sound classification to obtain dairy cow call category data, including: performing noise reduction processing on the audio data of the dairy cow; performing extraction on the dairy cow audio data after noise reduction processing based on an MFCC (Mel-Frequency Cepstral Coefficients) acoustic feature extraction algorithm, converting the original audio into low-dimensional and stable feature representation for model learning, to obtain dairy cow sound features; and performing sound classification on the dairy cow sound features based on a sound classification model to obtain dairy cow call categories, the dairy cow call categories including normal chewing call, estrus high-frequency call, and abnormal groan. In some embodiments of the present application, the dairy cow call categories can also include other categories of calls, and the above-mentioned dairy cow call categories are only used for illustration and do not limit the dairy cow call categories.

[0067] Figure 6 is a MFCC feature extraction process schematic diagram provided by an embodiment of the present application, Figure 7 is a CNN network structure schematic diagram provided by an embodiment of the present application, please refer to Figure 6 and Figure 7 The process of extracting sound features by MFCC mainly includes the following steps: the original speech signal is pre-emphasized, framed, windowed, fast Fourier transformed (FFT), Mel filter banked, logarithm calculated, and discrete cosine transformed (DCT), and finally the MFCC coefficients are obtained.

[0068] When extracting features from the dairy cow audio, pre-emphasis processing is first used, that is, the original dairy cow audio signal is passed through a high-pass filter. Pre-emphasis processing can improve the high-frequency components of the dairy cow audio spectrum features, improve the signal-to-noise ratio, and reduce signal distortion. The formula is as follows:

[0069] Among them, is the original dairy cow audio signal, is the pre-emphasized audio signal, is the filter coefficient, which is usually between 0.9 and 1.

[0070] The speech signal is a short-time stationary signal, so Fourier transform is performed on the short-time frame, and a good approximation of the signal frequency profile is obtained by connecting adjacent frames. After the signal is framed, each frame is multiplied by a window function, such as a Hamming window, to increase the continuity of the left and right ends of the frame. The windowing formula is as follows:

[0071]

[0072] Among them, is the framed audio signal, is the window function formula, is the windowed audio signal, is the window length, a is the window function parameter, when is called Hamming window.

[0073] After framing and windowing, a fast Fourier transform of N points is performed on each frame of the audio signal, and the formula is as follows: ,

[0074] wherein, is the number of points of the Fourier transform, usually is 256 or 512.

[0075] After the fast Fourier transform, the log energy of each filter bank output is as follows: ,

[0076] wherein, is the number of Mel filters, usually is 40.

[0077] Finally, the discrete cosine transform (DCT) is applied to decorrelate the filter bank coefficients and generate a compressed representation of the filter bank, the MFCC coefficient formula is as follows: ,

[0078] wherein, is the order of the MFCC coefficient.

[0079] Further, the cloud is also used for: based on the data sent by the edge, the behavior recognition model and the sound classification model are trained; the newly trained behavior recognition model is evaluated, and a first evaluation result is obtained; the newly trained sound classification model is evaluated, and a second evaluation result is obtained; if the first evaluation result is better than the evaluation result of the behavior recognition model of the edge, the newly trained behavior recognition model is sent to the edge for model updating; and if the second evaluation result is better than the evaluation result of the sound classification model of the edge, the newly trained sound classification model is sent to the edge for model updating.

[0080] The cloud performs deep learning model training tasks based on multi-modal data to determine whether the deep learning model is updated, so as to continuously improve the performance of the behavior recognition model and the sound classification model.​

[0081] Further, the cloud is also used for double-chain fusion, the double-chain fusion comprising: inferring a health status of the dairy cow based on the dairy cow behavior category, the dairy cow call category and the aggregated data; inferring a management strategy based on the health status of the dairy cow; establishing a causal relationship chain based on the multi-modal data, the health status of the dairy cow and the management strategy; and updating the knowledge graph based on the causal relationship chain.

[0082] It should be noted that the double-chain fusion is executed after the knowledge graph is updated based on the data sent by the edge end. Figure 8 is a management decision generation process schematic diagram provided by an embodiment of the present application, including the process from the cloud receiving the data sent by the edge end to generating a management decision. As shown in Figure 8 After the cloud receives the dairy cow behavior category data, the dairy cow call category data and the aggregated data, the data is converted into the form of triples of entities, attributes and relationships for updating the knowledge graph. When updating, for each triple, it is judged whether the entity exists in the knowledge graph, if it exists, the corresponding entity attribute or edge weight is updated based on the triple, if it does not exist, a new entity node and a relationship edge are created, finally the updated knowledge graph is written into the Neo4j database using py2neo, completing the updating of the knowledge graph, so that the knowledge graph always keeps synchronized with the actual state in the farm. Then, double-chain fusion is performed, which is to deeply fuse the multi-modal data chain (images, sounds, sensors, etc.) collected in the farm with the knowledge graph (knowledge chain), perform causal reasoning and intelligent decision output, that is, combine the behavior category data, the call category data and the aggregated data with the knowledge graph to output a management decision, realizing real-time judgment of individual dairy cows and group health status, abnormal early warning and management suggestion output. Specifically, multi-modal data reasoning is first performed based on the multi-modal data chain, the health status of the dairy cow is comprehensively analyzed according to the behavior category data, the call category data and the aggregated data, and then the corresponding management strategy is output, such as adjustment of feeding strategy, human health intervention measures, etc., establishing a causal relationship chain. It should be noted that the knowledge graph is constructed according to the causal relationship chain established based on the previous data when constructing. The causal relationship chain established at this time is used to match and update the knowledge graph according to the "rule template" in the existing knowledge graph, thereby completing the double-chain fusion. Then the behavior category data, the call category data and the aggregated data are mapped into the knowledge graph, joint reasoning is realized by finding nodes, and a management decision is output.

[0083] The connection between multi-modal data and knowledge graph is established through double-chain fusion, so that the knowledge graph can be dynamically updated using multi-modal data, and a dynamic semantic reasoning framework of double-chain fusion of data chain and knowledge chain is constructed. The knowledge chain adopts a graph database (Neo4j) as a graph management tool, and the behavior characteristics information of the cows in the image, the sounds of different types of cows in the audio, and the pasture environment monitored by the sensor are taken as nodes. By analyzing the potential relationship between multi-modal data, such as the association between cow behavior and environmental data, the association between sound characteristics and health status, etc., the edge connection between nodes is established, and a complete data correlation graph is formed, so that the internal relationship between multi-modal data is clearly displayed. After combining multi-modal data with a graph reasoning engine, the system has the ability to accurately identify the complex state of the behavior of the cattle, the sound of the cows, the health abnormalities, the environmental stress, etc., and can generate accurate and real-time decision support and health warning prompts according to real-time multi-modal data and relying on the prior knowledge in the knowledge graph, which significantly improves the intelligent management level of the pasture and the animal welfare protection ability of the cows.

[0084] Among them, the system can use existing technical means to identify health abnormalities and environmental stress, and the present application will not be described in detail. As an example, for health abnormalities, the possible health problems can be analyzed and inferred according to the behavior, sound, environmental information, etc. of the cows. For environmental stress, environmental problems can be judged according to the temperature and humidity sensor data, such as detecting that the temperature is too high through the temperature sensor, which may cause heat stress response in the cows, and then affect the foraging and milk yield of the cows; detecting that the temperature is too low through the temperature sensor, cold will cause the body function of the cows to decline, and the activity will decrease, which will increase the foraging demand; detecting that the humidity is too high through the humidity sensor, which will aggravate the heat stress response of the cows and increase the risk of hoof disease and bacterial disease.

[0085] After the system identifies the complex state, the prior knowledge in the knowledge graph can also be used to implement the reasoning, and the present application will not be described in detail. As an example, the reasoning path includes the mourning sound → activates the node "pain signal" → hoof disease risk → manual inspection intervention; high-frequency mating call → activates the node "cow estrus" → arranges mating; the foraging time of the cows is reduced → activates the node "foraging time reduction" → poor palatability of feed, high feeding density → changes the type of feed, controls the feeding time density, etc.

[0086] In the reasoning process, first, according to the triple set of the mapped behavior category data, the call category data and the aggregation data, trigger condition matching is carried out in the updated knowledge graph to activate the corresponding node; then based on the activated node, graph rule reasoning is carried out to obtain a reasoning path; finally, based on the reasoning path, management suggestion generation is carried out to obtain a management decision. As an example, when the temperature is greater than 30 DEG C, the node "high temperature environment" is activated, the reasoning obtains that the cow will produce heat stress reaction, and finally the management suggestion is given. The intelligent reasoning mode of the double-chain fusion can better provide more fine decision support for the pasture managers.

[0087] Figure 9 is a cloud edge end architecture technical process schematic diagram provided by an embodiment of the application, Figure 10 is a cloud edge end architecture schematic diagram provided by an embodiment of the application. As shown in Figure 9 and Figure 10 The working process of the cloud edge end architecture of the system provided by the application includes the following steps S201 to S206.

[0088] Step S201: device end multi-modal data acquisition.

[0089] Step S202: the device end transmits the data to the edge end through the MQTT protocol.

[0090] Specifically, it includes: after the original multi-modal data collected is encrypted by TLS, safe data is obtained, and then the safe data is sent to the edge end based on the edge end address through the MQTT communication protocol.

[0091] Step S203: edge end data processing.

[0092] Specifically, it includes: for the image, the behavior category recognition is carried out by using the YOLOv11n-FCA_SFA model to obtain the cow behavior category data; for the audio, noise reduction processing, MFCC acoustic feature extraction and CNN sound classification are adopted to obtain the cow call category data; for the sensor data, the outlier filtering is adopted to obtain the filtered data, and then the aggregation data is obtained through data aggregation; the three types of data are standardized; and the cow behavior category data, the cow call category data and the sensor aggregation data are encapsulated, and the standardized data are encapsulated.

[0093] Step S204: the edge end transmits the preliminary processing result to the cloud end through the MQTT protocol.

[0094] Specifically, it includes: first, all the encapsulated data are encrypted by TLS to obtain safe data, and then the safe data is sent to the cloud end based on the cloud end address through the MQTT communication protocol.

[0095] It should be noted that during transmission, the second encapsulated data, i.e., the encapsulated cow behavior category data, cow call category data and sensor aggregated data, is preferentially transmitted for inference and output of decisions. Then the first encapsulated data, i.e., the standardized multi-modal data, is transmitted for training of the model. Through this transmission mode, the amount of data transmitted at a time can be avoided to be too large, and the transmission speed can be affected. Based on the setting of preferentially transmitting the second encapsulated data, the response speed of the cloud can be ensured, and decisions can be output in time.

[0096] Step S205: cloud processing.

[0097] Specifically, it includes: receiving data of the edge end and storing it into a MySQL database; training a model by using historical data in the MySQL database and newly received data; determining whether the model needs to be updated, if so, performing TLS encryption on the model, and then sending it to the edge end through an MQTT communication protocol; updating a knowledge graph; double-chain fusion; decision support; and enabling a manager to realize management through an API and a development interface.

[0098] Step S206: cloud output of decisions.

[0099] Specifically, cloud decision inference is performed according to the knowledge graph, decisions are obtained and output.

[0100] The cloud performs data storage, training of a deep learning model, and multi-modal data analysis and knowledge inference. This architecture can reasonably allocate tasks, optimize resource configuration, ensure reasonable calling of resources, improve response speed, realize fast inference of intelligent decisions based on multi-modal data and a knowledge graph, and meet the demand for real-time performance.

[0101] Through collaborative work of the device end, the edge computing platform and the cloud, the system can dynamically adjust real-time monitoring of the pasture, optimize configuration of resources, and support optimization of scheduling decisions. A cloud administrator can access the cloud through an API to perform operations such as resource deployment and environment configuration; and a pasture manager uses a development interface to obtain data query, model training and other related cloud services through a service agent.

[0102] Figure 11 is a flowchart of a pasture decision management method provided by an embodiment of the present application, as shown in the figure, the method includes the following steps S1 to S6. Figure 11

[0103] Step S1: multi-modal data is collected, and the multi-modal data includes cow image data, cow audio data and sensor data.

[0104] Step S2: based on the cow image data, a behavior recognition model is used for behavior recognition to obtain cow behavior category data.

[0105] ​Step S3: based on the audio data of the dairy cow, sound classification is performed by using a sound classification model to obtain dairy cow call category data.

[0106] Step S4: based on the sensor data, outlier filtering and aggregation processing are performed to obtain aggregated data.

[0107] Step S5: based on the dairy cow behavior category data, the dairy cow call category data, and the aggregated data, a knowledge graph is updated.

[0108] Step S6: based on the updated knowledge graph, a management decision is generated.

[0109] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer-usable program code embodied therein.

[0110] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions that are executed by the processor of the computer or other programmable data processing apparatus generate an apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0111] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatuses that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 steps of a function specified in one or more blocks.

[0113] In one typical arrangement, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0114] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or other memory technologies, about which the processor can execute instructions. The memory can be a memory storage device of any type.

[0115] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technologies, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0117] The above merely provides an embodiment of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A ranch decision management system, characterized in that, The system includes: On the device side, it is used to collect multimodal data and send the multimodal data to the edge end, wherein the multimodal data includes cow image data, cow audio data and sensor data; At the edge, the system is used to perform behavior recognition based on the cow image data using a behavior recognition model to obtain cow behavior category data; to perform sound classification based on the cow audio data using a sound classification model to obtain cow moo category data; to perform outlier filtering and aggregation processing based on the sensor data to obtain aggregated data; and to send the multimodal data, the cow behavior category data, the cow moo category data, and the aggregated data to the cloud; and The knowledge graph is stored in the cloud; the knowledge graph is updated based on data sent from the edge device; and management decisions are generated based on the updated knowledge graph.

2. The system according to claim 1, characterized in that, The device includes: Cameras are used to collect image data of dairy cows; Microphone array used to collect audio data from dairy cows; Temperature sensor, used to collect temperature sensor data; and A humidity sensor is used to collect humidity data. Each device on the device side performs time synchronization processing before collecting data.

3. The system according to claim 1, characterized in that, The device sends the multimodal data to the edge device, including: The multimodal data is encrypted using TLS encryption technology; and The encrypted multimodal data is sent to the edge terminal using the MQTT communication protocol.

4. The system according to claim 1, characterized in that, Before the edge device sends the multimodal data to the cloud, it also includes: Spatial calibration is performed on the multimodal data to obtain the spatial coordinates of each multimodal data point; The timestamps of each of the aforementioned multimodal data are unified; and Based on the timestamp, it is determined whether the times of the multimodal data are aligned. If they are aligned, all the multimodal data are uniformly encapsulated to obtain the first encapsulated data. If they are not aligned, all the multimodal data are first time aligned, and then all the multimodal data are uniformly encapsulated to obtain the first encapsulated data.

5. The system according to claim 4, characterized in that, The edge device sends the multimodal data, the cow behavior category data, the cow moo category data, and the aggregated data to the cloud, including: The cow behavior category data, the cow moo category data, and the aggregated data are uniformly encapsulated to obtain the second encapsulated data; The first encapsulated data and the second encapsulated data are respectively encrypted using TLS encryption technology; and The encrypted first and second encapsulated data are sent to the cloud using the MQTT communication protocol.

6. The system according to claim 1, characterized in that, The behavior recognition model employs a pre-trained YOLOv11n-FCA_SFA model, which is an improved version of the YOLOv11n model. The improvements include: Replace the residual block in the C3k2 module of the YOLOv11n model with the FCA_SFA module, which includes an SFA module and a DFFN module connected in sequence.

7. The system according to claim 1, characterized in that, Based on the aforementioned cow audio data, a sound classification model was used to classify the sounds, resulting in cow mooing category data including: The audio data of the cows is subjected to noise reduction processing; The MFCC acoustic feature extraction algorithm was used to extract features from the denoised cow audio data, resulting in the characteristic sounds of the cows; and The features of the cow's sounds are classified based on a sound classification model to obtain cow sound categories, which include normal chewing sounds, high-frequency estrus sounds, and abnormal groans.

8. The system according to claim 1, characterized in that, The cloud is also used for: The behavior recognition model and the sound classification model are trained based on the data sent from the edge device; The newly trained behavior recognition model is evaluated to obtain a first evaluation result; The newly trained sound classification model is evaluated to obtain a second evaluation result; If the first evaluation result is better than the evaluation result of the behavior recognition model at the edge, then the newly trained behavior recognition model is sent to the edge for model update; as well as If the second evaluation result is better than the evaluation result of the sound classification model at the edge, then the newly trained sound classification model is sent to the edge for model update.

9. The system according to claim 1, characterized in that, The cloud platform is also used for dual-chain fusion, which includes: Based on the cow behavior categories, cow call categories, and aggregated data, the health status of the cows is inferred; Based on the health status of the cows, a management strategy is deduced; Based on the multimodal data, the health status of the dairy cows, and the management strategy, a causal relationship chain is established; and The knowledge graph is updated based on the causal relationship chain.

10. A ranch decision-making management method, characterized in that, The method includes: Collect multimodal data, including cow image data, cow audio data, and sensor data; Based on the cow image data, a behavior recognition model is used to identify the cow behavior categories. Based on the aforementioned cow audio data, a sound classification model is used to classify the sounds and obtain cow moo category data. Based on the sensor data, outlier filtering and aggregation are performed to obtain aggregated data; The knowledge graph is updated based on the cow behavior category data, the cow moo category data, and the aggregated data; and Management decisions are generated based on the updated knowledge graph.

Images