Novel cow digital twin behavior perception modeling method and system based on fusion of sensor data and video data
The fusion of sensor and video data for cattle behavior recognition enhances accuracy and reduces stress by leveraging the strengths of both modalities, addressing the limitations of single-modal methods in distinguishing visually similar behaviors.
Patent Information
- Application Number
- CN202411190892.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing cattle behavior recognition technologies face challenges in accurately distinguishing visually similar behaviors and require multiple sensors, leading to stress and inefficiency, while non-contact methods struggle with performance under long distances and occlusion.
A method and system for cattle digital twin behavior perception modeling that fuses sensor data and video data, involving data set construction, feature extraction, alignment, and multimodal data fusion using EfficientNet, BiLSTM, and Transformer encoder for enhanced classification.
Improves the accuracy and comprehensiveness of cattle behavior recognition, reducing stress on animals and enhancing classification capabilities by integrating video and sensor data, addressing the limitations of single-modal approaches.
Smart Images

Figure CN119091211A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of animal husbandry technology and discloses a new method and system for digital twin behavior perception modeling of cattle based on the fusion of sensor data and video data. Background Art
[0002] The construction of the digital shadow of cattle can fully understand the behavior, health status and production performance of each cattle, which plays an important role in ensuring animal welfare, improving yield and promoting the breeding efficiency of cattle. The digital shadow of cattle covers many aspects of cattle information, among which the collection and analysis of cattle behavior data is of great significance to understanding the health status, estrus status and disease status of cattle. With the advancement and development of sensors, artificial intelligence and big data technology, it has become possible to obtain data in real time to identify cattle behavior. At present, ranches use a variety of sensing devices, such as accelerometers, cameras, cattle sensors, scales, etc., to monitor cattle activities, diet, physical condition and other indicators, as well as to detect diseases and reveal estrus. The behavior recognition methods of cattle using the above sensing devices can be divided into wearable and non-contact methods. Among them, the wearable method uses wearable sensors (such as accelerometers, inertial measurement units, pedometers and magnetometers, etc.) to collect different behavior action data, and uses machine learning technology to identify and track animal behavior. The non-contact method is mainly based on the camera using machine vision technology to identify animal behavior, which can identify the basic movement behavior of cattle without interfering with animals. The data used by the two methods are currently heterogeneous in form and independent of each other in method. Therefore, how to make full use of the data generated by existing equipment, taking into account the characteristics and advantages of different modal data, improve the accuracy and precision of cattle behavior recognition methods, and construct a digital shadow of cattle with richer information is the key issue to be solved in this study.
[0003] The behavior perception method of cattle based on wearable devices such as collars uses various sensor data to automatically identify the behavior of cattle, reduce manual labor, improve management efficiency, and reduce human interference and errors. Early researchers used pedometers as an auxiliary means to reveal the estrus behavior of cattle. With the development of computer technology, researchers use accelerometers and personal computers to record and measure the activity level of cattle. In recent years, more and more researchers have used machine learning or deep learning methods to study the behavior of cattle. Shen et al. used three-axis acceleration sensor data through machine learning algorithms to identify the feeding and rumination behaviors of cattle. Tian et al. used multiple sensors to collect cattle behavior data and used KNN and RF models to identify seven behaviors, among which the average recognition accuracy of the KNN-RF fusion model reached 98.51%. Liu et al. used a fully convolutional network (FCN) algorithm to automatically classify 7 types of cattle behavior patterns through an inertial measurement unit (IMU), with a recognition accuracy of more than 80%. However, there are also some problems with the behavioral recognition of cattle based on wearable devices. For example, due to insufficient use of existing data, some behaviors are not well distinguished based on sensor data alone. Existing studies have shown that eating, standing, and moving are the most difficult behaviors to distinguish in classic machine learning models. In addition, the wearing position of the sensor will have a great impact on the classification results. The combined data of accelerometers installed on the neck and legs are used to classify the behavior of cattle. The model classifies lying, standing, and eating behaviors with high accuracy (80-99%) and high sensitivity (87-99%). In order to achieve high-precision behavioral differentiation, multiple sensors are worn at different locations for cattle, which increases the risk of stress reactions in cattle.
[0004] The non-contact behavior perception method based on the camera uses RGB or grayscale images or video data of cattle and deep learning algorithms to classify the behavior of cattle. Deep learning has the advantages of non-contact and high precision, and can accurately identify animal behavior using only image and video data. The existing technology uses EfficientNet for spatial feature extraction, BiFPN for feature fusion, and attention mechanism BiLSTM for temporal aggregation, achieving a recognition accuracy of 97.87% and an average recognition frame rate of 134f / s per second. However, the study also shows that standing and walking behaviors are difficult to distinguish in natural environments, because the behavior of cattle is often a complex mixed state, and the recognition effect is easily disturbed by posture and environment. The two behaviors are very similar visually, which leads to difficulties in recognition. A deep learning-based framework is proposed in the prior art for automatically identifying the behavior of cattle in a real farm environment, including 15 different levels of behavior, such as eating, walking, resting, standing, lying down, etc. The framework combines spatial and temporal information, and can monitor the behavior of cattle in real time and provide behavior positioning. However, the performance of the algorithm will be affected at long distances and under occlusion. Summary of the invention
[0005] The present invention provides a new method for cattle digital twin behavior perception modeling based on the fusion of sensor data and video data, which is used to solve the problems existing in the existing technology of single modality data in cattle behavior recognition and the problem of identifying similar behaviors of sensor data.
[0006] The present invention provides a new cattle digital twin behavior perception modeling system based on the fusion of sensor data and video data, which is used to solve the problems of identity recognition and target tracking difficulties under long-angle viewing and small target conditions using non-contact methods.
[0007] The present invention is achieved through the following technical solutions:
[0008] A new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion, the method comprising the following steps:
[0009] Step 1: Obtain sensor data and video data of cattle;
[0010] Step 2: Building a data set based on the cattle data obtained in step 1, unifying the time length of the cattle sensor data and the cattle video data in step 1 with the same start time;
[0011] Step 3: Perform data fusion based on the sensor data of the cattle after the unified time length in step 2 and the video data of the cattle;
[0012] The step 3 specifically includes the following steps:
[0013] Step 3.1: Extract key frames from video data;
[0014] Step 3.2: crop the image of the cattle in the key frame in step 3.1;
[0015] Step 3.3: Extract features from the cropped cattle images in step 3.2;
[0016] Step 3.4: Extract features from sensor data;
[0017] Step 3.5: Align the sensor features extracted in step 3.4 and the image features of the cattle extracted in step 3.3;
[0018] Step 3.6: Perform multimodal data fusion on the feature aligned data in step 3.5.
[0019] Step 4: Train the data fused in step 3;
[0020] Step 5: Perform indicator evaluation on the data trained in step 4.
[0021] Furthermore, the step 2 is specifically as follows: manually screen out 228 video segments, each behavior is 5000 seconds and 75000 frames, a total of 375000 frames; in these video segments, there are 35 segments of drinking water, 15 segments of eating, 26 segments of lying down, 46 segments of standing, and 105 segments of walking, with an average duration of 109.6 seconds; for sensor data, according to the start time and duration of the video, intercept sensor data segments with the same length as the video; and use data with the same start time and equal duration to construct a data set.
[0022] Furthermore, the step 3.1 is specifically as follows: the frame rate of the video stream is 15fps, and for each second of video, video frames numbered 0, 4, 8, and 12 are intercepted as key frames.
[0023] Furthermore, the step 3.2 is specifically to cut out the area of each image containing the recognition target as a source file for calculating image features.
[0024] Furthermore, the step 3.3 is specifically to use Efficient V2 to train a feature extraction model and use the model to extract features.
[0025] Furthermore, the step 3.4 is specifically to process the sensor data sensor through LSTM, and pass it through the fully connected layer FC2 after standardization;
[0026] Furthermore, the step 3.5 is specifically to align the features of the four relatively even key frame screenshots per second with the sensor features according to the sampling frequency of the sensor of 4 Hz;
[0027] Furthermore, the step 3.6 is specifically as follows: for each image img, a combined feature combined_features is obtained through global average pooling and maximum pooling, all image features of the batch are standardized and passed through the fully connected layer FC1; the image features and sensor features are spliced in the last dimension; the spliced features are input into the Transformer encoder; and finally the output result is obtained through the fully connected layer FC output.
[0028] Furthermore, the step 4 specifically includes using a 2-layer Transformer encoder to complete the behavior classification of cattle after multimodal data fusion, and the expected number of input features is 256;
[0029] Among them, the first 128 are image features, the last 128 are sensor features, the number of multi-head attention heads is 8, the learning rate is 0.001, the optimizer is Adam, the batch size is 32, and the training is 20 rounds;
[0030] Furthermore, step 5 is specifically as follows: Precision, Recall and Accuracy are used as performance evaluation indicators of the model, and Precision and Recall are in the range of [0,1]; wherein Precision represents the proportion of correct samples in the positive prediction results; Recall represents the proportion of correct samples in the positive samples in the test set; Accuracy is the proportion of the number of correct samples to the total number of samples; the calculation method of each indicator is detailed in formulas (1)-(3):
[0031]
[0032] Among them, TP: predicted positive, actual positive; FN: predicted negative, actual positive; TN: predicted negative, actual negative; FP: predicted positive, actual negative.
[0033] A new system for digital twin behavior perception modeling of cattle based on sensor data and video data fusion, the system uses the new method for digital twin behavior perception modeling of cattle based on sensor data and video data fusion as mentioned above, the system includes
[0034] An acquisition module, used to acquire sensor data and video data of cattle;
[0035] The data fusion module is used to establish a data set based on the acquired cattle data, and unify the time length of the cattle sensor data and the cattle video data under the condition that the start time is the same;
[0036] Then, the sensor data of the cattle after the unified time is fused with the video data of the cattle;
[0037] A training module, used to train the fused data;
[0038] The indicator evaluation module is used to perform indicator evaluation on the trained data.
[0039] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above method is implemented.
[0040] A computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0041] The beneficial effects of the present invention are:
[0042] The model used in the present invention is closer to actual production applications, improves the utilization rate of production equipment data, and achieves the purpose of reducing costs and increasing efficiency for ranches.
[0043] The present invention can fully integrate different data sources, provide more comprehensive data analysis, and enhance the model's understanding and classification capabilities.
[0044] The present invention can more comprehensively understand the video content by fusing the video information and the sensor information.
[0045] The present invention can improve the performance of model classification by learning data of different modalities.
[0046] Due to the habits of cattle, when the sensor data cannot distinguish the behavior due to the short-term stillness in a certain behavior, the state of the cattle can be well understood through the posture of the cattle in the video. Therefore, multimodal data improves the comprehensiveness and depth of behavior information processing by integrating video and sensor data, thereby enhancing the understanding and classification capabilities of the model, which plays an important role in solving the complex behavior classification problem of cattle. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a top view schematic diagram of the measurement system of the present invention.
[0048] Figure 2 This is a sample example diagram of the data set of the present invention, where (0) drinking water, (1) eating, (2) lying down, (3) standing, and (4) walking.
[0049] Figure 3 It is a multimodal behavior recognition flow chart of the present invention.
[0050] Figure 4 It is a multimodal data fusion architecture diagram of the present invention.
[0051] Figure 5 It is a schematic diagram of the confusion matrix based on image data behavior classification of the present invention.
[0052] Figure 6 It is a schematic diagram of the confusion matrix based on sensor data behavior classification of the present invention.
[0053] Figure 7 It is a schematic diagram of the confusion matrix for behavior classification based on multimodal data of the present invention.
[0054] Figure 8 It is a comparison chart of classification performance of different sequence length models of the present invention.
[0055] Fig. 9 It is a comparison chart of classification performance of different classification methods of the present invention.
[0056] Fig.10 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0057] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0058] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.
[0059] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.
[0060] The following is attached to this application specification Figure 1-10 , the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.
[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.
[0062] Implementation Method 1
[0063] This embodiment proposes a new method for digital twin behavior perception modeling of cattle based on the fusion of sensor data and video data. It aims to give full play to the advantages of different modal data through multimodal data fusion, make up for the shortcomings of single modal data in the behavior recognition of cattle, and improve the recognition accuracy of visually similar behaviors such as walking and standing behaviors of cattle by combining sensor data. On the other hand, by fusing video data, the recognition problem of similar behaviors of sensor data such as standing and lying is solved, and the multimodal data generated by the existing equipment on the ranch is fully utilized to learn from each other's strengths and weaknesses, effectively generate digital shadows of cattle with rich behavioral information, and lay the foundation for building a large model of cattle behavior data. At the same time, the present invention only wears a homemade collar on the top of the cattle's neck to reduce the stress of the cattle. The identity information of the cattle contained in the collar solves the problem of identity recognition and target tracking difficulties under the condition of small targets at a long viewing angle by non-contact methods, which is achieved through the following technical solutions.
[0064] A multimodal cattle behavior recognition method using sensor data and video data fusion can be extended to application scenarios where different types of sensor data and video data are combined to complete cattle behavior recognition; for example, a 6-axis sensor is converted into a 3-axis sensor, and the present invention can also be applied after fine-tuning.
[0065] The present invention constructs a digital shadow of the cattle rather than a digital twin of the pasture. The cattle are subjected to five behaviors, namely drinking, eating, lying down, standing and walking, so that the five behavior patterns can be accurately judged in the digital twin.
[0066] like Figure 1 As shown, the method comprises the following steps:
[0067] Step 1: Obtain sensor data and video data of cattle;
[0068] Step 2: Building a data set based on the cattle data obtained in step 1, unifying the time length of the cattle sensor data and the cattle video data in step 1 with the same start time;
[0069] Step 3: Perform data fusion based on the sensor data of the cattle after the unified time length in step 2 and the video data of the cattle;
[0070] Step 4: Train the data fused in step 3;
[0071] Step 5: Perform indicator evaluation on the data trained in step 4.
[0072] Further, the step 1 is specifically as follows: there are a total of 6 UWB signal receiving anchor points, which are connected in series through an RS422 bus, one of which is used as a main anchor point and is connected to a serial device server through RS422; the serial server communicates with a ranch server through a twisted pair using a TCP protocol; the collar worn on the neck of the cattle collects sensor data at a frequency of 4 Hz, and the sensor data is acceleration data in three directions and angular velocity data in three directions collected by the collar worn on the neck of the cattle, and the sensor data is transmitted to the main anchor point through a UWB signal every 90 seconds; the main anchor point is sent to the ranch server through the serial server, and finally saved in the server in csv format, wherein each line of data includes packet arrival time, data packet number, acceleration in three directions, and angular velocity data in three directions;
[0073] The video data is that multiple network cameras (CS-C1C, Hikvision, Hangzhou, China) are placed on the steel beam of the cowshed 3m high and 2.6m in front of the feeding area of the cattle; each camera is placed directly in front of a cow to record the behavior of each cow separately, and there is a common field of view between two adjacent cameras to provide videos from different perspectives; each camera is equipped with a 128GB Micro SD memory card, and the researchers will copy the video to the ranch server and clear the SD card data every week; the resolution of the video data is 1920×1080, which can clearly observe the various behaviors of the cattle.
[0074] Further, the step 2 is specifically to observe and record the drinking, eating, lying, standing and walking behaviors of 5 cattle by 3 experienced breeders, and manually screen out 228 video segments, each behavior is 5000 seconds and 75000 frames, a total of 375000 frames; in these video clips, there are 35 segments of drinking, 15 segments of eating, 26 segments of lying, 46 segments of standing, and 105 segments of walking, with an average duration of 109.6 seconds; because of the size limit of the fence where the cattle are located, the walking duration of the cattle is generally short, and because of the habits of the cattle, the duration of standing still is also short; in order to avoid the negative impact of the model caused by the inconsistent total length of different types of videos and ensure the balance of each behavior data, 5000 seconds of video are intercepted for each type of action; for sensor data, according to the start time and duration of the video, the data line of the behavior start time is found in the corresponding sensor file and the sensor data segment with the same length as the video is intercepted; the data set is constructed using data with the same start time and duration;
[0075] Table 1 Behavior video data of cattle used in the present invention
[0076]
[0077] Because the sampling frequency of the sensor is 4Hz, in order to facilitate the alignment of video data and sensor data, video decomposition technology is used to capture 4 frames per second relatively evenly. 20,000 frames are captured for each behavior, totaling 100,000 frames of video screenshots; while capturing the video frames, the image resolution of the frames is adjusted to 640×360; finally, the target detection technology is used to detect the position of the cattle in the video frame and crop it for storage; some pictures of the data set are as follows: Figure 2 As shown in the figure, the videos selected in the dataset follow the following standards:
[0078] (1) The dataset should contain as many of the five behavioral clips of different lights, angles, and different cattle as possible to ensure sample diversity and improve the generalization ability of the model;
[0079] (2) All the behaviors of cattle selected in the present invention are ongoing behavior fragments of cattle, and do not include the beginning and end of the behavior;
[0080] (3) Drinking behavior starts when the cow’s tongue first touches the water surface and ends when the cow’s head leaves the drinking area;
[0081] (4) Walking behavior starts from the first movement of the cow's legs to the last time the cow's hoof touches the ground;
[0082] (5) All behavior segments are captured from the end of the second before the behavior occurs to the beginning of the second after the behavior ends;
[0083] The training set and the test set are divided in a ratio of 9:1, wherein the training set contains 90,000 cropped images of cattle and the test set contains 10,000 images. The two set folders are named train and val respectively and have the same internal structure. Each behavior is numbered and a behavior number folder is established. All data of the same behavior are stored under the behavior number folder. A subdirectory is named after the file name of each video clip. The subdirectory contains cropped images of all cattle in the video clip and the corresponding original CSV file fragment of the sensor of the cattle; screenshots of behavior images of some cattle in the dataset established by the present invention are as follows: Figure 2 As shown;
[0084] like Figure 2 As shown in the figure, each row of data is a key frame screenshot of the same behavior video clip in the data set. The behavior numbered 0 is drinking behavior, the behavior numbered 1 is eating behavior, the behavior numbered 2 is lying behavior, the behavior numbered 3 is standing behavior, and the behavior numbered 4 is walking behavior; the numbers a, b, and c in each column represent different key frame screenshots;
[0085] Furthermore, in order to effectively integrate the two modalities of video and sensor data and achieve better classification results, the processing process is as follows: Figure 3 As shown in the figure, the process can be roughly divided into two stages: feature extraction and feature fusion. The feature extraction stage of sensor data is relatively simple. BiLSTM is used to pre-train the model to find the model with the best recognition effect, and then the pre-trained model is used to extract the feature vector of sensor data. Relatively speaking, the feature extraction stage of video data requires relatively more work:
[0086] (1) Using the OpenCV library, in a video with a frame rate of 15 FPS, save the frames numbered 0, 4, 8, and 12 per second in the video as the sample frames of the dataset. At the same time, in order to reduce the amount of convolution calculation, the length and width of the sample frame are scaled to 1 / 3 of the original, that is, 640×360 pixels;
[0087] (2) Using YoloV7 Tiny, 3,000 labeled images of 5 cows were used to train a target detection model. The detection model was used to detect the sample frames generated in step (1). Folders were named with the cow numbers from c1 to c5. The cows in each image were cropped and placed in folders named with the cow numbers.
[0088] (3) Use the EfficientNet V2 S network to classify the images of cattle cropped in step (2) according to five behaviors to obtain a pre-trained model, and use the model to extract the characteristic behavior tensor of the cattle image;
[0089] When the image feature tensor of the video data and the feature vector of the sensor data are prepared, the two features are fused and sent to the Transformer Encoder, and the output is classified to obtain the final classification result;
[0090] The step 3 specifically includes the following steps:
[0091] Step 3.1: Extract key frames from video data;
[0092] Step 3.2: crop the image of the cattle in the key frame in step 3.1;
[0093] Step 3.3: Extract features from the cropped cattle images in step 3.2;
[0094] Step 3.4: Extract features from sensor data;
[0095] Step 3.5: Align the sensor features extracted in step 3.4 and the image features of the cattle extracted in step 3.3;
[0096] Step 3.6: Perform multimodal data fusion on the feature aligned data in step 3.5.
[0097] Furthermore, the step 3.1 is specifically as follows: the frame rate of the video stream is 15fps, and for each second of video, the video frames numbered 0, 4, 8, and 12 are intercepted as key frames;
[0098] Specifically, in order to effectively extract the movement information of cattle contained in the video, balance the training cost, and facilitate modal alignment with sensor data, it is necessary to extract the key frames in the video; the present invention uses the OpenCV library to read the video and save the frames, and by traversing each frame of the video, retains the video frames with specified numbers as needed, skips 4 frames for each interception of 1 frame for the first three times, and skips 3 frames after intercepting 1 frame for the fourth time, so that 4 images can be saved per second in a relatively uniform manner; experiments have shown that under the condition of stable frame rate, this method can save 4 frames of images per second of video; through this method, 100,000 frames of key frames of 5 behaviors of the experimental object are finally extracted, and 20,000 frames for each behavior.
[0099] The step 3.2 specifically includes cutting out the region of each image containing the recognition target as a source file for calculating the image features;
[0100] Specifically, in order to be able to capture the area of interest in the video frame and identify the identity of the cattle, while ensuring that the proportion of cattle in the image is greater than 80%, the present invention uses the Yolo V7 network based on PyTorch as the backbone network for training, and uses the trained model to capture the video frame; based on PASCAL VOC The 2007 dataset standard created a dataset containing all test objects, which included 3,000 images, 5 categories, 7,514 targets, and a training set to test set ratio of 9:1. Before starting training, the general pre-trained weights were loaded to avoid the backbone network parameters being too random. The model was trained for a total of 300 cycles, and the best performing model appeared at the end of the 290th training cycle. At this time, the loss rate of the training set was 0.023, and the loss rate of the validation set was 0.022. When detecting targets in the validation set, the average accuracy of all categories (mAP: the average weighted average of the average accuracy of all categories of detection) reached 98.60% when the intersection over union (IoU: the overlap between the predicted box and the original labeled box) was set to 0.5. Through this model, the cows in the key frames were detected and the images were cropped, and finally the image data of the cow's behavior was obtained.
[0101] Specifically, step 3.3 includes training a feature extraction model using Efficient V2 and extracting features using the model;
[0102] Specifically, in order to unify the dimensions of motion image data and sensor data and improve the training efficiency of the fusion model, the present invention uses EfficientNet V2 as the pre-training network of image features to pre-train and extract features from the motion image data of cattle; combined with the complexity of this task and the characteristics of the image, EfficientNet V2 S is selected as the backbone network and pre-trained on the data set; for each behavior of 20,000 images, the training set and the validation set are divided into a ratio of 9:1;
[0103] When the present invention uses the EfficientNet V2 S backbone network for pre-training according to 5 categories, the key frame screenshots of the cattle are adjusted to 256×256 size, the batch size is 256, and the input size is 100; in order to make the model generalize better, the image is horizontally flipped with a probability of 50%, and the pre-training is performed for 300 rounds. The model with the highest verification accuracy on the verification set appears in the 117th round, and the overall recognition accuracy of the five behaviors is 89.46%; using the model, the images in the data set are feature extracted to obtain the image feature tensor before the fully connected layer, whose shape is (256,8,8);
[0104] Specifically, step 3.4 processes the sensor data sensor through BiLSTM, and passes it through the fully connected layer FC2 after standardization;
[0105] Specifically, a bidirectional recurrent neural network (BRNN) can use past and future information for prediction at the same time without presetting delay parameters; by training the network in two time directions at the same time, BRNN can better utilize available input information, thereby achieving better results in practical applications; BiLSTM is an extension and application of this concept, applying LSTM units to bidirectional RNN, thereby enhancing the model's ability to process long-term dependencies and contextual information; the sensor data acquired by the present invention contains acceleration in three directions and angular velocity information in three directions, including the acceleration and angular velocity changes of different actions of the experimental object in three-dimensional space, and has a time series nature, so using BiLSTM is very suitable for classification; at the time point corresponding to the key frame, the present invention extracts 100,000 corresponding sensor data, and also forms a training set and a verification set in a ratio of 9:1; in the data preprocessing stage, StandardScaler is used to standardize all data;
[0106] After several trial trainings, we finally selected BiLSTM with 7 hidden layers, 128 nodes in each hidden layer, a batch size of 32, a sequence length of 16, and the input data is acceleration and angular velocity in three directions; in order to make the model have better generalization ability and avoid overfitting, the dropout ratio is 0.5, the L2 regularization coefficient weight decay is 0.0002, the optimizer is Adam, and the data of each time step is classified; after 200 rounds of training, the model with the highest accuracy on the validation set appeared in the 49th round, and the average recognition accuracy of the five behaviors was 96.83%; using this model, the sensor data in the dataset was feature extracted to obtain the sensor feature vectors before the fully connected layer, and the length of each feature vector was 256.
[0107] Specifically, step 3.5 is to align the features of four relatively even key frame screenshots per second with the sensor features according to the sampling frequency of the sensor of 4 Hz;
[0108] Specifically, the video data of the present invention has 15 frames per second, and the sensor data has 4 data per second. The data alignment of the two modes is an important problem to be solved by the present invention. On the time scale, when extracting the video key frames, only the video frames numbered 0, 4, 8, and 12 in the 15 frames per second are retained as key frames, and the sampling frequency of the video frames is adjusted to 4Hz, the same as the sensor. In terms of data feature alignment, the feature tensor shape of the key frame screenshot data is (256, 8, 8). In order to retain the global features in the feature tensor, global average pooling is applied to obtain a feature vector with a length of 256. In order to retain the significant features in the feature tensor, maximum pooling is applied to obtain a feature vector with a length of 256. After connection, it is standardized and converted into a feature vector with a length of 128 using a fully connected layer. For the sensor feature vector with a length of 256, it is first standardized, and then a fully connected layer is defined to convert its length into a feature vector of 128, thereby completing the feature alignment of the video data and the sensor data.
[0109] Specifically, step 3.6 includes: for each image img, a combined feature combined_features is obtained by global average pooling and maximum pooling; all image features of the batch are standardized and passed through the fully connected layer FC1; the image features and sensor features are concatenated in the last dimension; the concatenated features are input into the Transformer encoder; and finally the output result is obtained through the fully connected layer FC output;
[0110] Specifically, the input of the Transformer can include one or more token sequences and the attributes of each sequence, which enables it to complete the multimodal learning task without modifying the architecture; therefore, the present invention uses the Transformer to complete the deep modal fusion work, and the multimodal data fusion method is as follows Figure 4 As shown;
[0111] First, before the multimodal data feature fusion, the single-modal sequence data is feature extracted. On the one hand, the pre-trained Efficient V2 S model is used to extract features from each frame in the cropped key frame sequence to obtain the feature tensor before the fully connected layer, and then global average pooling and maximum pooling are performed on it. The pooling results are connected and stacked into the feature tensor of the key frame screenshot sequence, and then standardized and finally sent to the fully connected layer to obtain the key frame cropped image features; on the other hand, the pre-trained BiLSTM model is used to extract features from the sensor data, first standardize the features, and then send the features to the fully connected layer to transform the shape to obtain the sensor data features; finally, the image features with the same shape are spliced with the sensor features and used as the input of the Transformer Encoder, thereby completing the deep feature fusion of feature extraction of different modalities;
[0112] Furthermore, the step 4 specifically includes using a 2-layer Transformer encoder to complete the behavior classification of cattle after multimodal data fusion, and the expected number of input features is 256;
[0113] Among them, the first 128 are image features, the last 128 are sensor features, the number of multi-head attention heads is 8, the learning rate is 0.001, the optimizer is Adam, the batch size is 32, and the training is 20 rounds; the model with the highest accuracy appears in the first round, and the average accuracy of the five behaviors is 98.8%;
[0114] Furthermore, step 5 is specifically as follows: in order to evaluate the recognition performance of the algorithm of the present invention for different behaviors, confusion matrix, precision, recall and accuracy are used as performance evaluation indicators of the model; wherein, confusion matrix can better understand the errors in classification; precision, recall and accuracy are used as performance evaluation indicators of the model, and precision and recall are in the range of [0,1]; wherein, precision represents the proportion of correct samples in positive prediction results; recall represents the proportion of correct samples in positive samples in the test set; accuracy is the proportion of correct samples to the total number of samples; the calculation method of each indicator is detailed in formula (1)-(3):
[0115]
[0116] Among them, TP (True Positive): predicted to be positive, the actual result is positive; FN (False Negative): predicted to be negative, the actual result is positive; TN (True Negative): predicted to be negative, the actual result is negative; FP (False Positive): predicted to be positive, the actual result is negative.
[0117] Implementation Method 2
[0118] According to the method of implementation mode 1, an experiment was conducted on 5 cows. Specifically, the sensors and video data used in the present invention are multimodal behavioral data of cows collected at the Acheng Experimental Base of Northeast Agricultural University from May 27 to June 13, 2022. The experimental subjects are five healthy Holstein non-lactating cows, aged 3±0.5 years old and weighing 410±40kg. The experimental process complies with the relevant provisions of the Regulations on the Administration of Laboratory Animals of the People's Republic of China.
[0119] The top view of the multimodal data acquisition system is as follows: Figure 1 As shown: Among them, number 1 is the camera, 2 is the RS422 bus, 3 is the anchor point, 4 is the collar, 5 is the serial device server, 6 is the twisted pair cable, and 7 is the ranch server. The system can collect collar sensor data and video data of 5 experimental cows at the same time.
[0120] This study collected 100,000 video key frames to form a data set, 20,000 key frames for each behavior, and divided the training set and validation set in a 9:1 manner. During the validation process, the length of each video key frame sequence is 16, that is, the time width is 4 seconds. The remaining key frames and sensor data that are less than 16 will not be loaded. The Accuracy, Precision, and Recall of the five behaviors are shown in Table 2. The average performance of the model on the validation set is as follows: the accuracy is 98.80%, the precision is 97.15%, and the recall is 96.93%. The results show that the Transformer-based multimodal behavior recognition network proposed in this invention can effectively identify the five basic movement behaviors of cattle.
[0121] Table 2 Prediction results based on Transformer multimodal network
[0122]
[0123] As shown in Table 2, the performance of the cattle behavior model based on multimodal data varies in different behaviors. The classification results of eating behavior are the best, with all accuracy, precision and recall reaching 100%, which shows the high accuracy and reliability of the classification model in detecting eating behavior. In contrast, although drinking behavior shows extremely high accuracy (100%), the recall rate is only 87.93%, indicating that the model has some missed detection. For lying and walking behaviors, the model can fully identify all actual behaviors (recall rate is 100%), but the accuracy is slightly lower, 93.85% and 96.72% respectively, indicating that there are a few false positives. The classification of standing behavior also shows high accuracy (98.33%), precision (95.16%) and recall (96.72%), which is slightly lower than other behaviors. Overall, the cattle behavior classification model based on multimodal data performs well, with an average accuracy of nearly 99%, and precision and recall rates close to 97%, reflecting its high efficiency and reliability in practical applications.
[0124] Single-mode result analysis
[0125] Image-based behavior classification results
[0126] This study first uses the features extracted by Efficient V2 S and uses a 2-layer Transformer encoder for training. The expected number of input features is 128, the number of multi-head attention heads is 8, the learning rate is 0.001, the optimizer is Adam, the batch size is 32, and the training is 20 rounds. The highest accuracy occurs in the 17th round, with an average accuracy of 94.93% for the five categories, a precision of 87.39%, and a recall of 87.23%. The confusion matrix obtained on the validation set is shown in Figure 5.
[0127] The confusion matrix shows that the recognition effect of walking and standing behavior is poor. The reason for this phenomenon is that walking and standing are very similar in posture. In addition, the experimental subjects raised their heads to rest while drinking water, which caused the drinking behavior to be misidentified as lying down and walking. Walking behavior is misidentified as other behaviors because the posture of walking behavior is similar to other behaviors. Therefore, if the behavior of cattle is only identified through video data, some behaviors are difficult to distinguish.
[0128] Sensor-based behavioral outcomes
[0129] This study uses a BiLSTM with 7 hidden layers and 128 nodes in each layer. To prevent the model from overfitting, the dropout ratio is 0.5, the L2 regularization coefficient weight_decay is 0.0002, the input data is the acceleration and angular velocity in the three directions of xyz, the read sequence length is 16, the batch size is 32, and the training is 200 rounds. The highest accuracy appears in the 17th round. The average accuracy of the five categories is 96.83%, the precision is 92.02%, and the recall rate is 92.07%. The confusion matrix obtained on the validation set is shown in Figure 5.
[0130] Through the confusion matrix, we can find that the true value is lying down, but it is mistakenly identified as walking. This misjudgment occurs because cows occasionally shake their heads or raise their heads when lying down, which is similar to the neck movement of cows when walking. Lying down is mistakenly identified as standing up because the neck is still when lying down and standing up. At the same time, there are cases where standing behavior is mistakenly identified as lying down or walking, and walking behavior is also mistakenly identified as drinking, eating, lying down, and standing. This is because the neck movements of these behaviors are similar to a certain extent, making it difficult to distinguish them based on sensor data alone.
[0131] Behavior classification results based on multimodal data
[0132] This study fuses video features with sensor features and uses a 2-layer Transformer encoder for training. The expected number of input features is 128, the number of multi-head attention heads is 8, the learning rate is 0.001, the optimizer is Adam, the batch size is 32, and the training is 20 rounds. The highest accuracy appears in the first round, and the confusion matrix obtained on the validation set is shown in Figure 7.
[0133] Through the confusion matrix, it can be found that the recognition effect of standing and walking is significantly better than the classification result using only video data. The introduction of sensor data makes up for the lack of recognition of visually similar behaviors and better distinguishes the five behaviors, especially the distinction between standing and walking behaviors. Although the recall rate of drinking water behavior has not been improved, the model performance is more balanced. On the other hand, by integrating video data features, the classification effect of the three behaviors of lying down, standing and walking has been greatly improved, especially the recall rate of walking behavior has been greatly improved, from 80.83% to 100%. It can be seen that the behavior classification method based on multimodal data can make up for the shortcomings of using different modal data alone, and its classification results are significantly better than single-modal data behavior classification.
[0134] Comparison of results with different data sequence lengths
[0135] In order to select the appropriate sequence length and achieve better behavior recognition results, this study selected five cases with sequence lengths of 8, 12, 16, 20 and 24 as comparative experiments. Under the condition that the various conditions of the experiment remain unchanged, in the feature extraction stage, different sequence lengths are used to extract the features of sensor data using BiLSTM. In the feature fusion stage, different sequence lengths are also used to fuse the key frame screenshot feature tensor and sensor feature vector. Finally, the model with the best performance on the validation set is obtained. The model classification performance line chart under different sequence length conditions is shown in the figure below. Figure 8 shown.
[0136] like Figure 8 As shown in the figure, the accuracy increases with the increase of sequence length, reaching the maximum value when the sequence length is 16. The accuracy will decrease slightly with the further increase of sequence length, but it can still maintain a high level. The precision and recall are stable at around 94% when the sequence length is 8 to 12. When the sequence length increases to 16, the value increases significantly, close to 97%. When it continues to increase to 20 and 24, the value decreases but is still higher than the initial level of about 96%. When the sequence length is short (8 to 12), the performance of the model is relatively stable but relatively low, especially in terms of recall; when the sequence length is long (20 to 24), although the performance of the model decreases slightly, it still remains at a high level, indicating that the generalization ability of the model is good at these lengths, but it has not reached the optimal state. Therefore, the model performs best when the sequence length is 16, with the best accuracy, precision and recall, indicating that at this length, the model can better capture the features in the data and perform accurate classification, while achieving the best balance between performance and computing resources.
[0137] Comparison with other methods
[0138] In order to verify the effectiveness of the model constructed by multimodal data in classifying cattle behavior, this study used three different single-modal behavior classification methods, trained and verified the models on the data set of this study, and evaluated the effect of the multimodal fusion method. The three methods are: using 3D convolutional neural network to classify video data, applying EfficientNet V2 S to extract features, using Transformer Encoder for classification, and directly applying BiLSTM to classify sensor data. The average classification accuracy, precision, and recall of various methods are shown in Table 3:
[0139] Table 3 Comparison of classification results of different methods
[0140]
[0141] As shown in Table 3, the 3D CNN based on video data makes most correct predictions when predicting positive examples. However, its recall rate is low, indicating that the model is not effective in detecting all actual positive examples. The combination of EfficientNet V2 S and Transformer has a slightly higher accuracy than 3D CNN, and the recall rate is also improved, indicating that this method is more effective in identifying all actual positive examples. BiLSTM based on sensor data has significant improvements in all indicators, especially in accuracy and recall, indicating that it outperforms the previous two methods in overall performance. The multimodal fusion method performs best in all performance indicators, especially precision and accuracy, indicating that it performs well in predicting positive examples and overall classification correctness. The high recall rate also shows that it is very effective in detecting all actual positive examples. The performance comparison of the above four classification methods is as follows: Fig. 9 As shown:
[0142] In comparison, although BiLSTM performs well in all indicators, it is slightly inferior to the multimodal fusion method. Although the performance of EfficientNet V2 S&Transformer and 3D CNN is also good, they are somewhat insufficient in accuracy and recall. Overall, the multimodal method that integrates video and sensor data has significant advantages in dealing with complex behavior analysis tasks.
[0143] The present invention uses a multimodal data-based deep learning algorithm to realize the recognition of cattle's drinking, eating, lying, standing and walking behaviors. Compared with the single-modal data behavior recognition method, it has the characteristics of relatively small stress, not limited by long-angle viewing angle and occlusion, and complementary advantages of different modal data. The results obtained in this study show that when the sequence length is 16, the model has a precision rate of 97.67% and a recall rate of 87.93% for drinking behavior; the recognition precision rate of 100.00% and the recall rate of 100.00% for eating behavior; the recognition precision rate of 98.67% and the recall rate of 100.00% for lying behavior; the recognition precision rate of 98.33% and the recall rate of 96.72% for standing behavior; the recognition precision rate of 99.33% and the recall rate of 100.00% for walking behavior. This method can make full use of the multimodal data collected by existing equipment, generate digital shadows of cattle with rich behavioral information, improve the model and simulation layers in the digital twin architecture of cattle, and lay the foundation for building a large behavioral data model of cattle.
[0144] Implementation Method 3
[0145] This embodiment provides a new cattle digital twin behavior perception modeling system based on sensor data and video data fusion, the system uses the new cattle digital twin behavior perception modeling method based on sensor data and video data fusion as described in embodiment 1, the system includes:
[0146] An acquisition module, used to acquire sensor data and video data of cattle;
[0147] The data fusion module is used to establish a data set based on the acquired cattle data, and unify the time length of the cattle sensor data and the cattle video data under the condition that the start time is the same;
[0148] Then, the sensor data of the cattle after the unified time is fused with the video data of the cattle;
[0149] A training module, used to train the fused data;
[0150] The indicator evaluation module is used to perform indicator evaluation on the trained data.
[0151] Implementation Method 4
[0152] An embodiment of the present invention provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory is used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory and the processor are connected via a bus. Specifically, the processor implements any step in the first embodiment above by running the computer program stored in the memory.
[0153] It should be understood that in the embodiments of the present invention, the processor referred to may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0154] The memory may include a read-only memory, a flash memory, and a random access memory, and provides instructions and data to the processor. A part or all of the memory may also include a nonvolatile random access memory.
[0155] It should be understood that if the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned implementation method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method implementations when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.
[0156] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined in the present invention may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in the present invention, but rather to the widest range consistent with the principles and novel features disclosed in the present invention.
[0157] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the implementation method can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method implementation method, which will not be repeated here.
[0158] It should be noted that the methods and detailed examples provided in the above embodiments can be combined with the devices and equipment provided in the embodiments, and references can be made to each other, and no further details will be given.
[0159] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0160] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal equipment and method can be implemented in other ways. For example, the device / equipment implementation described above is only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0161] The above-mentioned embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above-mentioned embodiments, or replace some of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion, characterized in that: The method comprises the following steps, Step 1: Obtain sensor data and video data of cattle; Step 2: Building a data set based on the cattle data obtained in step 1, unifying the time length of the cattle sensor data and the cattle video data in step 1 with the same start time; Step 3: Perform data fusion based on the sensor data of the cattle after the unified time length in step 2 and the video data of the cattle; The step 3 specifically includes the following steps: Step 3.1: Extract key frames from the video data of cattle; Step 3.2: crop the image of the cattle in the key frame in step 3.1; Step 3.3: Extract features from the cropped cattle images in step 3.2; Step 3.4: Extract features from the cattle sensor data; Step 3.5: Align the sensor features extracted in step 3.4 and the image features of the cattle extracted in step 3.3; Step 3.6: Perform multimodal data fusion on the feature aligned data in step 3.5; Step 4: Train the data fused in step 3; Step 5: Perform indicator evaluation on the data trained in step 4.
2. The new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion according to claim 1 is characterized in that: Specifically, step 2 includes manually selecting 228 video segments, each behavior is 5000 seconds and 75000 frames, for a total of 375000 frames; in these video segments, there are 35 segments of drinking water, 15 segments of eating, 26 segments of lying down, 46 segments of standing, and 105 segments of walking, with an average duration of 109.6 seconds; for sensor data, according to the start time and duration of the video, sensor data segments with the same length as the video are intercepted; and data sets are constructed using data with the same start time and the same duration.
3. The new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion according to claim 1 is characterized in that: Specifically, step 3.1 includes: the frame rate of the video stream is 15fps, and for each second of video, the video frames numbered 0, 4, 8, and 12 are intercepted as key frames; Specifically, the step 3.2 is to cut out the area of each image containing the recognition target as a source file for calculating image features.
4. The new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion according to claim 3 is characterized in that: Specifically, step 3.3 includes training a feature extraction model using Efficient V2 and extracting features using the model; Specifically, step 3.4 processes the sensor data sensor through BiLSTM, and passes it through the fully connected layer FC2 after standardization.
5. The new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion according to claim 4 is characterized in that: Specifically, step 3.5 is to align the features of four relatively even key frame screenshots per second with the sensor features according to the sampling frequency of the sensor being 4 Hz; Specifically, step 3.6 is as follows: for each image img, a combined feature combined_features is obtained through global average pooling and maximum pooling; all image features of the batch are standardized and passed through the fully connected layer FC1; the image features and sensor features are spliced in the last dimension; the spliced features are input into the Transformer encoder; and finally the output result is obtained through the fully connected layer FC output.
6. The new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion according to claim 4 is characterized in that: Specifically, step 4 uses a 2-layer Transformer encoder to complete the behavior classification of cattle after multimodal data fusion, and the expected number of input features is 256; Among them, the first 128 are image features, the last 128 are sensor features, the number of multi-head attention heads is 8, the learning rate is 0.001, the optimizer is Adam, the batch size is 32, and the training is 20 rounds.
7. The new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion according to claim 1 is characterized in that: Specifically, step 5 uses Precision, Recall and Accuracy as the performance evaluation indicators of the model, and Precision and Recall are in the range of [0,1]. Precision represents the proportion of correct samples in the positive prediction results; Recall represents the proportion of correct samples in the positive samples in the test set; Accuracy is the proportion of the number of correct samples to the total number of samples. The calculation method of each indicator is detailed in formulas (1)-(3): Among them, TP: predicted positive, actual positive; FN: predicted negative, actual positive; TN: predicted negative, actual negative; FP: predicted positive, actual negative.
8. A new cattle digital twin behavior perception modeling system based on sensor data and video data fusion, characterized in that: The system uses the new method for cattle digital twin behavior perception modeling based on sensor data and video data fusion as described in any one of claims 1 to 7, and the system includes: An acquisition module, used to acquire sensor data and video data of cattle; The data fusion module is used to establish a data set based on the acquired cattle data, and unify the time length of the cattle sensor data and the cattle video data under the condition that the start time is the same; Then, the sensor data of the cattle after the unified time is fused with the video data of the cattle; A training module is used to train the fused data; The indicator evaluation module is used to perform indicator evaluation on the trained data.
9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Data integration method and system applied to digital twin smart villages
CN117272247A
Dairy cow behavior identification digital twin system based on ultra wide band and inertial measurement unit and digital twin method of dairy cow behavior identification digital twin system
CN117322358A
Fish tank online monitoring method and system based on digital twinning
CN118470622A
Method and arrangement for identifying animals
US20200367470A1
Cited By
Multi-mode beef cattle behavior identification method and system
CN120220249A
A multimodal beef cattle behavior recognition method and system
CN120220249B
