Pig abnormality detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the application discloses a pig abnormality detection method and device, electronic equipment and storage medium, the method comprises the following steps: collecting a plurality of audio and video data of pigs; extracting features from each audio and video data to obtain corresponding target features; inputting the target features of each audio and video data in the plurality of audio and video data into a multi-instance learning model to train the multi-instance learning model; and detecting the abnormal behavior of the pigs by using the trained multi-instance learning model.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and to, but is not limited to, a method, apparatus, electronic device, and storage medium for detecting anomalies in pigs. Background Technology

[0002] In related technologies, audio detection and image detection methods are mainly used to detect abnormalities in pigs. Audio detection methods involve collecting standard signals of abnormal pig sounds, extracting feature parameters from these signals, establishing a sound recognition model based on these parameters, and then monitoring the sounds emitted by the pigs in real time. The sound recognition model then determines whether the sounds are abnormal (such as coughing or biting sounds), thus indicating whether the pigs are exhibiting abnormal behavior. However, many abnormal pig behaviors do not contain detectable audio data, and the audio responses to various abnormalities, stress, and pain in pigs are often very similar.

[0003] In image detection methods, environmental information is labeled, the backs of fattening pigs are marked, and then computer vision technology is used to detect the target. Finally, an SVM (Support Vector Machine) is used to learn the relationship between the detected pig behavior and environmental information. However, abnormal pig behavior is often dynamic and always occurs over time; a single image feature is insufficient to define abnormal pig behavior. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for detecting abnormalities in pigs.

[0005] In a first aspect, embodiments of this application provide a method for detecting anomalies in pigs. The method includes: collecting multiple audio and video data of pigs; extracting features from each audio and video data to obtain corresponding target features; inputting the target features of each audio and video data into a multi-instance learning model to train the multi-instance learning model; and using the trained multi-instance learning model to detect abnormal behavior of the pigs.

[0006] Secondly, embodiments of this application provide an anomaly detection device for pigs, comprising: a collection module for collecting multiple audio and video data of pigs; an extraction module for extracting features from each audio and video data to obtain corresponding target features; a training module for inputting the target features of each audio and video data into a multi-instance learning model to train the multi-instance learning model; and a detection module for detecting abnormal behavior of the pigs using the trained multi-instance learning model.

[0007] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the steps in any of the abnormal detection methods for pigs described in the embodiments of this application.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the abnormal detection methods for pigs described in embodiments of this application.

[0009] In this embodiment, by extracting features from audio and video data to obtain target features, and inputting the target features into a multi-instance learning model, the learning ability of the multi-instance learning model is enhanced, making the anomaly classification ability of the multi-instance learning model stronger and improving the accuracy of classifying abnormal and non-abnormal audio and video data. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an abnormality detection method for pigs according to an embodiment of this application;

[0011] Figure 2 This is a schematic diagram illustrating the acquisition of audio and video data according to an embodiment of this application;

[0012] Figure 3 This is a schematic diagram illustrating a method for video feature extraction using a convolutional neural network according to an embodiment of this application;

[0013] Figure 4 This is a schematic diagram illustrating an audio feature extraction method according to an embodiment of this application;

[0014] Figure 5 This is a schematic diagram illustrating the fusion of audio and video features according to an embodiment of this application;

[0015] Figure 6 This is a schematic diagram of training inference using a Multiple Instant Learning Network (MIL) according to an embodiment of this application;

[0016] Figure 7 This is a schematic diagram illustrating a data augmentation method according to an embodiment of this application;

[0017] Figure 8 This is a schematic diagram of a process for detecting and issuing early warnings for abnormal pig conditions, as described in an embodiment of this application.

[0018] Figure 9 This is a schematic diagram of the structural composition of an abnormality detection device for pigs according to an embodiment of this application;

[0019] Figure 10 This is a schematic diagram of a hardware entity of an electronic device according to an embodiment of this application. Detailed Implementation

[0020] The technical solution of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0021] Figure 1 This is a flowchart illustrating an abnormality detection method for pigs according to an embodiment of this application. Figure 1 As shown, the method includes:

[0022] Step 102: Collect multiple audio and video data from the pigs;

[0023] Since abnormalities in pigs often exhibit a dual-state behavior and sound, such as fighting often accompanied by biting sounds, and groaning due to illness often accompanied by lethargy, multiple audio-visual data can be generated based on the acquired pig behavior and sound. Audio-visual data can also be called sample data packets. Each audio-visual data can include at least one set of video data and corresponding audio data. The video data can be called behavioral data, and the audio data can be called sound data.

[0024] Step 104: Extract features from each of the audio and video data to obtain the corresponding target features;

[0025] Among them, audio features can be extracted from the sound frequency of pigs, and video features can be extracted from the behavior of pigs. Through feature extraction, key audio and video features can be preserved. By processing the audio and video features, the corresponding target features can be obtained.

[0026] Step 106: Input the target features of each of the multiple audio and video data into the multi-instance learning model to train the multi-instance learning model;

[0027] Here, model training can also be called model update, and the Multiple Instance Learning (MIL) model can also be called a Multiple Instance Learning network. The MIL model can learn from a data packet composed of multiple instances, which may include audio and video data. The instances may include audio features, image features, and video features, etc. In this embodiment, the audio and video data may include audio features and video features. In other embodiments, the audio and video data may also include two different video features, and the audio and video data may further include two different audio features. The audio and video data may also include audio features and image features. The audio and video data may further include a first audio feature, a first video feature, a second audio feature, and a second video feature, wherein the first audio feature and the first video feature correspond to each other, and the second audio feature and the second video feature correspond to each other.

[0028] Step 108: Use the trained multi-instance learning model to detect abnormal behavior in the pigs.

[0029] In this embodiment, by extracting features from audio and video data to obtain target features, and inputting the target features into a multi-instance learning model, the learning ability of the multi-instance learning model is enhanced, making the anomaly classification ability of the multi-instance learning model stronger and improving the accuracy of classifying abnormal and non-abnormal audio and video data.

[0030] This application also provides a method for detecting abnormalities in pigs, the method comprising:

[0031] Step S202: Collect multiple audio and video data from the pigs;

[0032] Each of the aforementioned audio and video data includes audio data and video data; Figure 2 This is a schematic diagram illustrating audio and video data acquisition according to an embodiment of this application. See also... Figure 2 The image acquisition device 201 can be used to collect video data 202 of the pigs, and the audio acquisition device 203 can be used to collect audio data 204 of the pigs. The image acquisition device 201 can be a camera, camcorder, camera, or other device with shooting function (such as a mobile phone, tablet computer, etc.), and the audio acquisition device 203 can be a microphone, pickup, walkie-talkie, or other device with recording function (such as a mobile phone, tablet computer, etc.).

[0033] Step S204: Using a three-dimensional convolutional neural network, feature extraction is performed on the video data in each of the audio and video data to obtain the video features of the corresponding audio and video data;

[0034] Figure 3This is a schematic diagram illustrating a method for video feature extraction using a convolutional neural network according to an embodiment of this application. See also... Figure 3 A 2D (2-dimensional) convolutional neural network 301 can be used for feature extraction from a single image, a 2D multi-frame convolutional neural network 302 can be used for feature extraction from multiple images, and a 3D (3-dimensional) convolutional neural network 303 can be used for feature extraction from images and time series data, i.e., video stream feature extraction. Since the data acquisition stage collects not only video stream data (also called video data) but also audio stream data (also called audio data), and audio data differs from image frames which are distributed in two-dimensional space, audio data is strongly correlated with time series data and belongs to a time series distribution. Therefore, it cannot be directly concatenated with the image stream dimension and cannot be fused with the image stream. Thus, a 3D convolutional neural network can be used to extract features from video data so that the extracted video features can be fused with audio features.

[0035] See Figure 3 k*k can represent the size of the convolution kernel of the convolutional neural network, L can represent the number of images in the video stream, H*W can represent the size of the image, and d can represent the number of images in the selected video segment.

[0036] Step S206: Perform cepstral analysis on the audio data in each of the audio and video data to obtain the audio features of the corresponding audio and video data;

[0037] It should be noted that not only can feature extraction be performed on video data in audio and video data to obtain the corresponding video features, but also feature extraction can be performed on audio data in audio and video data to obtain the corresponding audio features.

[0038] Step S208: Fuse the audio features and video features of each audio and video data to obtain the corresponding target features;

[0039] Step S210: Input the target features of each of the multiple audio and video data into the multi-instance learning model to train the multi-instance learning model;

[0040] Step S212: Use the trained multi-instance learning model to detect abnormal behavior of the pigs.

[0041] In this embodiment, a three-dimensional convolutional neural network is used to extract features from video data, and cepstral analysis is used to extract features from audio data. This enables more accurate extraction of key video features from video data and key audio features from audio data. The key video features and key audio features are then fused to obtain the fused target features. This enhances the learning ability of the multi-instance learning network, making it more capable of anomaly segmentation and improving the accuracy of classifying abnormal and non-abnormal audio and video data.

[0042] This application also provides a method for detecting abnormalities in pigs, the method comprising:

[0043] Step S302: Collect multiple audio and video data from the pigs;

[0044] Step S304: Using a three-dimensional convolutional neural network, feature extraction is performed on the video data in each of the audio and video data to obtain the video features of the corresponding audio and video data;

[0045] Step S306: Perform Fourier transform, logarithmic transformation, and inverse Fourier transform on the audio data of each audio / video data to obtain the envelope function of the corresponding audio / video data;

[0046] Figure 4 This is a schematic diagram of an audio feature extraction embodiment of this application. See also: Figure 4 The time-domain information of the acquired audio (such as the spectrogram 401) can be converted into frequency-domain information using Fourier transform to obtain the audio spectrum X[k] in the audio spectrum diagram 402. Furthermore, the frequency-domain signal X[k] can be decomposed into the product of two parts: the envelope of the audio spectrum and the details of the audio spectrum. The resulting frequency-domain signal X[k] can be expressed by formula (1):

[0047] X[k]=H[k]E[k] (1);

[0048] Secondly, we can take the logarithm of both sides of formula (1), that is, perform a log operation. The logarithmic transformation process can be represented by formula (2):

[0049] log||X[k]||=log||H[k]||+log||E[k]|| (2);

[0050] Where ||X[k]|| represents the norm of X[k], ||H[k]|| represents the norm of H[k], and ||E[k]|| represents the norm of E[k].

[0051] Furthermore, the inverse Fourier transform can be taken on both sides of formula (2) to obtain the cepstrum x[k]. The inverse Fourier transform process can be represented by formula (3):

[0052] x[k]=h[k]+e[k] (3);

[0053] In formula (3), h[k] describes the low-frequency part of the cepstral x[k], which is also the core feature of audio data. In this embodiment, h[k] can be named envelope function diagram 403 and e[k] can be named spectrum detail diagram 404.

[0054] Step S308: Transpose the envelope function of each audio / video data to obtain the audio features of the corresponding audio / video data;

[0055] In order to match the image convolution term for joint training, the temporal data h[k] needs to be converted into one-dimensional data U (i.e., envelope transpose block 405). The envelope function can be transposed to obtain the audio features (i.e., envelope transpose block 405). The transpose transformation process can be represented by formula (4):

[0056] U = h[k] T (4);

[0057] Since the audio data and image data are acquired synchronously and both are based on time, the dimension of their envelope transpose block can be (1,L) or (1,1,L).

[0058] Step S310: Update the convolution kernel of the three-dimensional convolutional neural network using the audio features of each audio and video data to obtain the updated three-dimensional convolutional neural network.

[0059] Step S312: Using the updated 3D convolutional neural network, perform convolution operations on the video features of the corresponding audio and video data to obtain the corresponding target features;

[0060] Figure 5 This is a schematic diagram illustrating the fusion of audio and video features according to an embodiment of this application. See also... Figure 5 After feature extraction using a three-dimensional convolutional neural network (3D convolution) and envelope transpose blocks, the convolution activation formula of the three-dimensional convolutional neural network is shown in formula (5):

[0061] y = Wx + b (5);

[0062] Where y is the activation value, W is the convolution kernel parameter learned during the training of the three-dimensional convolutional neural network, x is the activated value, and b is the bias compensation term; in order to enable the video data and the envelope transposed block data (audio data) to be fused, this application embodiment proposes to re-convolve the envelope parameter with the convolution kernel of the three-dimensional convolutional neural network, as shown in formula (6):

[0063] W = W f xf (6);

[0064] Among them, W f x represents the recorded values ​​for the time period corresponding to the envelope transpose block. f Given the convolution kernel parameters, combining equations (5) and (6), we can obtain equation (7):

[0065] y = W f x f x+b (7);

[0066] The physical meaning expressed by formula (7) can be: First, through the parameter W of the sound corresponding to the time slice (or time period). f For convolution kernel x f Perform parameter updates, and then use this convolution kernel W f x f The video clips are convolved to obtain target features; that is, the convolution kernel of the three-dimensional convolutional neural network is updated using the audio features 501 of each audio and video data to obtain an updated three-dimensional convolutional neural network; the updated three-dimensional convolutional neural network is then used to convolve the video features 502 of the corresponding audio and video data to obtain the corresponding target features 503.

[0067] This approach organically combines audio and video to extract various features. The extracted features can be: image features extracted using 2D convolutional neural networks; behavioral features (video features) extracted using 3D convolutional neural networks; or behavioral features extracted by combining behavioral features using network architecture; sound features extracted using 3D convolutional sequences, i.e., envelope transpose blocks; or audio features activated by convolutional kernels, followed by convolution operations on video features to extract both behavioral and sound features; finally, two sets of behavioral and sound features can be extracted simultaneously.

[0068] Step S314: Input the target features of each of the multiple audio and video data into the multi-instance learning model so that the multi-instance learning model can output the probability of the corresponding audio and video data;

[0069] Step S316: Determine the probability distribution information of the plurality of audio and video data based on the probability of each audio and video data in each plurality of audio and video data;

[0070] As the collected video and audio data continue to expand, the probabilities inferred by the multi-instance learning network gradually become distinguishable. Based on the probability corresponding to each audio and video data, the probability distribution information of multiple audio and video data can be determined. The probability distribution information can be used to characterize the distribution pattern of the probabilities corresponding to the audio and video data.

[0071] Step S318: Determine the classification result for each audio / video data based on the probability distribution information;

[0072] The classification results may include abnormal audio and video data and normal audio and video data. If the audio and video data is abnormal, it indicates that the pig's behavior and / or sound is abnormal. If the audio and video data is normal, it indicates that the pig's behavior and sound are normal.

[0073] In general, the number of abnormal pigs is much smaller than the number of normal pigs. Therefore, audio and video data with concentrated and similar probabilities in a normal distribution can be identified as normal audio and video data, i.e., audio and video data corresponding to normal pigs; audio and video data with scattered probabilities in a normal distribution can be identified as abnormal audio and video data, i.e., audio and video data corresponding to abnormal pigs. The probability values ​​of various abnormal situations will be slightly different from the probability values ​​of non-abnormal situations.

[0074] Step S320: Update the multi-instance learning model based on the classification result of each audio and video data;

[0075] Figure 6 This is a schematic diagram of training and inference of a Multiple Instant Learning Network (MIL) according to an embodiment of this application. The biggest differences from related technologies are twofold: first, the training data here needs to be trained in an unsupervised learning manner; second, the output of the MIL is a probability value rather than a multi-class label probability. The following details how unsupervised learning is carried out, the unsupervised online iterative growth method, and how the probability value enhances its anomaly detection accuracy.

[0076] See Figure 6The default pre-trained weight file of a 3D convolutional neural network can be used. For target features obtained from feature extraction and feature fusion of video and audio data, the dimensionality is first reduced to a 4096-dimensional fully connected layer, and then similarly passed through 1024-dimensional and 256-dimensional fully connected layers, finally outputting a scalar probability value. In supervised learning, the true category and the predicted value can be compared, and then backpropagation is used to update the numerical weights of each convolutional kernel. This embodiment of the application is unsupervised learning, so there is no true label and therefore no backpropagation can be performed. To solve this problem, this embodiment of the application assumes that the amount of abnormal data is much smaller than that of normal data, which is also the definition of abnormal. Therefore, when video and audio features are effectively extracted by a 3D convolutional neural network, after the default 3 fully connected layers, the predicted probability values ​​can exhibit a normal distribution. That is, the probability values ​​generated by a large number of non-abnormal data are similar and concentrated. Non-abnormal audio and video data can be distributed within region 601. The probability values ​​of various abnormal audio and video data will be offset from the probability values ​​of non-abnormal data. For example, they can be distributed within region 602 and outside region 601. Without training, this offset is small, and the probability distribution may even have a lot of overlap and intersection. The embodiment of this application aims to update online through posterior labels. In related technologies, posterior labels are used, that is, data with corresponding labels are collected first, and then training is performed. In the embodiment of this application, the posterior labels are used to allow the multi-instance learning system (multi-instance learning network) to classify itself, and then the class is labeled and named, and then fed back to the multi-instance learning network for training to improve the system's classification accuracy.

[0077] It should be noted that a uniform abnormal label can be added to abnormal audio and video data, and a uniform normal label can be added to normal audio and video data; furthermore, different abnormal labels can be added to different abnormal sample data.

[0078] In the early stages of training a multi-instance learning network, since it lacks learning capabilities, it can only detect extreme anomalies. As the amount of data increases, the distribution gradually includes two sets of probability values: extreme anomalies and non-extreme anomalies. Once the existence of extreme anomalies is verified, the initial training task is completed.

[0079] By manually labeling extreme anomalies, the multi-instance learning network (MIL) performs cohesion and decoupling on its two main sets (abnormal audio / video data and normal audio / video data). Specifically, after manual guidance, the MIL confirms the existence of these extreme anomalies, calculates the probability difference between abnormal audio / video data, and feeds this difference back to the neural network (MIL) to automatically learn to propagate smaller probability differences for extreme anomalies. This is defined as cohesion in this embodiment. Similarly, the probability difference between abnormal and non-abnormal audio / video data is calculated and maximized; this is defined as decoupling. When the training of extreme and non-extreme anomalies reaches a plateau—that is, when the loss value can no longer decrease through iteration—it means that at least some identical extreme anomalies can be inferred to have the same probability value, while other anomalies will be distributed around and in the intermediate zone between extremes and normal. At this point, a data point with the highest probability value is manually selected for verification. If the verified anomaly types are similar, its weight is fixed as the pre-training weight for the next data training, completing the intermediate task.

[0080] In the later stages of training a multi-instance learning network, the main focus is on distinguishing different anomalies, as well as differentiating between anomalies with similar behaviors but different sounds, and anomalies with similar sounds but different behaviors. Similar to the early and mid-stage training methods of multi-instance learning networks, a class is manually determined through posterior labeling, that is, different anomaly labels are added to different categories of anomalies. For example, different labels are added to the abnormal audio and video data in region 603 and the abnormal audio and video data in region 604. Then, this type of data is fed back to adjust the output probability values ​​to make the cohesiveness of the output probability values ​​more cohesive and the exponentialness of the output probability values ​​more exponentially. That is, the probability value difference of the same type of anomaly points is made smaller in the forward propagation. Similarly, the probability value difference of different categories of anomalies is calculated and made to generate as large a probability value difference as possible to achieve the purpose of predicting anomalies in pigs.

[0081] Step S322: Use the trained multi-instance learning model to detect abnormal behavior in the pigs.

[0082] In this embodiment, the convolution kernel of the 3D convolutional neural network is first updated using audio features, and then the updated 3D convolutional neural network is used to perform convolution operations on video features. This allows for a more accurate fusion of audio and video features to obtain the target features. Furthermore, this embodiment employs an online learning approach, requiring only the labeling of abnormal and normal audio / video data after the multi-instance learning network detects anomalies. This eliminates the cumbersome process of labeling large amounts of abnormal data in classification detection algorithms. Online learning is more adaptable to the initial stage when abnormal data is scarce; it improves the detection rate of abnormal audio / video data; and it saves significant labor and time costs.

[0083] This application also provides a method for detecting abnormalities in pigs, the method comprising:

[0084] Step S402: Collect multiple audio and video data from the pigs;

[0085] Step S404: Extract features from each of the audio and video data to obtain the corresponding target features;

[0086] Step S406: Input the target features of each of the multiple audio and video data into the multi-instance learning model so that the multi-instance learning model can output the probability of the corresponding audio and video data;

[0087] Step S408: Determine the probability distribution information of the plurality of audio and video data based on the probability of each audio and video data in each plurality of audio and video data;

[0088] The probability distribution information includes a normal distribution.

[0089] Step S410: Determine the classification result of the audio and video data corresponding to the probability of the concentrated distribution in the normal distribution as normal audio and video data;

[0090] Step S412: Determine the classification result of audio and video data other than the normal audio and video data as abnormal audio and video data;

[0091] Step S414: Perform data enhancement on the abnormal audio and video data to obtain enhanced abnormal audio and video data; the data enhancement methods include at least: conventional enhancement, frame interpolation enhancement, and reverse playback enhancement;

[0092] Figure 7 This is a schematic diagram illustrating a data augmentation method according to an embodiment of this application. See also... Figure 7 The data enhancement methods may include conventional enhancement 701, frame interpolation enhancement 702, and reverse playback enhancement 703.

[0093] Among them, conventional enhancement can be achieved using MI. norm It is stated that conventional enhancement refers to commonly used enhancement methods for video stream data. The principles of these methods come from the field of image processing, such as random image cropping, distance-limited cropping, multi-image fusion, color space transformation, salt and pepper noise, Gaussian noise, rotation, and mirroring.

[0094] Frame interpolation enhancement can be achieved using MI. iFThis indicates that an anomaly must be caused by a normal transition, and conversely, a normal occurrence must be caused by an anomaly transition. An anomaly segment typically has a normal phase, a pre-transition phase, an anomaly duration phase, a post-transition phase, and a normal phase. The prediction of the transition from an anomaly phase to a normal phase for an anomaly segment is inevitable, while the prediction of the transition from a normal phase to an anomaly phase for an anomaly segment is uncertain. Therefore, the post-transition phase of an anomaly behavior is more important than the pre-transition phase. In this embodiment, the number of frames in the post-transition phase (post-transition frames) can be used for data augmentation. When the camera's sampling frequency is 15 frames, every 15 frames in the video data of the multi-instance learning network can be set as a video segment. The post-transition frame breakpoint used in this embodiment can be the last 5 frames. The determination of the video segment can be expressed by formula (8):

[0095] U MI =U T / / 15 (8);

[0096] Among them, U T Represents video data, U MI This represents a video segment (also known as a behavior segment), and / / represents the floor division method.

[0097] The transition late stage can be decomposed every 5 frames to obtain the stage decomposition set of the transition late stage. The decomposition process of the transition late stage can be shown in formula (9):

[0098] U time =U 0～4 +U 5～9 +U 10～14 ,x∈U time (9);

[0099] Wherein, referring to formula (9), x can be represented as each video frame in the stage decomposition set of the late transition stage; the last 5 frames U of the late transition stage can be taken. 10～14 As a transitional post-processing frame segment that needs to be inserted into a video clip.

[0100] The frame interpolation formula can be expressed by formula (10):

[0101]

[0102] Among them, MI iF Indicates a segment insertion (i.e., inserting a video segment after the transitional post-production stage), U MI This represents a video clip. This indicates a frame segment in the later stage of the transition.

[0103] The specific data augmentation methods include the following:

[0104] The first method: insert the transitional late frame from the abnormal segment after each video segment; the second method: insert the transitional late frame from the abnormal segment before each video segment except the first video segment; the third method: insert the transitional late frame from the abnormal segment every other frame in each video segment; the fourth method: insert the transitional late frame from the abnormal segment every multiple frames in each video segment.

[0105] Reverse playback enhancement can be achieved using MI back It is indicated that the reverse playback method is similar to the principles of image mirroring and image rotation. For example, it is impossible for a mirror image person or a person to walk upside down in reality, but this method can be used to enhance the data to improve the robustness of the model. Even if a video time slice of an abnormal behavior is reversed, the abnormality of the behavior can still be seen. Increasing the reversal of abnormal behavior segments can also enhance the generalization characteristics of model training. Assuming that the reversal of normal behavior Act is -Act, then the reversal of abnormal behavior Abn is -Abn. There are several ways to enhance the visit: the first is to reverse the entire abnormal event time slice (abnormal segment); the second is to reverse the abnormal duration phase; the third is to reverse the transition phase. The reversal formula can be expressed by formula (11):

[0106] MI back =-U MI (11);

[0107] Step S416: Update the multi-instance learning model using each of the enhanced abnormal audio and video data and each of the normal audio and video data;

[0108] Step S418: Use the trained multi-instance learning model to detect abnormal behavior of the pigs.

[0109] This application also provides a method for detecting abnormalities in pigs, the method comprising:

[0110] Step S502: Collect multiple audio and video data from the pigs;

[0111] Step S504: Extract features from each of the audio and video data to obtain the corresponding target features;

[0112] Step S506: Input the target features of each of the multiple audio and video data into the multi-instance learning model so that the multi-instance learning model can output the probability of the corresponding audio and video data;

[0113] Step S508: Determine the probability distribution information of the plurality of audio and video data based on the probability of each audio and video data in each plurality of audio and video data;

[0114] The probability distribution information includes a normal distribution.

[0115] Step S510: Determine the classification result of the audio and video data corresponding to the probability of the concentrated distribution in the normal distribution as normal audio and video data;

[0116] Step S512: Determine the classification result of audio and video data other than the normal audio and video data as abnormal audio and video data;

[0117] Step S514: Determine the loss function of the multi-instance learning model using the prior and posterior probabilities of each of the abnormal audio and video data and the prior and posterior probabilities of each of the normal audio and video data.

[0118] The probability distribution is stable, and each type of anomaly can be labeled manually using a posterior method. Data augmentation is used to expand the audio and video data, and a multi-instance learning network learns from the augmented audio and video data to make similar audio and video data tend towards a fixed probability value. In this embodiment, the probability values ​​can be normalized to a range of (0, 1), and 0.5 can be used as the dividing line between abnormal and abnormal audio and video data. The closer the probability value of audio and video data is to 1, the more abnormal the audio and video data; the closer the probability value is to 0, the more normal the audio and video data. Furthermore, the minimum probability value among the abnormal audio and video data should be higher than the maximum probability value among the normal sample data.

[0119] The prior probability of abnormal audio and video data is the probability of the corresponding audio and video data output by the multi-instance learning network. The posterior probability of abnormal audio and video data can be calculated using Bayes' theorem, along with the prior probability and likelihood function of the abnormal audio and video data. Similarly, the prior probability of normal audio and video data is the probability of the corresponding audio and video data output by the multi-instance learning network. The posterior probability of normal audio and video data can be calculated using Bayes' theorem, along with the prior probability and likelihood function of the normal audio and video data.

[0120] The prior probability value f(V) corresponding to abnormal audio and video data a The range of values ​​for ) can be expressed by formula (12), and the prior probability value f(V) of normal audio and video data should be... n The range of values ​​can be represented by formula (13), and the relationship between the prior probability values ​​corresponding to abnormal audio and video data and the prior probability values ​​corresponding to normal audio and video data can be represented by formula (14):

[0121] 0.5≤f(V a )<1 (12);

[0122] 0 < f(V) n ) < 0.5 (13);

[0123]

[0124] in, Let represent the minimum probability value among the probability values ​​corresponding to the i abnormal audio and video data. This represents the highest probability value among the probability values ​​corresponding to the i abnormal audio and video data.

[0125] Assuming F(a) is the current probability value of identifying an abnormal segment in the posterior (posterior probability of abnormal audio and video data), and F(n) is the current probability value of identifying a normal segment in the posterior (posterior probability of normal audio and video data), and the loss is iterated continuously during the loss process until the loss is minimized, the loss function of the multi-instance learning network can be expressed by formula (15):

[0126]

[0127] Where, |F(a)-f(V) a )| represents F(a) and f(V) a The absolute value of the difference between ) and |F(n)-f(V) n )| represents F(n) and f(V) n The absolute value of the difference between ).

[0128] To avoid overfitting during training of the multi-instance learning network, smoothing constraints and sparsity constraints can be added to this loss function. The smoothing constraint can be expressed by formula (16), and the sparsity constraint can be expressed by formula (17).

[0129]

[0130]

[0131] Where γ1 and γ2 are penalty coefficients, which are hyperparameters used to express the degree of smoothness and sparsity required by the model (multi-instance learning network), and l1 and l2 are regularization penalty terms.

[0132] The final loss function can be expressed by formula (18):

[0133] LOSS=l(a,n)+γ1l1+γ2l2+γ3||W|| (18);

[0134] Where γ3 is the regularization penalty term, W is the parameter to be learned and trained, and ||W|| represents the norm of W; the multi-instance learning network can gradually improve the types of anomalies that can be inferred by online learning, as well as the detection accuracy and recall, after a long period of posterior annotation.

[0135] Step S516: Adjust the parameter weights of the multi-instance learning model using the loss function to update the multi-instance learning model;

[0136] Step S518: Use the trained multi-instance learning model to detect abnormal behavior of the pigs.

[0137] In this embodiment, data augmentation can be used to address the problem of insufficient negative sample data (i.e., abnormal sample data) in the sample data; improve the balance between positive and negative samples, thereby increasing the detection rate of abnormal audio and video data; in addition, by using the loss function of the multi-instance learning network to adjust the parameter weights of the multi-instance learning network, an updated multi-instance learning network is obtained, which can further improve the detection rate of the multi-instance learning network for abnormal audio and video data.

[0138] Deep learning is a branch of artificial intelligence. It's a means to achieve artificial intelligence, specifically, to solve problems within the field. Over the past 30 years, deep learning has developed into a multidisciplinary field, involving probability theory, statistics, approximation theory, convex analysis, computational complexity theory, and many others. Deep learning theory primarily focuses on designing and analyzing algorithms that allow computers to automatically "learn." Deep learning algorithms are a class of algorithms that automatically analyze data to obtain patterns and use these patterns to predict unknown data. Because learning algorithms involve a significant amount of statistical theory, deep learning is closely related to inferential statistics and is also known as statistical learning theory. In terms of algorithm design, deep learning theory focuses on feasible and effective learning algorithms. Many inference problems are extremely difficult, with no set procedures to follow, so some deep learning research involves developing easily manageable approximation algorithms.

[0139] Video processing technology broadly refers to various techniques for capturing, recording, processing, storing, transmitting, and reproducing a series of still images using electrical signals. This application's embodiments employ OpenCV (Open Source Computer Vision Library) for video processing. OpenCV is a cross-platform computer vision and machine learning software library released under the BSD (Berkeley Software Distribution) license (open source), capable of running on Linux, Windows, Android, and Mac OS operating systems. It is lightweight and efficient, consisting of a series of C functions and a small number of C++ classes, while also providing interfaces for languages ​​such as Python, Ruby, and MATLAB, implementing many common algorithms in image processing and computer vision.

[0140] Anomaly detection technology is used to identify anomalous patterns of behavior that do not conform to expectations. Also known as outlier detection, it is primarily used to identify items, events, or observations that do not match expected patterns or other items in a dataset. Anomalous items often transform into problems such as bank fraud, structural defects, medical issues, or text errors. Anomalies are also referred to as outliers, novelties, noise, biases, and exceptions.

[0141] The main methods for detecting and issuing early warnings of abnormalities in pigs in related technologies are as follows:

[0142] The first method is audio detection: It uses standard signals of abnormal sounds from pigs, extracts feature parameters from these signals, establishes a sound recognition model based on these parameters, and then monitors the sounds emitted by pigs in real time. The sound recognition model is used to determine whether the sounds emitted by pigs are abnormal (such as coughing or biting sounds). This allows for timely detection of abnormal behavior in pigs, enabling appropriate measures to be taken to ensure the normal growth of the herd.

[0143] The second method is image detection: This involves calibrating environmental information, marking the backs of fattening pigs, and then using computer vision technology for target detection. Finally, an SVM (Support Vector Machine) is used to learn the relationship between the detected pig behavior and environmental information. This method primarily addresses the problem of traditional fattening pig detection methods that only track the pig's behavior while ignoring the influence of the pigpen environment on behavior. It aims to accurately detect the growth process of fattening pigs and issue alerts for abnormal behaviors.

[0144] The first method described above has the following disadvantages for detecting abnormalities in pigs:

[0145] First, the detection of abnormalities in pigs is relatively limited. Many abnormal behaviors in pigs do not include audio data that can be detected by sound, and many different abnormalities in pigs have very similar audio manifestations in response to stress and pain.

[0146] Secondly, audio data acquisition is difficult. Pigs differ greatly from humans in both their ability to distinguish and generate audio, making recording equipment generally designed for humans unusable on pigs. For example, the human hearing frequency range is 20 to 20,000 Hz, while the pig's hearing frequency range is 42 to 40,500 Hz.

[0147] Third, the data processing and algorithm recognition are poor. Because sound data is serialized and difficult to visualize and process manually, it is very difficult to separate standard label audio frequencies due to the fusion of multiple sounds and the influence of noise. This affects the recognition accuracy of the algorithm.

[0148] The second method described above has the following disadvantages for detecting abnormalities in pigs:

[0149] First, the data features are singular. Image data only contains two-dimensional image information. From the perspective of data definition, abnormal behavior in pigs is often dynamic and will always be accompanied by time segments. The single feature of images alone cannot define the abnormal behavior of pigs.

[0150] Secondly, algorithmic identification is difficult. Defining anomalies at the source of the data is already challenging. While detection-based algorithms can detect the location information of pigs, they still struggle to identify anomalies. In particular, data on pig anomalies is inherently difficult to collect, and detection techniques rely on a large number of balanced positive and negative samples, making effective training difficult.

[0151] Third, offline training has significant limitations. It requires extensive data balancing beforehand, and if new anomalies are found, a large amount of new anomalous data needs to be labeled before retraining. This approach is difficult to universally adapt to issues of anomalies.

[0152] The technical problem to be solved by the embodiments of this application may include:

[0153] First, outlier data is relatively rare, which can lead to an imbalance between positive and negative samples, making it impossible to train the algorithm.

[0154] Second, data augmentation of abnormal video data is highly dependent on image enhancement, but it is difficult to enhance temporal features. That is, it only changes the color, channel, rotation, etc. of a single image, without involving the relationship between images.

[0155] Third, since there are few negative samples, they can be used up easily during training. If the training time is extended, it will lead to overfitting. If training is stopped, the model training iteration cannot be satisfied at all.

[0156] Fourth, it is difficult to integrate video and audio data inputs in terms of algorithm, and the ways in which pigs express abnormalities are limited to these two observable phenomena. Therefore, a new algorithm that can process video and audio simultaneously is needed.

[0157] Fifth, because of the difficulty in collecting abnormal data, the demand for online learning algorithms is more urgent. Offline training methods are difficult to meet the almost infinite number of abnormal scenarios. Algorithms need to automatically adapt to various abnormal scenarios without requiring a large amount of data labeling.

[0158] This application proposes a pig anomaly detection and early warning system and method based on 3D Convolutional Neural Networks (C3D) and Multiple Instance Learning (MIL). Figure 8 This is a schematic diagram of a pig anomaly detection and early warning process according to an embodiment of this application. See [link / reference] Figure 8 The method includes the following steps:

[0159] Step 801: Data Acquisition;

[0160] The raw data for training and prediction can be obtained through data acquisition, and some simple data processing can be performed, such as cropping, rotating, and scaling, to meet the input dimensions of the subsequent neural network; the raw data includes audio data and corresponding video data.

[0161] See Figure 2 To better analyze and detect the types of anomalies, it is necessary to extract features between behaviors and between behaviors and sounds. In this embodiment, video and audio fusion data are used as the input of the model, which not only makes it difficult to detect anomalies using single video data features, but also makes it difficult to confirm anomaly detection using audio data.

[0162] Step 802: C3D audio-visual fusion feature extraction;

[0163] Among them, C3D audio and video fusion feature extraction can extract features from the collected audio and video data, so that key features are preserved, making it convenient for subsequent network inference to combine and use these features for anomaly detection and classification.

[0164] It should be noted that step 802 may include steps 8021 to 8023:

[0165] Step 8021: C3D feature extraction;

[0166] See Figure 3 In this embodiment of the application, a three-dimensional convolutional neural network 303 can be used to extract features from the video data (video stream) of each audio and video data to obtain the video features of the corresponding audio and video data.

[0167] Step 8022: Convert the audio graph into an envelope transpose block;

[0168] See Figure 4This allows for cepstral analysis of audio data. First, Fourier transform can be used to convert the time-domain information of the acquired audio (such as spectrogram 401) into frequency-domain information, obtaining the audio spectrum X[k] in the audio spectrum diagram 402. Second, the logarithm of both sides of the audio spectrum X[k] 402 can be taken simultaneously, followed by an inverse Fourier transform, to obtain the cepstral x[k]. The cepstral x[k] includes the low-frequency part h[k], which is also the core feature of the audio data. In this embodiment, h[k] can be named the envelope function diagram 403. The cepstral x[k] also includes a spectral detail diagram 404. The envelope function diagram 403 is converted into one-dimensional data U, i.e., the envelope transpose block 405.

[0169] Step 8023: Fusion of 3D features and envelope features;

[0170] See Figure 5 First, the parameters of the convolution kernel are updated using the parameters of the time slice corresponding to the sound (audio feature 501). Then, this convolution kernel is used to perform a convolution operation on the video slice (video feature 502) to obtain the target feature 503.

[0171] Step 803: MIL training inference;

[0172] Among them, see Figure 6 MIL training and inference use the fused target features to fit and separate data through the MIL network, ultimately mapping them to a low-dimensional space to achieve linear separability between anomalous and non-anomalous data. The low-dimensional spatial distribution of the input video packet set has been obtained through MIL.

[0173] It should be noted that the learning process of the multi-instance learning network still faces two major challenges: First, even though online learning can solve the problem of insufficient prior labeled training data, it cannot solve the fundamental issue that outlier data is always a small sample size. In the case of imbalanced positive and negative samples, the model (multi-instance learning network) will treat all samples as positive samples to improve the accuracy. For example, if an outlier event occurs only once in 10,000 events, without any processing, the model will learn that all events are normal in order to meet the 99.99% accuracy requirement. However, if a balance between positive and negative samples is desired, only two images (one positive and one negative) are needed, which obviously cannot be effectively trained.

[0174] The second point is how to mathematically express the loss metric throughout the online learning and training process, so as to programmatically achieve the goal of making the output probability values ​​cohesive and decoupled. Related technologies all use cross-entropy as the loss metric to describe the difference between the true and predicted labels, and update the gradient of each layer weight through backpropagation of the difference. Since the embodiments of this application use an unsupervised posterior method, a new loss method needs to be proposed to update this difference.

[0175] Regarding the first issue, embodiments of this application can employ data augmentation to address the problem of insufficient negative sample data (i.e., outlier sample data) in the sample data. Regarding the second issue, embodiments of this application can employ LOSS... MIL A new loss function is used to train the model to address the problem of how to learn iteratively.

[0176] Step 804: Obtain the classification results.

[0177] Since the vast majority of the data distribution should be considered non-abnormal, with only a very small portion being anomalous, the classification results can directly represent the most abundant portion as non-abnormal data and automatically add a non-abnormal label. The remaining portion (excluding the non-abnormal data) is labeled as anomalous. The various data categories within the anomaly labels have completed preliminary clustering. Only the anomalous name needs to be labeled for each anomalous category. Subsequently, data augmentation can be used to expand the anomalous data through continuous online learning. Following the defined loss function, the distance between anomalous and non-abnormal data, and the distance between various anomalous elements within the anomalous data, will gradually increase, ultimately achieving the goal and effect of detecting and issuing early warnings for pig anomalies.

[0178] This application also provides a method for detecting abnormalities in pigs, the method including the following steps:

[0179] Step S601: Use a multi-instance learning network to monitor the status of pigs in the pigsty;

[0180] It can simultaneously collect video and audio data from pigs.

[0181] Step S602: The collected video and audio data are used to predict probability values ​​through the default online learning platform.

[0182] The online learning platform can deploy a multi-instance learning network (MIL). It can first extract features from the video data and corresponding audio data separately, obtaining video features and corresponding audio features. Then, it fuses the video features and corresponding audio features to obtain the target feature. The target feature can be input into the MIL, which can predict the probability value of the target feature and output the predicted probability value. As the collected video and audio data expands, the probability distribution predicted by the default online learning platform gradually becomes distinguishable.

[0183] Step S603: Based on prior knowledge that there is far more normal than abnormal, manually label the abnormality posteriorly.

[0184] It should be noted that, under normal circumstances, the number of normal pigs is much greater than the number of abnormal pigs. Therefore, the data corresponding to the probability of the concentrated distribution in the probability distribution can be identified as normal data, and the other data besides the normal data can be identified as abnormal data. Abnormal data is labeled with an abnormal label, and normal data is labeled with a normal label.

[0185] Step S604: Perform data augmentation on the label data;

[0186] This includes data augmentation of tagged data, and the data augmentation methods can include conventional augmentation, frame interpolation augmentation, and reverse playback augmentation.

[0187] Step S605: Learn from the probability values ​​of the label data as the initial point to achieve cohesion and decoupling;

[0188] The labeled data refers to data with added normal or abnormal labels. The augmented labeled data can be used to re-adjust the multi-instance learning network. Based on the prior and posterior probabilities of the labeled data, the loss function of the multi-instance learning network is determined, and the parameter weights of the multi-instance learning network are adjusted using the loss function to obtain an updated multi-instance learning network. This updated multi-instance learning network outputs probability values ​​that are cohesive yet disjointed, meaning that extreme outliers propagate forward with smaller probability value differences. Similarly, the probability value difference between outliers and non-outliers is calculated and made to generate as large a probability value difference as possible.

[0189] Step S606: Iterate and stabilize to obtain the ability to distinguish between anomalies and non-anomalies;

[0190] Step S607: Manually verify the anomaly type of the specific label category;

[0191] It should be noted that step S606 mainly distinguishes between abnormal and non-abnormal (i.e., normal), while step S607 mainly distinguishes between different abnormalities, such as similar behavior but different voices, or similar voices but different behaviors. Similar to the method of learning abnormalities and non-abnormalities, the specific abnormality type can be determined by manual posterior labeling. Then, the data with specific abnormality category labels is used to re-adjust the feedback, so that the output probability value is cohesive and extroverted. That is, it makes the probability value difference of the same type of abnormal points propagate forward smaller. Similarly, the probability value difference of different types of abnormalities is calculated, and the probability value difference is made as large as possible.

[0192] Step S608: Repeat steps S604 to S606 to extract and learn new anomalies;

[0193] Step S609: Complete the training to obtain a model with strong robustness and strong generalization, which can be used to detect abnormal behaviors in pigs.

[0194] This application uses pre-trained C3D to extract video and audio features, and utilizes online learning to solve the problem that offline training in related technologies heavily relies on a large amount of prior anomaly labeling data. It also describes in detail the specific principles, data fusion enhancement methods, diverse instance learning (MIL) methods, and specially constructed loss functions for model training.

[0195] The embodiments of this application can provide an online learning method for detecting audio-visual fusion anomalies, as well as analysis and detection of relationships between abnormal behaviors, between abnormal behaviors and abnormal sounds, and between abnormal behavior sounds and abnormal behavior sounds.

[0196] In this embodiment, online learning is more adaptable to the early stages when abnormal data is scarce; the detection rate is higher, and the fusion extraction of multiple features from video and audio enhances the model's learning ability; the ability to classify anomalies is stronger, as the relationships between behaviors and between behaviors and sounds are learned, making it easier to distinguish between anomalies of similar behaviors and anomalies of similar sounds; it can save a lot of labor and time costs, as online learning only requires the algorithm to label the anomalies after detection, eliminating the cumbersome process of labeling a large amount of abnormal data before classification detection algorithms.

[0197] Based on the foregoing embodiments, this application provides an abnormal detection device for pigs. The device includes various modules, which can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0198] Figure 9 This is a schematic diagram of the structural composition of an abnormality detection device for pigs according to an embodiment of this application, as shown below. Figure 9 As shown, the device 900 includes a data acquisition module 901, an extraction module 902, a training module 903, and a detection module 904, wherein:

[0199] The acquisition module 901 is used to acquire multiple audio and video data from pigs;

[0200] Extraction module 902 is used to extract features from each of the audio and video data to obtain the corresponding target features;

[0201] The training module 903 is used to input the target features of each of the multiple audio and video data into the multi-instance learning model to train the multi-instance learning model;

[0202] The detection module 904 is used to detect abnormal behavior of the pigs using a trained multi-instance learning model.

[0203] In one embodiment, each audio / video data includes audio data and video data. The extraction module 902 includes: a first extraction unit, used to extract features from the video data in each audio / video data using a three-dimensional convolutional neural network to obtain video features of the corresponding audio / video data; a second extraction unit, used to perform cepstral analysis on the audio data in each audio / video data to obtain audio features of the corresponding audio / video data; and a fusion unit, used to fuse the audio features and video features of each audio / video data to obtain corresponding target features.

[0204] In one embodiment, the training module 903 includes: an input unit, configured to input the target features of each of the plurality of audio and video data into a multi-instance learning model, so that the multi-instance learning model outputs the probability of the corresponding audio and video data; a first determining unit, configured to determine the probability distribution information of the plurality of audio and video data based on the probability of each of the plurality of audio and video data; a second determining unit, configured to determine the classification result of each of the audio and video data based on the probability distribution information; and an updating unit, configured to update the multi-instance learning model based on the classification result of each of the audio and video data.

[0205] In one embodiment, the probability distribution information includes a normal distribution, and the second determining unit includes: a first determining subunit, used to determine that the classification result of the audio and video data corresponding to the probability of the concentrated distribution in the normal distribution is normal audio and video data; and a second determining subunit, used to determine that the classification result of other audio and video data besides the normal audio and video data is abnormal audio and video data.

[0206] In one embodiment, the updating unit includes: an enhancement subunit, configured to perform data enhancement on the abnormal audio and video data to obtain enhanced abnormal audio and video data; the data enhancement method includes at least: conventional enhancement, frame interpolation enhancement, and reverse playback enhancement; and a first updating subunit, configured to update the multi-instance learning model using each of the enhanced abnormal audio and video data and each of the normal audio and video data.

[0207] In one embodiment, the updating unit includes: a third determining subunit, configured to determine the loss function of the multi-instance learning model using the prior and posterior probabilities of each of the abnormal audio and video data, and the prior and posterior probabilities of each of the normal audio and video data; and a second updating subunit, configured to adjust the parameter weights of the multi-instance learning model using the loss function to update the multi-instance learning model.

[0208] In one embodiment, the second extraction unit includes: a first transformation subunit, configured to perform Fourier transform, logarithmic transformation, and inverse Fourier transform on the audio data of each audio / video data to obtain the envelope function of the corresponding audio / video data; and a second transformation subunit, configured to perform transpose transformation on the envelope function of each audio / video data to obtain the audio features of the corresponding audio / video data.

[0209] In one embodiment, the fusion unit includes: a third update subunit, used to update the convolution kernel of the three-dimensional convolutional neural network using the audio features of each audio and video data to obtain an updated three-dimensional convolutional neural network; and a convolution subunit, used to perform convolution operations on the video features of the corresponding audio and video data using the updated three-dimensional convolutional neural network to obtain the corresponding target features.

[0210] It should be noted that, in this application embodiment, if the above-mentioned abnormal pig detection method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, in essence, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, tablet computer, desktop computer, personal digital assistant, navigator, digital phone, video phone, television, sensor device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, this application embodiment is not limited to any specific hardware and software combination.

[0211] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0212] Correspondingly, embodiments of this application provide an electronic device, Figure 10 This is a schematic diagram of a hardware entity of an electronic device according to an embodiment of this application, such as... Figure 10 As shown, the hardware entity of the electronic device 1000 includes a memory 1001 and a processor 1002. The memory 1001 stores a computer program that can run on the processor 1002. When the processor 1002 executes the program, it implements the steps in the abnormal detection method for pigs described in the above embodiment.

[0213] The memory 1001 is configured to store instructions and applications executable by the processor 1002, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 1002 and various modules in the electronic device 1000. It can be implemented by flash memory or random access memory (RAM).

[0214] Correspondingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the abnormal detection method for pigs provided in the above embodiments.

[0215] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects as the device embodiments. For technical details not disclosed in the storage medium and method embodiments of this application, please refer to the descriptions of the device embodiments of this application for understanding.

[0216] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0217] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0219] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this application may all be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in hardware or in a combination of hardware and software functional units.

[0220] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the related technology, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause computer devices (which may be mobile phones, tablets, desktops, personal digital assistants, navigators, digital phones, video phones, televisions, sensing devices, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0221] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined to obtain new product embodiments without conflict. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined to obtain new method embodiments or device embodiments without conflict.

[0222] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting abnormalities in pigs, characterized in that, The method includes: Collect multiple audio and video data from pigs; each audio and video data includes audio data and video data; Using a three-dimensional convolutional neural network, feature extraction is performed on the video data in each of the audio and video data to obtain the video features of the corresponding audio and video data; Perform cepstral analysis on the audio data in each of the aforementioned audio and video data to obtain the audio features of the corresponding audio and video data; The audio and video features of each audio and video data are fused to obtain the corresponding target features; The target features of each of the multiple audio and video data are input into the multi-instance learning model to train the multi-instance learning model; The abnormal behavior of the pigs was detected using a trained multi-instance learning model.

2. The method according to claim 1, characterized in that, The step of inputting the target features of each of the plurality of audio and video data into a multi-instance learning model to train the multi-instance learning model includes: The target features of each of the multiple audio and video data are input into the multi-instance learning model so that the multi-instance learning model can output the probability of the corresponding audio and video data. Based on the probability of each of the plurality of audio and video data, determine the probability distribution information of the plurality of audio and video data; Based on the probability distribution information, determine the classification result for each audio / video data. The multi-instance learning model is updated based on the classification results of each audio and video data.

3. The method according to claim 2, characterized in that, The probability distribution information includes a normal distribution, and determining the classification result for each audio / video data based on the probability distribution information includes: The classification result of the audio and video data corresponding to the probability of the concentrated distribution in the normal distribution is determined as normal audio and video data; The classification result of audio and video data other than the normal audio and video data is determined to be abnormal audio and video data.

4. The method according to claim 3, characterized in that, The step of updating the multi-instance learning model based on the classification result of each audio / video data includes: The abnormal audio and video data is augmented to obtain enhanced abnormal audio and video data; the data augmentation methods include at least: conventional enhancement, frame interpolation enhancement, and reverse playback enhancement. The multi-instance learning model is updated using each of the enhanced abnormal audio / video data and each of the normal audio / video data.

5. The method according to claim 3, characterized in that, The step of updating the multi-instance learning model based on the classification result of each audio / video data includes: The loss function of the multi-instance learning model is determined by using the prior and posterior probabilities of each of the abnormal audio and video data and the prior and posterior probabilities of each of the normal audio and video data. The loss function is used to adjust the parameter weights of the multi-instance learning model in order to update the multi-instance learning model.

6. The method according to claim 1, characterized in that, The step of performing cepstral analysis on the audio data in each of the audio and video data to obtain the audio features of the corresponding audio and video data includes: Perform Fourier transform, logarithmic transformation, and inverse Fourier transform on the audio data of each audio / video data to obtain the envelope function of the corresponding audio / video data; The envelope function of each audio / video data is transposed to obtain the audio features of the corresponding audio / video data.

7. The method according to claim 1, characterized in that, The step of fusing the audio features and video features of each audio / video data to obtain the corresponding target features includes: The convolution kernel of the three-dimensional convolutional neural network is updated using the audio features of each audio and video data to obtain the updated three-dimensional convolutional neural network. The updated 3D convolutional neural network is used to perform convolution operations on the video features of the corresponding audio and video data to obtain the corresponding target features.

8. An abnormality detection device for pigs, characterized in that, The device includes: The acquisition module is used to acquire multiple audio and video data of pigs; each audio and video data includes audio data and video data; Extraction module, used for Using a three-dimensional convolutional neural network, feature extraction is performed on the video data in each of the audio and video data sets to obtain the video features of the corresponding audio and video data sets; cepstral analysis is performed on the audio data in each of the audio and video data sets to obtain the audio features of the corresponding audio and video data sets; the audio features and video features of each of the audio and video data sets are fused to obtain the corresponding target features; The training module is used to input the target features of each of the multiple audio and video data into the multi-instance learning model to train the multi-instance learning model. The detection module is used to detect abnormal behaviors of the pigs using a trained multi-instance learning model.

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the abnormal detection method for pigs according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the abnormal detection method for pigs according to any one of claims 1 to 7.

Images