Livestock and poultry abnormal sound recognition method and device based on multi-modal audio representation and fusion, medium and equipment

Abstract

The application discloses a livestock and poultry abnormal sound recognition method and device based on multi-modal audio representation and fusion, a medium and equipment, which comprises the following steps: collecting audio signals of livestock and poultry; inputting the audio signals of livestock and poultry into a multi-modal audio feature extraction module to extract time sequence modal features and spectrum modal features of the audio; inputting the extracted time sequence modal features and spectrum modal features into a cross-modal audio feature extraction module for feature fusion; finally, a classifier is used to predict the audio of livestock and poultry, and the livestock and poultry abnormal sound recognition is completed. Therefore, the application significantly improves the recognition accuracy of livestock and poultry cough sound in an actual production environment, and has higher practical engineering application value.

Description

Technical Field

[0001] This invention relates to a method, apparatus, medium, and equipment for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion, and relates to the fields of animal health monitoring and speech signal processing. Background Technology

[0002] In livestock and poultry farming, such as pig farming, respiratory diseases are one of the main factors restricting the development of the pig farming industry, and coughing is an early clinical symptom of respiratory diseases in pigs. Recent studies have shown that monitoring the sound of pig coughs is an effective means of early warning of respiratory diseases.

[0003] Many methods for identifying pig coughs based on deep learning models have been proposed, including feature selection, feature fusion, classifier optimization, and classifier fusion, in order to improve classification performance.

[0004] However, existing classification algorithms are all based on single-modal feature models. In real-world scenarios, sound types are diverse and easily affected by noise. Furthermore, due to the limited number of sound samples, it is difficult to further improve classification accuracy. Summary of the Invention

[0005] The present invention aims to at least solve one of the technical problems existing in the prior art. Therefore, in response to the above-mentioned problems, the purpose of the present invention is to provide a method, apparatus, medium, and device for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion, which can achieve multimodal fusion and improve the recognition accuracy of livestock and poultry coughing sounds in actual production environments.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] In a first aspect, the present invention provides a method for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion, comprising:

[0008] Collect audio signals from livestock and poultry;

[0009] The audio signals of livestock and poultry are input into the multimodal audio feature extraction module to extract the temporal modal features and spectral modal features of the audio.

[0010] The extracted temporal modal features and spectrogram modal features are input into the cross-modal audio feature extraction module for feature fusion. Finally, the classifier predicts the audio of livestock and poultry to complete the identification of abnormal sounds in livestock and poultry.

[0011] In some possible implementations, the multimodal audio feature extraction module includes a temporal modal encoder and a spectrogram modal encoder, wherein the training process of the temporal modal encoder and the spectrogram modal encoder includes:

[0012] Collect animal and poultry sounds to build a corpus;

[0013] Based on the corpus, a pre-trained dataset is constructed;

[0014] The pre-trained dataset is input into the multimodal audio feature extraction module of the multimodal audio representation and fusion framework. First, the pre-trained dataset is randomly pruned from a single sample to two samples with overlap. The two randomly pruned samples are then fed into the temporal modal encoder and the spectrogram modal encoder, respectively, for feature learning and extraction. The temporal modal encoder uses a Transformer encoder to learn the features of the audio waveform, while the spectrogram modal encoder uses SeResNet18 to extract the spatial features of the spectrogram. The model loss value is calculated using these two representations.

[0015] In some possible implementations, the temporal modal encoder first uses a convolutional sampling layer with a kernel size of 17 and a stride of 6. After normalization and ReLU activation, the kernel is fed into the Transformer encoder, which contains a three-layer MLP. The two hidden linear layers have a dimension of 512, each followed by a normalization layer and a ReLU activation function. The third layer is a linear output layer with a dimension of 2048, followed by a normalization layer, and the output is a 2048-dimensional feature.

[0016] In some possible implementations, the spectrogram modal encoder first uses the SeResNet18 model, followed by a three-layer MLP. The two hidden linear layers have a dimension of 512, each followed by a normalization layer and a ReLU activation function. The third layer is a linear output layer with a dimension of 2048, followed by a normalization layer. The three-layer MLP and the normalization layer have the same structure as the temporal modal encoder and share weights, and the output is a 2048-dimensional feature.

[0017] In some possible implementations, the training process for the cross-modal audio feature extraction module and the classifier is as follows:

[0018] Collect animal and poultry sounds to build a corpus;

[0019] Based on the corpus, a supervised dataset is constructed, which uses labeled sample data, with labels including coughing, screaming, sneezing, and other sounds of livestock and poultry.

[0020] The cross-modal audio feature extraction module takes temporal modal features and spectrogram modal features from supervised dataset samples as input. It employs feature fusion and two types of convolutional blocks. The feature fusion operation uses pixel-wise summation to fuse the temporal modal features and spectrogram modal features, resulting in fused features. The first convolutional block contains three linear layers, with different dimensions forming a bottleneck structure. The first two linear layers are followed by a normalization function and a ReLU function, while the output layer has only one normalization function. The second convolutional block contains two linear layers; the first layer has a dimension of 256, followed by a ReLU function. The output of the linear layer is the cross-modal feature, which is then input into a classifier for training to predict the sounds of livestock and poultry.

[0021] In some possible implementations, a sliding window is used to extract segments from the collected animal sounds. The length of the sliding window is a second and the step size is b seconds. x segments of fixed length and no overlap are extracted and used as a pre-training dataset or a supervised dataset. The pre-training dataset or the supervised dataset does not have overlapping data.

[0022] In some possible implementations, the audio signals of livestock and poultry are collected using a Zoom stereo recorder with a built-in microphone. The quantization bit depth of the collected livestock and poultry audio is 16 bits, the sampling rate is 44.1 kHz, and it is stored in WAV format. After data collection, the livestock and poultry recordings are sampled to 16 kHz.

[0023] Secondly, the present invention also provides a livestock and poultry abnormal sound recognition device based on multimodal audio representation and fusion, comprising:

[0024] The signal acquisition unit collects audio signals from livestock and poultry;

[0025] The feature extraction unit inputs the audio signals of livestock and poultry into the multimodal audio feature extraction module to extract the temporal modal features and spectral modal features of the audio.

[0026] The sound recognition unit inputs the extracted temporal modal features and spectrogram modal features of the audio into the cross-modal audio feature extraction module for feature fusion. Finally, the classifier predicts the audio of livestock and poultry to complete the identification of abnormal sounds in livestock and poultry.

[0027] Thirdly, the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the method described thereon.

[0028] Fourthly, the present invention also provides a computer-readable storage medium for storing one or more programs, characterized in that the one or more programs include computer instructions for causing a computer to perform the method.

[0029] Because the present invention adopts the above technical solution, it has the following characteristics:

[0030] 1. This invention utilizes a method that combines multimodal audio representation and contrastive learning to identify the cough sounds of livestock and poultry. First, it uses contrastive learning and a large amount of unlabeled data to enable the model to acquire robust audio representation capabilities, thereby improving the model's recognition accuracy in downstream tasks of livestock and poultry cough identification. Second, it uses a constructed supervised training set to fine-tune downstream tasks and uses a classifier to determine whether the cough sound is correctly identified, thus solving the problem of limited labeled training samples in actual production environments.

[0031] 2. This invention constructs a multimodal audio representation and fusion framework, and achieves cross-modal supervised self-supervised learning through contrastive learning, which effectively improves the feature representation capability. Compared with existing methods, this invention significantly improves the accuracy of pig cough sound recognition in actual production environment and has higher practical engineering application value.

[0032] In summary, this invention can be widely applied to the recognition of animal and poultry sounds. Attached Figure Description

[0033] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings:

[0034] Figure 1 This is a flowchart of the livestock and poultry abnormal sound recognition method based on multimodal audio representation and fusion according to an embodiment of the present invention;

[0035] Figure 2 This is a flowchart of the MSS module according to an embodiment of the present invention;

[0036] Figure 3 This is a flowchart of the CFE module according to an embodiment of the present invention;

[0037] Figure 4 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0038] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0039] Although terms such as first, second, third, etc., may be used in this document to describe multiple elements, components, regions, layers, and / or segments, these elements, components, regions, layers, and / or segments should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or segment from another. Unless the context clearly indicates otherwise, terms such as "first," "second," and other numerical terms used herein do not imply order or sequence. Therefore, the first element, component, region, layer, or segment discussed below may be referred to as the second element, component, region, layer, or segment without departing from the teachings of the exemplary embodiments.

[0040] For ease of description, spatial relative terms may be used in the text to describe the relationship of one element or feature relative to another element or feature as shown in the figure. These relative terms include, for example, "inside," "outside," "middle," "outer," "below," "above," etc. Such spatial relative terms are intended to include different orientations of the device in use or operation, other than those depicted in the figure.

[0041] Existing methods suffer from drawbacks such as reliance on large amounts of labeled data and poor adaptability to complex environments. This invention provides a method, apparatus, device, and medium for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion. The method includes: collecting audio signals from livestock and poultry; inputting the audio signals into a multimodal audio feature extraction module to extract temporal modal features and spectral modal features; inputting the extracted temporal modal features and spectral modal features into a cross-modal audio feature extraction module for feature fusion; and finally, using a classifier to predict the audio signals of livestock and poultry to complete the identification of abnormal sounds. Therefore, this invention can significantly improve the performance of identifying livestock and poultry coughs under actual production conditions, is particularly suitable for data-sparse production scenarios, and provides a solution for early warning of respiratory diseases in livestock and poultry.

[0042] Exemplary embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art. It should be noted that the livestock and poultry used in this embodiment are pigs as an example, but this is not a limitation.

[0043] Example 1: As Figure 1 As shown, the livestock and poultry abnormal sound recognition method based on multimodal audio representation and fusion provided in this embodiment includes:

[0044] S1. Collect the sounds of pigs in the pigsty and build a corpus.

[0045] In this embodiment, the sound of pigs in the pigsty was collected using a Zoom stereo recorder with a built-in microphone. The quantization bit depth of the collected pig audio was 16 bits, the sampling rate was 44.1 kHz, and it was stored in WAV format. In order to enhance the model's ability to adapt to complex acoustic environments and improve its applicability under actual production conditions, no noise reduction procedure was applied to the collected data. After data collection, the pig recordings were sampled to 16 kHz.

[0046] In this embodiment, the acquired raw audio data is preprocessed: since the acquired raw audio data is stereo and has two channels, the two channels of the raw audio data are averaged to obtain a single-channel audio data.

[0047] S2. Based on the corpus, construct a pre-trained dataset and a supervised dataset.

[0048] In this embodiment, the construction of the pre-training dataset and the supervised dataset is carried out as follows:

[0049] The corpus data is divided into two parts: a pre-training dataset and a supervised dataset. The data in the supervised and pre-training datasets do not overlap; each can use 50% of the corpus data, for example, but not limited to this. The pre-training dataset is mainly used for model pre-training, employing unlabeled data to enhance the model's ability to extract pig sound features. The supervised dataset requires labeled sample data to identify specific typical sounds such as coughs. Based on the pre-training, the supervised dataset typically requires a relatively small amount of data, thus addressing the problem of insufficient labeled training samples.

[0050] In this embodiment, a sliding window is used to extract segments from the collected pig sounds. The length of the sliding window is a seconds and the step size is b seconds. x segments of fixed length and no overlap are extracted and used as a pre-training dataset or a supervised dataset. For example, if the length of the sliding window is set to 1.5 seconds and the step size is 1.5 seconds, 40,548 segments of fixed length and no overlap are extracted and used as a pre-training dataset. This is just one example, and it is not limited to this.

[0051] In this embodiment, the supervised dataset is constructed by labeling four sound category samples of pigs from the original audio. The acquired supervised dataset is labeled with various sound occurrence categories, including cough, scream, sneezing, and other pig sounds. The number of samples in each category is shown in Table 1. Considering the different durations of sound events, samples with a duration less than a certain number of seconds, such as 1 second, are padded with 0, while samples with a duration exceeding a certain number of seconds, such as 1 second, are randomly pruned to y seconds. In this embodiment, y seconds is set to 1 second, but this is not a limitation.

[0052] Furthermore, the supervised training set was further divided into training, validation, and test datasets in an 8:1:1 ratio. Data augmentation was also applied to the training set, including methods such as adding Gaussian noise, pitch variation, time stretching, time masking, and bit reduction. The number of each sound category in the augmented supervised dataset is shown in Table 1.

[0053] Table 1

[0054]

[0055] S3. Input the pre-trained dataset into the Multimodal Audio Feature Extraction (MFE) module of the Multimodal Audio Representation and Fusion Framework (MSS) for feature extraction.

[0056] In this embodiment, the implementation principle of MSS is as follows: using the SimSiam architecture as the baseline framework, in order for MSS to learn more generalized representations, MSS creates two partitions from two different parts of a sample, thereby increasing the difficulty of identifying the two partitions as the same sample. One partition uses a Transformer encoder to learn the features of the original temporal waveform of the audio; the other partition uses SeResNet18 to extract the spatial features of the spectrogram. The model loss value is calculated using these two representations. The spectrogram can be a log-Mel spectrogram.

[0057] Furthermore, MSS randomly prunes the data in the pre-training dataset by splitting a single sample into two samples. The two pruned samples are then fed into the temporal modal encoder block and the spectrogram modal encoder, respectively, for feature extraction.

[0058] Furthermore, the specific implementation method of the MSS method for creating two partitions, i.e., the multimodal audio feature extraction module, is as follows: For a given audio segment of 'a' seconds, MSS randomly cuts it into two different segments, segment 1 and segment 2, with each segment lasting for 't' seconds, where 2t > a; the overlap is to ensure that the similarity between the two segments is not lost, ensuring that MSS can identify the two segments as the same sample; different transformations are performed on segment 1 and segment 2 respectively, and the transformation methods include adding Gaussian noise, pitch variation, time stretching, time masking, and bit reduction, etc.; one of the transformed segments is the original audio waveform, which is fed into the Transformer encoder to learn the features of the audio waveform, while the other transformed segment is converted into a spectrogram representation, and the spatial features of the spectrogram are extracted using SeResNet18.

[0059] Specifically, such as Figure 2 As shown, the pre-training dataset is fed into the MFE module of MSS. First, the pre-training data is randomly pruned from a single sample, so that one sample becomes two samples. For example, in this embodiment, for a given 1.5-second audio clip, MSS randomly prunes it into two different clips, clip 1 and clip 2, each clip lasting 1 second. At least 50% overlap between the two clips is allowed. The overlap is to ensure that the similarity between the two clips is not lost, and to ensure that MSS can identify the two clips as the same sample. After data preprocessing, they are fed into the temporal modal encoder and spectrogram modal encoder respectively for feature learning and extraction.

[0060] The temporal modal encoder uses a Transformer encoder to learn features from the audio waveform. The temporal modal encoder first uses a convolutional sampling layer with a kernel size of 17 and a stride of 6. After normalization and ReLU activation, the kernel is fed into the Transformer encoder. This is followed by a three-layer MLP: two hidden linear layers with a dimension of 512, each followed by a normalization layer and a ReLU activation function; and a third linear output layer with a dimension of 2048, followed by a normalization layer. The output of the temporal modal encoder is a 2048-dimensional feature.

[0061] The spectrogram modal encoder uses SeResNet18 to extract spatial features from the spectrogram. It is followed by a three-layer MLP: two hidden linear layers with a dimension of 512, each followed by a normalization layer and a ReLU activation function; and a third linear output layer with a dimension of 2048, followed by a normalization layer. The three-layer MLP and normalization layer have the same structure and share weights as the temporal modal encoder. The output of the spectrogram modal encoder is a 2048-dimensional feature, and the model loss is calculated using these two feature outputs from the temporal modal encoder and the spectrogram modal encoder. The spectrogram modal module is largely the same as the temporal modal module, except that it lacks convolutional sampling layers and does not add a normalization layer after the third layer of the final MLP.

[0062] Furthermore, the loss function is calculated using the loss calculation method of the SimSiam framework.

[0063] S4. The extracted temporal modal and spectrogram modal features of the audio are input into the cross-modal audio feature extraction (CFE) module trained by the supervised dataset for feature fusion. Finally, the classifier predicts the coughing sounds of pigs to complete the recognition of pig coughing sounds.

[0064] In this embodiment, as Figure 3 As shown, the cross-modal audio feature extraction module is used to extract robust sound feature representations from the fused features. The structure of the cross-modal audio feature extraction module consists of a feature fusion operation and two types of convolutional blocks. The feature fusion operation uses pixel-wise summation to fuse temporal modal features and spectrogram modal features to obtain fused features. The fused features are input into two types of convolutional blocks. The first convolutional block contains three linear layers with dimensions of 2048, 512, and 2048, forming a bottleneck structure. The first two linear layers are followed by a normalization function and a ReLU function, respectively, and the output layer has only a normalization function. The second convolutional block contains two linear layers. The first layer has a dimension of 256, followed by a ReLU function, and the output linear layer contains the cross-modal features. The input of the cross-modal audio feature extraction module is the temporal modal features and spectrogram modal features of the supervised dataset, and the output is the cross-modal features. The cross-modal features are input into a classifier for training to achieve sound prediction of pigs.

[0065] Example 2: Following the method for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion provided in Example 1, this example provides a device for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion. The device provided in this example can implement the method for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion described in Example 1. This device can be implemented through software, hardware, or a combination of both. For ease of description, this example is described by dividing the functionality into various units. Of course, in implementation, the functions of each unit can be implemented in one or more software and / or hardware components. For example, the device may include integrated or separate functional modules or units to execute the corresponding steps in the methods of Example 1. Since the device in this example is basically similar to the method example, the description process of this example is relatively simple. Relevant details can be found in the description of Example 1. The embodiment of the device for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion provided by this invention is merely illustrative.

[0066] Specifically, the livestock and poultry abnormal sound recognition device based on multimodal audio representation and fusion provided in this embodiment includes:

[0067] The signal acquisition unit collects audio signals from livestock and poultry;

[0068] The feature extraction unit inputs the audio signals of livestock and poultry into the multimodal audio feature extraction module to extract the temporal modal features and spectral modal features of the audio.

[0069] The sound recognition unit inputs the extracted temporal modal features and spectrogram modal features of the audio into the cross-modal audio feature extraction module for feature fusion. Finally, the classifier predicts the audio of livestock and poultry to complete the identification of abnormal sounds in livestock and poultry.

[0070] Example 3: This example provides an electronic device corresponding to the livestock and poultry abnormal sound recognition method based on multimodal audio representation and fusion provided in Example 1. The electronic device can be an electronic device for the client, such as a mobile phone, laptop, tablet computer, desktop computer, etc., to execute the method of Example 1.

[0071] like Figure 4 As shown, the electronic device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus to enable communication between them. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the method of Embodiment 1. The implementation principle and technical effects are similar to those of Embodiment 1, and will not be repeated here. Those skilled in the art will understand that... Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computing device to which the present invention is applied. A specific computing device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0072] In a preferred embodiment, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), and optical discs.

[0073] In a preferred embodiment, the processor can be any type of general-purpose processor such as a central processing unit (CPU) or a digital signal processor (DSP), and is not limited thereto.

[0074] Example 4: This example provides a computer-readable storage medium for storing one or more programs, the one or more programs including computer instructions, which, when executed by a computer, cause the computer to perform the method provided in Example 1 above.

[0075] In a preferred embodiment, the computer-readable storage medium may be a tangible device for holding and storing instructions executable, such as, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. The computer-readable storage medium stores computer program instructions that cause a computer to perform the method provided in Embodiment 1 above.

[0076] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In the description of this specification, the terms "a preferred embodiment," "furthermore," "specifically," "in this embodiment," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments in this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion, characterized in that, include: Collect audio signals from livestock and poultry; The audio signals of livestock and poultry are input into a multimodal audio feature extraction module to extract temporal modal features and spectral modal features of the audio. The multimodal audio feature extraction module includes a temporal modal encoder and a spectrogram modal encoder. The training process of the temporal modal encoder and the spectrogram modal encoder includes: Collect animal and poultry sounds to build a corpus; Based on the corpus, a pre-trained dataset is constructed; The pre-trained dataset is input into the multimodal audio feature extraction module of the multimodal audio representation and fusion framework. First, a single sample in the pre-trained dataset is randomly pruned, resulting in two samples with overlap. These two pruned samples are then fed into the temporal modal encoder and the spectrogram modal encoder, respectively, for feature learning and extraction. The temporal modal encoder uses a Transformer encoder to learn the features of the audio waveform; the spectrogram modal encoder uses SeResNet18 to extract the spatial features of the spectrogram. These two representations are used to calculate the model loss. The temporal modal encoder first uses a convolutional sampling layer with a kernel size of 17 and a stride of 6, followed by normalization and ReLU. The activation function is then fed into the Transformer encoder, which contains a three-layer MLP. The two hidden linear layers have a dimension of 512, each followed by a normalization layer and a ReLU activation function. The third layer is a linear output layer with a dimension of 2048, followed by a normalization layer, and the output is a 2048-dimensional feature. The spectrogram modal encoder first uses the SeResNet18 model, which contains a three-layer MLP. The two hidden linear layers have a dimension of 512, each followed by a normalization layer and a ReLU activation function. The third layer is a linear output layer with a dimension of 2048, followed by a normalization layer. The three-layer MLP and the normalization layer have the same structure as the temporal modal encoder and share weights, and the output is a 2048-dimensional feature. The extracted temporal modal features and spectrogram modal features are input into the cross-modal audio feature extraction module for feature fusion. Finally, the classifier predicts the audio of livestock and poultry to complete the identification of abnormal sounds in livestock and poultry.

2. The method for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion according to claim 1, characterized in that, The training process for the cross-modal audio feature extraction module and classifier is as follows: Collect animal and poultry sounds to build a corpus; Based on the corpus, a supervised dataset is constructed, which uses labeled sample data, with labels including coughing, screaming, sneezing, and other sounds of livestock and poultry. The cross-modal audio feature extraction module takes temporal modal features and spectrogram modal features from supervised dataset samples as input. It employs feature fusion and two types of convolutional blocks. The feature fusion operation uses pixel-wise summation to fuse the temporal modal features and spectrogram modal features, resulting in fused features. The first convolutional block contains three linear layers, with different dimensions forming a bottleneck structure. The first two linear layers are followed by a normalization function and a ReLU function, while the output layer has only one normalization function. The second convolutional block contains two linear layers; the first layer has a dimension of 256, followed by a ReLU function. The output of the linear layer is the cross-modal feature, which is then input into a classifier for training to predict the sounds of livestock and poultry.

3. The method for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion according to claim 2, characterized in that, Segments are extracted from collected livestock and poultry sounds using a sliding window. The sliding window has a length of time 'a' seconds and a step size of 'b' seconds. x segments of fixed length with no overlap are extracted and used as either a pre-training dataset or a supervised dataset. There is no overlapping data in either the pre-training dataset or the supervised dataset.

4. The method for identifying abnormal sounds in livestock and poultry based on multimodal audio representation and fusion according to claim 1, characterized in that, The audio signals of livestock and poultry were collected using a Zoom stereo recorder with a built-in microphone. The quantization bit depth of the collected livestock and poultry audio was 16 bits, the sampling rate was 44.1 kHz, and it was stored in WAV format. After data collection, the livestock and poultry recordings were sampled to 16 kHz.

5. An apparatus for implementing the livestock and poultry abnormal sound recognition method based on multimodal audio representation and fusion as described in any one of claims 1-4, characterized in that, include: The signal acquisition unit collects audio signals from livestock and poultry; The feature extraction unit inputs the audio signals of livestock and poultry into the multimodal audio feature extraction module to extract the temporal modal features and spectral modal features of the audio. The sound recognition unit inputs the extracted temporal modal features and spectrogram modal features of the audio into the cross-modal audio feature extraction module for feature fusion. Finally, the classifier predicts the audio of livestock and poultry to complete the identification of abnormal sounds in livestock and poultry.

6. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the method according to any one of claims 1-4.

7. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include computer instructions for causing a computer to perform the method according to any one of claims 1-4.

Images