A livestock abnormal sound on-site identification method, device, equipment and medium
Patent Information
- Application Number
- CN202511976219.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-25
- Publication Date
- 2026-08-21
- Estimated Expiration
- 2045-12-25
AI Technical Summary
但是这些分类算法均是基于单一模态特征模型，在现实场景中，声音种类繁杂，易受噪声干扰，并且由于声音样本过少，分类精度很难进一步提升
[0015]本发明由于采取以上技术方案，其具有以下特点：
Smart Images
Figure CN121811924B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of livestock health monitoring technology, specifically to a method, device, equipment, and medium for on-site identification of abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning. Background Technology
[0002] Identifying abnormal sounds in livestock is crucial in livestock farming. For example, respiratory diseases are a major factor hindering the development of the dairy industry, and coughing, as an abnormal sound, is an early clinical symptom of respiratory diseases in calves. Recent studies have shown that monitoring calf cough sounds is an effective means of early warning of respiratory diseases.
[0003] Many methods for recognizing calf coughs based on deep learning models have been proposed, mainly including feature selection, feature fusion, classifier optimization, and classifier fusion, to improve classification performance. However, these classification algorithms are all based on single-modal feature models. In real-world scenarios, sound types are diverse and easily affected by noise. Furthermore, due to the limited number of sound samples, it is difficult to further improve classification accuracy. Summary of the Invention
[0004] This invention aims to at least solve one of the technical problems existing in the prior art. Therefore, in response to the above-mentioned problems, the purpose of this invention is to provide a method, apparatus, device, and medium for on-site identification of abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning, which can achieve multimodal fusion and improve the accuracy of abnormal livestock sound identification in actual production environments.
[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: The first invention provides a method for on-site recognition of abnormal sounds in livestock based on multimodal sound feature pre-training and dense connection adapter fine-tuning, including: Collect audio recordings of livestock inside livestock sheds; Convert livestock audio into Mel spectrograms; The Mel spectrogram is input into the livestock abnormal sound recognition model for abnormal sound identification. The training of the livestock abnormal sound recognition model includes: The Mel spectrogram is masked based on the multimodal audio pre-training framework. The masked Mel spectrogram and the corresponding original audio waveform are then input into the multimodal audio pre-training framework. The multimodal audio pre-training framework is pre-trained using self-supervised learning to obtain the Mel spectrogram encoder. Set up a downstream task fine-tuning model, embed a dense connection adapter in the Mel spectrogram encoder to fine-tune the spectrogram features for specific tasks to obtain classification features; Classification features are used for classification and identification via a linear classification head.
[0006] In some possible implementations, the specific training process of the Mel spectrogram encoder is as follows: Construct a pre-trained dataset; Convert the raw audio waveform data of the pre-trained dataset into Mel spectrograms; A masked Mel spectrogram is formed by performing a masking operation on the Mel spectrogram. The masked Mel spectrogram and the original audio waveform are then input into the Mel spectrogram encoder and the waveform encoder, respectively, to extract the spectrogram features and the audio waveform features. The spectrogram features and audio waveform features are input into the modal fusion layer, and then the output fused features are input into the Mel spectrogram decoder. The Mel spectrogram decoder performs masked Mel spectrogram reconstruction to obtain the reconstructed Mel spectrogram. The loss value between the reconstructed Mel spectrogram and the original Mel spectrogram is calculated and used as the training parameter for the multimodal audio pre-training framework to complete the training of the Mel spectrogram encoder.
[0007] In some possible implementations, spectrogram features and audio waveform features are input into a modal fusion layer, and the output fused features are then input into a Mel spectrogram decoder. The Mel spectrogram decoder performs masked Mel spectrogram reconstruction to obtain the reconstructed Mel spectrogram. The process is as follows: spectrogram features and audio waveform features are fused using a waveform-Mel spectrogram cross-attention mechanism, and the fused features are sent to the Mel spectrogram decoder to reconstruct the masked Mel spectrogram.
[0008] In some possible implementations, the fine-tuning training process for the densely connected adapter is as follows: Build a fine-tuning dataset; Label various vocalization categories in the fine-tuning dataset, including coughing, normal humming, excretion sounds, and other sounds from livestock sheds; Convert the audio waveform data in the fine-tuning dataset into Mel spectrograms; The Mel spectrogram is fed into the downstream task to fine-tune the model. After each attention mechanism layer of the obtained Mel spectrogram encoder, an adapter module of dense connection adapter is embedded. All embedded adapter modules are connected in a dense connection manner to fine-tune the spectrogram features for abnormal sound recognition and obtain classification features. During the downstream task fine-tuning, all parameters of the Mel spectrogram encoder are frozen, and only the parameters of the dense connection adapter are trained.
[0009] In some possible implementations, each layer's adapter can not only process the features of the current layer, but also receive the adapter features of all previous layers, forming dense features. The implementation logic is as follows:
[0010] in, as well as This is the output of the previous adapter layers. These features are concatenated in a dimensional manner and then passed through a linear projection layer. Perform dimensionality reduction. As the output of the current layer dense adapter i Indicates the current layer.
[0011] In some possible implementations, the process of obtaining the pre-training dataset and the fine-tuning dataset is as follows: The ZoomH2n stereo recorder with a built-in microphone is used to capture audio from livestock. The sliding window method was used to extract segments from the collected raw audio. Since the duration of each type of livestock sound segment collected did not exceed 1 second, the sliding window length was set to 1 second and the step size was 1 second. Finally, several fixed-length and non-overlapping audio segments were extracted and used as the pre-training dataset. The fine-tuned dataset used Adobe Audition software to categorize the collected audio.
[0012] Secondly, the present invention also provides a device for on-site identification of abnormal sounds in livestock, comprising: An audio collection unit is configured to collect audio from livestock inside livestock sheds; The spectrum conversion unit is configured to convert livestock audio into a Mel spectrogram. An abnormal sound prediction unit is configured to input a Mel spectrogram into a livestock abnormal sound recognition model for abnormal sound recognition. The training of the livestock abnormal sound recognition model includes: The Mel spectrogram is masked based on the multimodal audio pre-training framework. The masked Mel spectrogram and the corresponding original audio waveform are then input into the multimodal audio pre-training framework. The multimodal audio pre-training framework is pre-trained using self-supervised learning to obtain the Mel spectrogram encoder. Set up a downstream task fine-tuning model, embed a dense connection adapter in the Mel spectrogram encoder to fine-tune the spectrogram features for specific tasks to obtain classification features; Classification features are used for classification and identification via a linear classification head.
[0013] Thirdly, the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the method described thereon.
[0014] Fourthly, the present invention also provides a computer-readable storage medium for storing one or more programs, said one or more programs including computer instructions for causing a computer to perform the method.
[0015] Because the present invention adopts the above technical solution, it has the following characteristics: 1. This invention collects livestock audio; converts the livestock audio into Mel spectrograms; constructs a multimodal audio pre-training framework (MAP); performs a masking operation on the Mel spectrograms; inputs the masked Mel spectrograms and the corresponding original audio waveforms of the livestock into the MAP framework; and uses self-supervised learning to pre-train the MAP framework to obtain a Mel spectrogram encoder, which has the ability to extract Mel spectrogram features, thereby solving the problem of limited labeled training samples of livestock in actual production environments.
[0016] 2. This invention constructs a dense connection adapter fine-tuning model, embedding a dense connection adapter into the obtained Mel spectrogram encoder to perform spectrogram feature fine-tuning for specific tasks, thereby obtaining classification features. During the downstream task fine-tuning, all parameters of the Mel spectrogram encoder are frozen, and only the parameters of the dense connection adapter are trained. Finally, the classification features are fed into a linear classification head for classification and recognition, thus solving the problems of insufficient feature extraction capability of the linear classification head and low efficiency of full-scale fine-tuning in downstream fine-tuning.
[0017] In summary, compared with existing methods, this invention significantly improves the accuracy of livestock cough sound recognition in actual production environments, has higher practical engineering application value, and can be widely applied in livestock breeding. Attached Figure Description
[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings: Figure 1 This is a flowchart of the method for on-site recognition of abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning according to an embodiment of the present invention; Figure 2 This is a flowchart of the MAP framework according to an embodiment of the present invention; Figure 3 This is a flowchart of the Mel spectrogram encoder and DA module according to an embodiment of the present invention; Figure 4 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0019] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.
[0020] Although terms such as first, second, third, etc., may be used in this document to describe multiple elements, components, regions, layers, and / or segments, these elements, components, regions, layers, and / or segments should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or segment from another. Unless the context clearly indicates otherwise, terms such as "first," "second," and other numerical terms used herein do not imply order or sequence. Therefore, the first element, component, region, layer, or segment discussed below may be referred to as the second element, component, region, layer, or segment without departing from the teachings of the exemplary embodiments.
[0021] For ease of description, spatial relative terms may be used in the text to describe the relationship of one element or feature relative to another element or feature as shown in the figure. These relative terms include, for example, "inside," "outside," "middle," "outer," "below," "above," etc. Such spatial relative terms are intended to include different orientations of the device in use or operation, other than those depicted in the figure.
[0022] To address the shortcomings of existing methods, such as reliance on large amounts of labeled data and poor adaptability to complex environments, this invention provides a method, apparatus, device, and medium for on-site identification of abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning. The method includes: collecting livestock audio from within livestock sheds; converting the livestock audio into a Mel spectrogram; inputting the Mel spectrogram into a livestock abnormal sound identification model for recognition. The model training process involves: masking the Mel spectrogram based on a multimodal audio pre-training framework; inputting the masked Mel spectrogram and the corresponding original audio waveform of the livestock into a MAP (Magnetic Mapping Assay), performing MAP pre-training using self-supervised learning to obtain a Mel spectrogram encoder; setting up a downstream task fine-tuning model, embedding a dense connection adapter in the Mel spectrogram encoder to fine-tune the spectrogram features for specific tasks to obtain classification features; and performing classification recognition using a linear classification head. Therefore, this invention significantly improves the performance of identifying abnormal sounds such as dairy cow coughing under actual production conditions, is particularly suitable for data-sparse real-world production scenarios, and provides a solution for early warning of respiratory diseases in calves.
[0023] Exemplary embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art. It should be noted that the livestock used in this invention are specifically illustrated using calves as an example, with the abnormal sound device citing the coughing sound of a calf as an example. However, this is not a limitation and can be applied to the identification of abnormal sounds in other livestock.
[0024] Example 1: As Figure 1 As shown, the method for on-site recognition of abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning provided in this embodiment includes: S1. Collect the sounds of calves in the calf shed and build a corpus.
[0025] In this embodiment, a ZoomH2n stereo recorder with a built-in microphone is used to collect 16-bit calf audio with a sampling rate of 44.1 kHz, which is stored in WAV format. This is just one example, and it is not limited to this.
[0026] Furthermore, in order to enhance the model's ability to adapt to complex acoustic environments and improve its applicability under actual production conditions, no noise reduction procedure was applied to the collected data. After data collection, the calf recordings were sampled to 16kHz.
[0027] S2. Based on the corpus, construct a pre-trained dataset and a fine-tuning dataset.
[0028] In this embodiment, audio segments are extracted from the collected raw audio using a sliding window method. Since the duration of each type of calf sound segment collected does not exceed 1 second, the sliding window length is set to 1 second and the step size is 1 second. Finally, several (for example, 26908 in this embodiment, but not limited to this) fixed-length and non-overlapping audio segments are extracted and used as a pre-training dataset.
[0029] In this embodiment, the fine-tuning dataset was obtained by classifying the collected audio using Adobe Audition software. The fine-tuning dataset required labeling various vocal categories, including coughing, normal humming, excretion sounds, and other sounds from the calf pen. Considering the different sound categories and their corresponding durations, vocal samples with durations less than a specific number of seconds (e.g., 1 second in this embodiment, but not limited to this) were padded with 0 to 1, while those with durations exceeding the specific number of seconds (e.g., 1 second in this embodiment, but not limited to this) were pruned. The fine-tuning dataset was further divided into training, validation, and testing datasets in an 8:1:1 ratio.
[0030] Furthermore, data augmentation can be performed on both the pre-training dataset and the fine-tuning dataset. Data augmentation methods used include adding Gaussian noise, pitch variation, volume variation, time stretching, and time masking. For example, the number of each sound category in the fine-tuning dataset after data augmentation is shown in Table 1.
[0031] Table 1
[0032] Wherein, Cough represents the sound of a calf coughing, Ab represents the normal humming sound of a calf, Excrement represents the sound of a calf excreting, Metal represents the metallic clanging sound of a calf in an outdoor calf pen, Total represents the number of all sound categories; Vocalizations represents the sound categories, Original dataset represents the number of original audio files, Training / Augmented training represents the number of samples in the training set / amplified training set for each sound category, Validation represents the number of validation set samples for each sound category, and Test represents the number of test set samples for each sound category.
[0033] S3. Reconstructing masked Mel spectrograms based on pre-trained datasets.
[0034] In this embodiment, a pre-trained dataset is used for pre-training based on the MAP framework. This applies the masked autoencoder self-supervised learning method to the MAP framework, completing the pre-training of a 12-layer Mel spectrogram encoder (Pre-TrainMelEncoder). Figure 2As shown, the training process is as follows: Convert the raw audio waveform data in the pre-training dataset into Mel spectrograms; The mel spectrogram is masked according to the masking ratio and masking method to form a masked mel spectrogram. For example, in this embodiment, for a given 1-second audio segment, it is first converted into a mel spectrogram, and the converted mel spectrogram is masked. The masking ratio is 0.6 and the masking method is block masking. The higher the masking ratio, the less information the model obtains in advance. The masked Mel spectrogram and the original audio waveform are input into the Mel spectrogram encoder and the waveform encoder, respectively, to extract Mel spectrogram features and audio waveform features. The spectrogram features and audio waveform features are input into the modal fusion layer (WMFL), and the output fused features are then input into the Mel spectrogram decoder. The Mel spectrogram decoder performs masked Mel spectrogram reconstruction to obtain the reconstructed Mel spectrogram. The modal fusion layer uses a waveform-Mel spectrogram cross-attention mechanism for modal feature fusion. Specifically, the masked Mel spectrogram is used as the query (Q) vector in the attention mechanism, and the original audio waveform is used as the key (K) vector and value (V) vector in the attention mechanism. Thus, the cross-attention mechanism is used to realize the Mel spectrogram reconstruction of the masked original audio waveform.
[0035] The loss value between the reconstructed Mel spectrogram and the original Mel spectrogram is calculated and used as the training parameter of the MAP framework to complete the pre-training of the Mel spectrogram encoder. The loss function is calculated using the mean absolute error loss method.
[0036] S4. Fine-tune downstream tasks based on the fine-tuning dataset.
[0037] In this embodiment, as Figure 3 As shown, after each layer of attention mechanism in the pre-trained Mel spectrogram encoder, an Adapter module of dense connection adapter (DA) is embedded. All embedded Adapter modules are connected in a dense connection manner. During the fine-tuning of the downstream task, the backbone parameters of the Mel spectrogram encoder are frozen, and only the parameters of the DA are trained. The DA connects the Adapter modules embedded in each layer in a dense connection manner, so that the Adapter can not only process the features of the current layer, but also receive the output features of all previous Adapters, and finally form classification features. The classification features are input into the linear classifier for cough sound recognition, and the fine-tuning of the downstream task is completed.
[0038] Specifically, the livestock abnormal sound recognition model in this embodiment includes a pre-trained Mel spectrogram encoder and an adapter module embedded with a dense connection adapter (DA) after each layer of attention mechanism. All embedded adapter modules are connected in a dense connection manner to achieve prediction of livestock coughing sounds. When fine-tuning downstream tasks, all parameters of the Mel spectrogram encoder are frozen, and only the parameters of the DA are trained. Each embedded adapter module can not only process the features of the current layer, but also receive the output features of all previous adapters to form a classification feature. The classification feature is input into the linear classification for cough sound recognition to complete the calf cough recognition work.
[0039] Specifically, in this embodiment, a DA (Data Adapter) module is embedded after each attention mechanism layer of the pre-trained Mel spectrogram encoder. This Adapter module has a standard "down-projection-up-projection" structure, and its weights are initialized to zero to minimize perturbation to the backbone network at the start of training. During the forward propagation of the Mel spectrogram encoder, an embedded Adapter module not only processes the features of the current layer but also employs a dense connection approach, enabling each embedded Adapter module to receive features output from all previous embedded Adapter modules, forming classification features. The specific implementation logic is as follows:
[0040] in, This is the output of the previous adapter layers. These features are concatenated in a dimensional manner and then passed through a linear projection layer. Reduce its dimensions back to d_model , As the output of the current layer i dense adapter.
[0041] The following detailed embodiments illustrate the verification results of the present invention's method for on-site recognition of abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning. The specific verification process is as follows: The calculation method for the calf cough identification index is as follows:
[0042] In the formula, This represents the accuracy rate of calf cough recognition. TP The number of calves whose coughs are correctly identified as coughs. FP The number of non-calf coughs that were incorrectly identified as coughs;
[0043] In the formula, FN represents the recall rate for identifying calf coughs, where FN is the number of calf coughs that were incorrectly identified as non-coughs.
[0044] In the formula, This represents a weighted average of precision and recall.
[0045] In the formula, This represents the accuracy of calf cough identification, where TN is the number of non-calves coughing that were correctly identified as non-coughing.
[0046] 1. To verify the performance of the proposed MAP framework in audio data of other species, this embodiment selects pig audio for performance verification and compares the combined effects of different species datasets: The calf audio data was used as pre-training data to verify the recognition effect of the pig audio data, achieving a recognition accuracy of 86.84%. The audio data of live pigs was used as pre-training data to verify the recognition effect of the audio data of live pigs, and a recognition accuracy of 91.67% was achieved. Audio data from calves and pigs were used as pre-training data to verify the recognition effect of pig audio data, achieving a recognition accuracy of 92.42%. 2. To verify the role of the densely connected adapter at each level in the Mel spectrogram encoder, this embodiment compares the effects of adding adapters at different levels on the calf dataset: By adding adapters to each of the 12 layers in the Mel spectrogram encoder, a recognition accuracy of 96.54% was achieved. By adding adapters to each of the 0-5 layers in the Mel spectrogram encoder, a recognition accuracy of 94.76% was achieved. By adding adapters to each of layers 6-12 in the Mel spectrogram encoder, a recognition accuracy of 98.27% was achieved. 3. This embodiment compares the MAP framework's performance in recognizing calf sounds using different masking methods and masking ratios, as shown in Table 2. The masking methods are block masking, time-domain masking only, frequency-domain masking only, and time-domain plus frequency-domain masking, with masking ratios of 0.45, 0.6, and 0.75.
[0047] Table 2 .
[0048] Example 2: Following the method for identifying abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning provided in Example 1, this example provides a device for identifying abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning. The device provided in this example can implement the method for identifying abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning in Example 1. This device can be implemented through software, hardware, or a combination of both. For ease of description, this example is described by dividing the functions into various units. Of course, in implementation, the functions of each unit can be implemented in one or more software and / or hardware components. For example, the device may include integrated or separate functional modules or units to execute the corresponding steps in the methods of Example 1. Since the device in this example is basically similar to the method example, the description process of this example is relatively simple. For relevant details, please refer to the description in Example 1. The example of the device for identifying abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning provided by this invention is merely illustrative.
[0049] Specifically, the present invention provides a device for on-site identification of abnormal sounds in livestock, comprising: An audio collection unit is configured to collect audio from livestock inside livestock sheds; The spectrum conversion unit is configured to convert livestock audio into a Mel spectrogram. An abnormal sound prediction unit is configured to input a Mel spectrogram into a livestock abnormal sound recognition model for abnormal sound recognition. The training of the livestock abnormal sound recognition model includes: The Mel spectrogram is masked based on the multimodal audio pre-training framework. The masked Mel spectrogram and the corresponding original audio waveform are then input into the multimodal audio pre-training framework. The multimodal audio pre-training framework is pre-trained using self-supervised learning to obtain the Mel spectrogram encoder. Set up a downstream task fine-tuning model, embed a dense connection adapter in the Mel spectrogram encoder to fine-tune the spectrogram features for specific tasks to obtain classification features; Classification features are used for classification and identification via a linear classification head.
[0050] Example 3: This example provides an electronic device corresponding to the method for on-site identification of abnormal livestock sounds based on multimodal sound feature pre-training and dense connection adapter fine-tuning provided in Example 1. The electronic device can be an electronic device for the client, such as a mobile phone, laptop, tablet computer, desktop computer, etc., to execute the method of Example 1.
[0051] like Figure 4As shown, the electronic device includes a processor, a memory, a communication interface, and a bus. The processor, memory, and communication interface are connected via the bus to enable communication between them. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the method of Embodiment 1. The implementation principle and technical effects are similar to those of Embodiment 1, and will not be repeated here. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computing device to which the present invention is applied. A specific computing device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0052] In a preferred embodiment, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), and optical discs.
[0053] In a preferred embodiment, the processor can be any type of general-purpose processor such as a central processing unit (CPU) or a digital signal processor (DSP), and is not limited thereto.
[0054] Example 4: This example provides a computer-readable storage medium for storing one or more programs, the one or more programs including computer instructions, which, when executed by a computer, cause the computer to perform the method provided in Example 1 above.
[0055] In a preferred embodiment, the computer-readable storage medium may be a tangible device for holding and storing instructions executable, such as, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. The computer-readable storage medium stores computer program instructions that cause a computer to perform the method provided in Embodiment 1 above.
[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0059] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In the description of this specification, the terms "a preferred embodiment," "furthermore," "specifically," "in this embodiment," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments in this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for on-site identification of abnormal sounds in livestock, characterized in that, include: Collect audio recordings of livestock inside livestock sheds; Convert livestock audio into Mel spectrograms; The Mel spectrogram is input into the livestock abnormal sound recognition model for abnormal sound identification. The training of the livestock abnormal sound recognition model includes: The Mel spectrogram is masked based on the multimodal audio pre-training framework. The masked Mel spectrogram and the corresponding original audio waveform are then input into the multimodal audio pre-training framework. The multimodal audio pre-training framework is pre-trained using self-supervised learning to obtain the Mel spectrogram encoder. Set up a downstream task fine-tuning model, embed a dense connection adapter in the Mel spectrogram encoder to fine-tune the spectrogram features for specific tasks to obtain classification features; The classification features are used for classification and recognition through a linear classification head, where: The specific training process of the Mel spectrogram encoder is as follows: Construct a pre-trained dataset; Convert the raw audio waveform data of the pre-trained dataset into Mel spectrograms; A masked Mel spectrogram is formed by performing a masking operation on the Mel spectrogram. The masked Mel spectrogram and the original audio waveform are then input into the Mel spectrogram encoder and the waveform encoder, respectively, to extract the spectrogram features and the audio waveform features. The spectrogram features and audio waveform features are input into the modal fusion layer, and then the output fused features are input into the Mel spectrogram decoder. The Mel spectrogram decoder performs masked Mel spectrogram reconstruction to obtain the reconstructed Mel spectrogram. The process is as follows: the spectrogram features and audio waveform features are fused into modal features through the waveform-Mel spectrogram cross attention mechanism, and the fused features are sent into the Mel spectrogram decoder to reconstruct the masked Mel spectrogram. By calculating the loss value between the reconstructed Mel spectrogram and the original Mel spectrogram, this loss value is used as the training parameter for the multimodal audio pre-training framework, thus completing the training of the Mel spectrogram encoder.
2. The method for on-site identification of abnormal sounds in livestock according to claim 1, characterized in that, The fine-tuning training process for the densely connected adapter is as follows: Build a fine-tuning dataset; Label various vocalization categories in the fine-tuning dataset, including coughing, normal humming, excretion sounds, and other sounds from livestock sheds; Convert the audio waveform data in the fine-tuning dataset into Mel spectrograms; The Mel spectrogram is fed into the downstream task to fine-tune the model. After each layer of attention mechanism in the obtained Mel spectrogram encoder, an adapter module of dense connection adapter is embedded. All embedded adapter modules are connected in a dense connection manner to fine-tune the spectrogram features for abnormal sound recognition and obtain classification features. During the downstream task fine-tuning, all parameters of the Mel spectrogram encoder are frozen, and only the parameters of the dense connection adapter are trained.
3. The method for on-site identification of abnormal sounds in livestock according to claim 2, characterized in that, Each layer's adapter can not only process the features of the current layer, but also receive features from all previous layers' adapters, forming dense features. The implementation logic is as follows: in, as well as This is the output of the previous adapter layers. These features are concatenated in a dimensional manner and then passed through a linear projection layer. Perform dimensionality reduction. As the output of the current layer dense adapter i Indicates the current layer.
4. The method for on-site identification of abnormal sounds in livestock according to claim 3, characterized in that, The process of obtaining the pre-training dataset and the fine-tuning dataset is as follows: The ZoomH2n stereo recorder with a built-in microphone is used to capture audio from livestock. The sliding window method was used to extract segments from the collected raw audio. Since the duration of each type of livestock sound segment collected did not exceed 1 second, the sliding window length was set to 1 second and the step size was 1 second. Finally, several fixed-length and non-overlapping audio segments were extracted and used as the pre-training dataset. The fine-tuned dataset used Adobe Audition software to categorize the collected audio.
5. A livestock abnormal sound on-site identification device for the livestock abnormal sound on-site identification method according to any one of claims 1-4, characterized in that, include: An audio collection unit is configured to collect audio from livestock inside livestock sheds; The spectrum conversion unit is configured to convert livestock audio into a Mel spectrogram. An abnormal sound prediction unit is configured to input a Mel spectrogram into a livestock abnormal sound recognition model for abnormal sound recognition. The training of the livestock abnormal sound recognition model includes: The Mel spectrogram is masked based on the multimodal audio pre-training framework. The masked Mel spectrogram and the corresponding original audio waveform are then input into the multimodal audio pre-training framework. The multimodal audio pre-training framework is pre-trained using self-supervised learning to obtain the Mel spectrogram encoder. Set up a downstream task fine-tuning model, embed a dense connection adapter in the Mel spectrogram encoder to fine-tune the spectrogram features for specific tasks to obtain classification features; Classification features are used for classification and identification via a linear classification head.
6. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the method according to any one of claims 1-4.
7. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include computer instructions for causing a computer to perform the method according to any one of claims 1-4.