Voiceprint and image multi-mode cat and dog intention recognition method, device and equipment

Abstract

The invention relates to sound acquisition equipment worn on cats and dogs, which transmits acquired sound data to a user terminal through Wi-Fi, and improves the accuracy of cat and dog intention recognition by combining video data acquired by a mobile phone camera and utilizing an optimized data acquisition process and an improved classification algorithm. Through the technical means of the multi-language pre-training model, the steering quantity module, the pre-training module and the like, accurate classification of cat and dog barks is realized, so that people are helped to better understand and cope with behaviors of pets.

Description

Technical Field

[0001] This application relates to the fields of big data classification and animal health protection. Specifically, it concerns a device worn by cats and dogs that collects sound, transmits it to a user terminal via Wi-Fi, and generates intent classification results. Simultaneously, it can utilize a mobile phone camera to capture video and audio, improving the accuracy of intent recognition for cats and dogs through dual recognition of images and audio tracks. Background Technology

[0002] Existing technologies utilize the unsupervised learning technique of wav2vec to recognize various vocalizations of dogs. wav2vec, proposed by Facebook AI Research, is an unsupervised pre-trained model for speech recognition. Its core idea is to learn useful feature representations from raw audio waveforms. This technique can significantly improve performance in speech recognition tasks, such as ASR (Automatic Speech-to-Text) tasks.

[0003] In this specific application scenario, the technology utilizes the wav2vec model to classify dog ​​barks, identifying the types of barks in different contexts. This not only helps in understanding the dog's emotional state and intentions but also assists owners or other personnel in better responding to dog behavior. However, the overall recognition accuracy achieved through wav2vec is only 60%.

[0004] Therefore, how to enhance the accuracy of intent recognition through AI and extend it to other animals such as cats is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to more accurately identify the intentions represented by cat and dog sounds. This is achieved by optimizing the data collection process, using supervised classification algorithms, and utilizing video-assisted recognition to improve classification accuracy.

[0006] To achieve the above objectives, the present invention discloses the following technical solution:

[0007] I. Optimize the data collection process

[0008] 1.1 Analysis of cat and dog intention categories, including states such as: drinking water, eating, panting, highly aggressive meowing at strangers, normal meowing at strangers, meowing because the owner is being attacked, negative purring when strangers are present, sad or anxious meowing, positive meowing, and dozens of other categories. Among these, cats are non-social animals and usually express their intentions and emotions through body language (rather than vocalizations), making video analysis more effective in determining their behavior.

[0009] 1.2 Interactive Annotation: Cat and dog barks learn a lot of human voices. In order to clearly determine whether the classification results are true, the classification model is based on a large model structure and provides the ability to continue writing and generate. The synthesized speech results are fed back to the animals to observe whether there is an interactive response, which helps to determine whether the results are valid.

[0010] 1.3 Cat and dog barks have certain differences in variety, similar to human dialects, but the tone of emotion is roughly the same across breeds. In the initial data preparation, we considered collecting multiple pet voiceprint samples and publicly releasing the data to create training and test sets.

[0011] II. Optimizing the Classifier

[0012] 2.1 Using Multilingual Pre-trained Models: Research indicates that using pre-trained human voice models can enhance the classification of cat and dog barks because cats and dogs sometimes mimic certain vocal characteristics of their owners. Technically, a model already pre-trained for human voices can be used to initialize a new model designed to distinguish between cat and dog barks. A large amount of human voice data is collected to pre-train a model, which can be based on speech recognition or speech synthesis tasks. The method for mixing datasets is as follows: Training is performed approximately five times using publicly available human voice data, stopping when the mean squared error between newly added voiceprint data and the closest original data does not exceed 0.3, and the pre-trained model loss no longer decreases (or is less than 0.01). Then, a data set with human voice: dog bark: cat meow = 1:2:2 is added, and the above process is repeated until the loss no longer decreases.

[0013] 2.2 Transformation Module: The wav2vec vector model is retrained using mixed data of human voices and cat / dog barks to convert voiceprints into vectors. Mel-frequency cepstral coefficients (MFCCs) are used before the feature extraction wav2vec neural network. The training objective of the wav2vec model is to maximize the conditional probability of predicting the uncovered portion. Let x be the original audio signal, f(x) be the feature vector sequence output by the feature extractor, m be the masking operation, g be the encoder, and p be the predictor, then the training objective of the model can be expressed as:

[0014]

[0015] Where L is the loss function, typically mean squared error (MSE) or another suitable loss function. m(f(x)) is the sequence of masked feature vectors. g is the encoder, which maps the masked feature sequence to the latent space. p is the predictor, which attempts to predict the unmasked feature vectors.

[0016] 2.3 Pre-training Module: The prepared vectors are input into a large model based on a decoder architecture for unsupervised pre-training to form a large voiceprint model. The specific process is as follows: 1) Select a decoder-based model architecture, which is good at processing sequential data and can capture the temporal dependencies in audio signals. 2) Add a self-attention mechanism: Use a self-attention mechanism to process the input vector, enabling the model to focus on important parts of the sequence and learn useful information from them. 3) Perform self-supervised learning: Use Masked Language Modeling (MLM): Apply a masking strategy to the audio vectors, i.e., randomly occlude a portion of the vector, allowing the model to learn to predict the occluded part. 4) Prediction objective: The model's objective is to predict the occluded vector portion as accurately as possible.

[0017] 2.4 Classification Module: Fine-tuning the pre-trained model to create a vocalization classification model. The fine-tuning process includes: 1) Adding a classification layer: Adding one or more fully connected layers on top of the pre-trained model so that the model can learn the mapping from feature representations to category labels. 2) Preparing labeled data: Obtaining a labeled dataset. The labels should be the specific category to which the audio segment belongs, such as "angry cat sound," "stressed dog sound," or "happy conversation between people." 3) Defining the loss function: Choosing cross-entropy as the loss function. 4) Fine-tuning the model: Training the model using labeled data and adjusting the parameters in the pre-trained model to minimize classification error.

[0018] 2.5 Video Assistance Module: By introducing a visual encoder (ViT-bigG) and a position-aware visual language adapter, the large model can receive image input and generate relevant text descriptions. Open-source samples of cat and dog images converted to text are used for training to optimize the model's understanding of specific video types, ensuring it can accurately convert video content into natural language descriptions. The video file is segmented into a series of consecutive frames, and each frame is processed by the large model to obtain a corresponding text description. For the entire video, a strategy is needed to integrate the single-frame descriptions into a coherent video narrative. We use a time-series model (such as LSTM) to capture the dynamic changes between frames, thereby generating more coherent text descriptions and generating category labels and scores. Assuming y(t) is the output vector at time step t, a weighted average is used to calculate the sum of the outputs. The value w(t) is determined by the volume of sound. The collected decibel data is uniformized and used as a weighting coefficient.

[0019] To improve classification accuracy, a method using loss function fusion is adopted to supervise the adjustment of weight distribution among different feature sources (video description and voiceprint features).

[0020] III. Carrier Equipment

[0021] The carrier is an Internet of Things (IoT) device, specifically a voice recorder with Wi-Fi functionality. It can be attached to a pet to collect ambient sounds and transmit data in real time. Additionally, it includes a backend system that receives this data and categorizes and interprets the sounds. The system also allows users to record video or audio using their mobile phones, and this data can also be categorized.

[0022] The modules or steps described above in this application can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by the computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0023] Although this application has been described in detail above with general descriptions and specific embodiments, some modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of this application fall within the scope of protection claimed in this application.

Claims

1. A system for cat and dog intent recognition, comprising: Wearable devices used to collect the sound data of cats and dogs and transmit it via Wi-Fi; The user terminal device receives the audio data sent by the wearable device and further collects video data through the mobile phone camera; The processing unit is used to process the audio data and video data, and generate intent classification results through supervised classification algorithms and video-assisted recognition technology.

2. The system according to claim 1, characterized in that, The processing unit further includes: The data acquisition module was optimized to analyze the different vocalizations of cats and dogs, and the effectiveness of the classification results was verified through interactive annotation methods. Multilingual pre-trained models are used to enhance the classification of cat and dog barks; The steering module is used to retrain the wav2vec model with data from a mixture of human voices and cat / dog barks to convert voiceprints into vectors. The pre-training module is used to form a large voiceprint model, which is based on a decoder architecture and includes a self-attention mechanism. Pre-training is completed through self-supervised learning.

3. The system according to claim 2, characterized in that, The steering module includes: A feature extractor is used to extract feature vectors from a raw audio signal. An encoder is used to map a masked sequence of features to a latent space; A predictor is used to predict the feature vectors that are not masked.

4. The system according to claim 2, characterized in that, The pre-training module applies a masking strategy to the audio vectors through masking language modeling, enabling the model to learn to predict the occluded parts.

Images