Multi-modal data-based mouse species classification method and system

Abstract

The invention relates to the technical field of data processing, in particular to a mouse type classification method and system based on multi-modal data, and the method comprises the steps: obtaining mouse type video data, gait video data and audio data; video image data and gait video image data are processed by adopting a deep convolutional neural network optimized aiming at mouse features to obtain a video feature image set and a gait feature image set respectively, high-quality audio data are extracted to obtain a frequency spectrum image set, multi-modal feature fusion is performed on the three image sets to obtain a feature representation set, and the feature representation set is subjected to multi-modal feature fusion; and combining time sequence modeling and global information modeling to obtain a final multi-modal feature vector, classifying the mouse species in the video based on the multi-modal feature vector, and judging the category of the mouse species. According to the mouse species classification method provided by the invention, richer feature information can be provided, the conditions of false identification and missed identification are reduced, real-time monitoring and classification of mouse species are realized, and the method is of great significance to accurate and efficient identification of harmful rats at customs ports.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a mouse species classification method and system based on multimodal data. Background Art

[0002] Accurate and rapid identification of pest species is crucial for strict monitoring and control at customs ports.

[0003] However, traditional methods for identifying rats (e.g., CN221709924U, published on September 13, 2024) and single modality methods (e.g., CN104320610B, published on August 10, 2018) have many difficulties. First, different types of rats may be very similar in morphology, and it is difficult to accurately distinguish them by appearance alone. Secondly, the behaviors of rats are diverse and complex, which makes behavior-based identification methods challenging. Furthermore, in actual environments, factors such as lighting, occlusion, and background noise may seriously affect the accuracy of identification. Therefore, traditional identification methods are often inefficient and easily limited by the experience and knowledge level of the identification personnel, and there is a high risk of misjudgment.

[0004] Therefore, providing a highly efficient and accurate method for automatically identifying pest rodents is of great significance for inspection at customs ports and reducing the manual burden. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention aims to provide a method and system for classifying rat species based on multimodal data. The method integrates information from different modalities, such as time series information in the video, dynamic behavior characteristics, static image characteristics, gait characteristics, and sound characteristics, etc., to more comprehensively capture the characteristic information of rat species. Automatic learning and classification of multimodal features through deep learning algorithms can significantly improve the accuracy and robustness of recognition. By achieving 24-hour uninterrupted monitoring of customs ports, customs departments can promptly detect potential risks of rodent invasion and quickly take corresponding measures to prevent and control them.

[0006] In a first aspect, the present invention provides a method for classifying rat species based on multimodal data, the method comprising the following steps: S1: Acquire mouse species video data, gait video data and audio data, wherein the video data includes video content information and video frame images, and the gait video data includes gait video content information and gait video frame images; S2: preprocessing the video frame image, gait video frame image and audio data to obtain high-quality video image frame data, gait image frame data and audio data; S3: extracting features of high-quality video image frame data using a deep convolutional neural network optimized for rodent features to obtain a video feature atlas F; extracting features of high-quality gait image frame data using a deep convolutional neural network optimized for rodent features to obtain a gait feature atlas B; extracting features of high-quality audio data to obtain a spectrum atlas M; S4: Perform multimodal feature fusion on the video feature atlas F, the gait feature atlas B and the spectrum atlas M to obtain the feature representation set FBM; S5: Use long short-term memory network to process the feature representation set FBM to obtain the time series feature vector; S6: Obtain a global information feature vector, obtain an embedding vector TR by linearly projecting the feature representation set FBM, and input the embedding vector TR into an encoder trained for the appearance and sound characteristics of mice to generate a global information feature vector; S7: feature fusion, using weighted summation to fuse the time series feature vector and the global information feature vector to obtain the mouse multimodal feature vector; S8: Classify the species of mice based on their multimodal feature vectors and determine the type of the species.

[0007] Preferably, the video content information includes frame rate.

[0008] Preferably, the preprocessing of the video frame images, gait video frame images and audio data in S2 includes: S201: unifying the frame rates of video frame images according to video content information and a reference frame rate to obtain a set of video images with unified frame rates; S202: unifying the frame rates of gait video frame images according to gait video content information and a reference frame rate to obtain a set of gait images with unified frame rates; S203: enhancing the video image set with unified frame rates to obtain high-quality video image frame data; S204: enhancing the gait image set with unified frame rates to obtain high-quality gait image frame data; S205: Standardizing the audio data; S206: Perform noise reduction and enhancement processing on the standardized audio data to obtain high-quality audio data.

[0009] More preferably, the reference frame rate in S201 is 20fps.

[0010] More preferably, the enhancement in S203 includes spatial domain enhancement, frequency domain enhancement or super-resolution reconstruction algorithm.

[0011] More preferably, the standardization processing of the audio data in S205 includes implementing it by using the preprocessing.scale() function.

[0012] More preferably, the noise reduction in S206 includes spectral subtraction, least mean square method or Wiener filtering, and the enhancement includes distortion, frequency domain masking, high pitch and speed adjustment.

[0013] Preferably, obtaining the spectrum atlas M in S3 includes: S301: extracting high-quality audio data features using a logarithmic Mel-spectrogram to obtain a spectrum graph M1; S302: using Mel frequency cepstral coefficients to process high-quality audio data to obtain a spectrum graph M2; S303: Process the spectrum graph M1 using a second-order difference method to obtain a spectrum graph M3; S304: Unify the dimensions of the spectrogram M1, the spectrogram M2, and the spectrogram M3 and concatenate them to obtain a feature fusion spectrogram M4; S305: Crop the feature fusion spectrum graph M4 according to the time interval of the video feature atlas F to obtain the spectrum atlas M.

[0014] More preferably, the Mel-frequency cepstral coefficients in S302 process high-quality audio data including: S3021: Preprocessing high-quality audio data; S3022: Perform fast Fourier transform on the preprocessed audio data to obtain a linear spectrum; S3023: Mel filter bank, mapping the linear spectrum to the Mel scale to obtain the Mel spectrum; S3024: perform logarithmic operation on the Mel spectrum; S3025: Perform discrete cosine transform on the logarithmic Mel spectrum to obtain a spectrum graph M2.

[0015] More preferably, the splicing in S304 is concat splicing.

[0016] Preferably, the multimodal feature fusion in S4 includes: S401: performing dimension unification and normalization processing on the video feature atlas F, the gait feature atlas B, and the spectrum atlas M; More preferably, the dimension unification and normalization processing in S401 includes respectively unifying the dimensions of the feature atlas F, the gait feature atlas B and the spectrum atlas M by using linear changes, and eliminating the influence of the dimension and the numerical range by using normalization.

[0017] S402: Multimodal feature fusion is performed through the attention network and weighted fusion method to obtain a unified feature representation set FBM.

[0018] More preferably, the S402 is to calculate the weight of each modality through the attention network, perform weighted fusion on the multi-modality according to the weight, obtain a fused feature vector, and aggregate the fused feature vector into a unified feature representation set FBM.

[0019] More preferably, the S402 uses a scoring function of scaled dot product attention to achieve weighted fusion.

[0020] Preferably, the S5 further includes performing average pooling on the hidden state set of each time step structure obtained by processing the long short-term memory network to obtain a time series feature vector.

[0021] More preferably, the S5 specifically includes: S501: Sequence input, input the feature representation set FM into the long short-term memory network, and each feature in the feature representation set corresponds to a time step structure; S502: Information transmission, in each time step structure, the long short-term memory network will update its internal memory cells and hidden state through the input gate, forget gate and output gate according to the current input and the hidden state of the previous time step; S503: Hidden state output. After each time step structure is completed, a hidden state is obtained. The hidden states of all time steps are obtained as a hidden state set YC. S504: Aggregate time steps and perform average pooling operation on the hidden state set YC to obtain a time series feature vector.

[0022] Preferably, the encoder in S6 trained for the appearance and sound characteristics of mice includes a Transformer encoder with an integrated self-attention mechanism.

[0023] Preferably, the classifying of rat species based on the rat multimodal feature vector in S8 includes: S801: Input the mouse multimodal feature vector into the classifier to obtain a classification array with a length equal to the mouse species confidence score; S802: Determine the rat species category, and the category corresponding to the maximum confidence score is the pest rat.

[0024] More preferably, the classifier includes a decision tree classifier, a selection tree classifier, a support vector machine, a K nearest neighbor algorithm, a Softmax function or a logistic regression method.

[0025] Preferably, the classifier determines the category of the fused multimodal feature vector through a Softmax function.

[0026] A second aspect of the present invention provides a rat species classification system based on multimodal data, the system comprising: T1: a module for inputting mouse species video data, gait video data and audio data, wherein the video data includes video content information and video frame images, and the gait video data includes gait video content information and gait video frame images; T2: Data preprocessing module, used for preprocessing video frame images, gait video data and audio data to obtain high-quality video image frame data, gait image frame data and high-quality audio data; T3: Feature extraction module, using a deep convolutional neural network optimized for rodent features to extract features of high-quality video image frame data, obtaining a video feature atlas F; using a deep convolutional neural network optimized for rodent features to extract features of high-quality gait image frame data, obtaining a gait feature atlas B; extracting features of high-quality audio data, obtaining a spectrum atlas M; T4: Multimodal feature fusion module, which performs multimodal feature fusion on the video feature atlas F, gait feature atlas B and spectrum atlas M to obtain the feature representation set FBM; T5: Time series modeling module, which uses long short-term memory network to process the feature representation set FBM to obtain the time series feature vector; T6: Global information modeling module, which performs linear projection on the feature representation set FBM to obtain the embedding vector TR, and inputs the embedding vector TR into the encoder trained for the appearance and sound characteristics of mice to generate the global information feature vector input; T7: Feature fusion module, which uses weighted summation to fuse the time series feature vector and the global information feature vector to obtain the mouse multimodal feature vector; T8: Rat species classification module, which classifies rat species based on the multimodal feature vectors of rats and determines the type of rat species.

[0027] Preferably, the video content information in the T1 mouse species video data and audio data input module includes a frame rate.

[0028] Preferably, the T5 time series modeling module includes averaging the hidden states of each time step structure obtained by long short-term memory network processing to obtain a time series feature vector.

[0029] Preferably, the encoder trained for the appearance and sound characteristics of mice in the T6 global information modeling module includes a Transformer encoder with an integrated self-attention mechanism.

[0030] The beneficial effects of the present invention are: (1) The present invention is based on intelligent monitoring equipment at customs ports, multimodal video classification and artificial intelligence technology, and comprehensively considers the temporal information, dynamic behavior, appearance characteristics and sound of pests in the video collected by port equipment to identify pests more accurately and efficiently. Compared with the traditional single-modal recognition method, multimodal fusion can provide richer feature information and reduce the cases of misidentification and missed recognition.

[0031] (2) The present invention first fuses the feature atlas F, the gait feature atlas B and the spectrum atlas M to obtain the feature representation set FBM, and then fuses the feature representation set FBM again after temporal feature extraction and global feature extraction. The advantages of the two fusions include: ① Enhanced feature complementarity. The first feature fusion fuses the feature atlas F, the gait feature atlas B and the spectrum atlas M, which can preliminarily combine feature information of different dimensions to form a more comprehensive feature representation. This fusion helps to capture the complex relationships implicit in the data. The second fusion extracts temporal features and global features from the feature representation set FBM after the first fusion, which can further mine the temporal dependency and global structural information in the data. Fusion of these features again can further enhance the complementarity of the features and improve the model's ability to represent complex data. ② Improved model generalization ability. Through multi-level feature fusion, the model can learn more robust and generalized feature representations. The first fusion provides the model with a basic feature combination, and the second fusion performs a deeper level of feature abstraction and integration on this basis, which helps the model better adapt to different data distributions and task requirements.

[0032] (3) By real-time monitoring and classification of rodent activities in the video, the customs department can promptly detect potential rodent invasion or epidemic risks, and immediately take corresponding measures to prevent and control them, effectively avoiding the spread of the epidemic and the occurrence of harm. At the same time, the present invention reduces the need for manual inspection and identification at customs ports through automated video analysis and processing, saving labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a flow chart of the mouse species classification method based on multimodal data of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below in conjunction with specific embodiments. It is to be understood that the specific embodiments described herein are represented by way of example and are not intended to be limiting of the present invention. The main features of the present invention may be used in various embodiments without departing from the scope of the present invention. Those skilled in the art will appreciate or be able to confirm that, using only routine experiments, many equivalents can be applied to the specific steps described herein. These equivalents are considered to be within the scope of the present invention and are covered by the claims.

[0035] Example 1: A method for classifying mouse species based on multimodal data The present invention takes the classification of rat species videos captured by customs port monitoring equipment as an example, combined with the flow chart ( Figure 1 ) Describe the technical solution of this application in detail.

[0036] S1: Obtaining mouse species video data, gait video data and audio data; The rat species video data, gait video data and audio data include rat species video data, gait video data and audio data of the customs port intelligent monitoring equipment.

[0037] The video data includes video content information and video frame images, and the gait video data includes gait video content information and gait video frame images; The video content information includes frame rate.

[0038] S2: Preprocessing of video frame images, gait video frame images and audio data: S201: unifying the frame rates of video frame images according to video content information and a reference frame rate to obtain a set of video images with unified frame rates; Specifically, the base frame rate is 20fps. (1) If the frame rate is greater than 20fps, frame extraction is required to make it reach the base frame rate of 20fps; (2) If it is equal to 20fps, no processing is performed; (3) If the frame rate is less than 20fps, interpolation is required to make it reach the base frame rate of 20fps. The interpolation method can be achieved by frame duplication, frame blending and optical flow method.

[0039] S202: unifying the frame rates of the gait video frame images according to the gait video content information and the reference frame rate to obtain a set of gait images after the frame rates have been unified; the method for unifying the frame rates is the same as that in S201; S203: enhancing the video image set with unified frame rates to obtain high-quality video image frame data; Specifically, image enhancement algorithms are used, such as spatial domain enhancement, frequency domain enhancement, and super-resolution reconstruction algorithms. The enhancement threshold and parameters vary according to the algorithm. In terms of effect, the enhanced image should maintain the original information and improve the visual effect while avoiding the introduction of obvious distortion or noise.

[0040] S204: enhancing the gait image set after the frame rate is unified to obtain high-quality gait image frame data; the enhancement method is the same as S203; S205: Standardizing the audio data; Specifically, the normalization of audio data refers to the unified volume adjustment of audio signals to ensure that the volume of different audios remains consistent. The purpose is to maintain the dynamic range of the audio so that the audio signal can be accurately reproduced on different devices and platforms. Normalization can eliminate the volume differences between audios and provide a more consistent listening experience.

[0041] The standardization of audio data depends on the specific application scenario, the requirements of the target device and platform, and the characteristics of the audio itself. It can be achieved by peak normalization, loudness normalization, fast Fourier transform or short-time Fourier transform.

[0042] S206: Perform noise reduction and enhancement processing on the standardized audio data to obtain high-quality audio data. The noise reduction includes spectral subtraction, least mean square method or Wiener filtering, and the enhancement includes time distortion, frequency domain masking, high pitch and speed adjustment.

[0043] S3: A deep convolutional neural network optimized for mouse characteristics is used to extract features of high-quality video image frame data to obtain a video feature atlas F{f1,f2,…}; a deep convolutional neural network optimized for mouse characteristics is used to extract features of high-quality gait image frame data to obtain a gait feature atlas B{b1,b2,…}; and features of high-quality audio data are extracted to obtain a spectrum atlas M{m1,m2,…}.

[0044] Specifically, deep convolutional neural networks include VGG, ResNet, MobileNet, and EfficientNet. ResNet is used as the main feature extraction model of this model.

[0045] The size and number of convolution kernels in ResNet vary depending on the different parts of the network. Generally speaking, the size of convolution kernels in ResNet networks is mainly 3x3, because smaller convolution kernels can reduce the amount of calculation and capture a wider range of contextual information by stacking multiple convolution layers.

[0046] In terms of the number of convolution kernels, as the network goes deeper, the number of convolution kernels usually increases gradually to extract more complex features. Specifically, each residual block of ResNet may contain multiple convolution layers, and each convolution layer has its own number of convolution kernels.

[0047] In a specific embodiment of the present invention, feature extraction is implemented using ResNet-50: ResNet-50 includes multiple residual blocks, each of which includes multiple convolutional layers; the convolutional layer in the residual block usually uses a 3x3 convolution kernel. As the network goes deeper, the number of convolution kernels gradually increases, and the downsampling operation is mainly implemented through a convolutional layer with a step size of 2.

[0048] Extracting the features of audio data and obtaining the spectrum atlas M specifically includes: S301: extracting features of high-quality audio data using a logarithmic Mel-spectrogram to obtain a spectrum graph M1; S302: using Mel frequency cepstral coefficients to process high-quality audio data to obtain a spectrum graph M2; S303: Process the spectrum graph M1 using a second-order difference method to obtain a spectrum graph M3; S304: unifying the dimensions of the spectrum graph M1, the spectrum graph M2 and the spectrum graph M3 and splicing them to obtain a feature fusion spectrum graph M4; the splicing includes using concat splicing; S305: Spectrum frame rate conversion: The spectrogram M4 is cropped according to the time interval of the video feature atlas F to obtain the spectrogram atlas M{m1,m2,…}; The time interval needs to be determined according to the reference frame rate of S201.

[0049] S4: Perform multimodal feature fusion on the video feature atlas F and the spectrum atlas M to obtain the feature representation set FM: S401: Feature preprocessing: linearly change the dimensions of the feature atlas F{f1,f2,…}, the gait feature atlas B{b1,b2,…} and the spectrum atlas M{m1,m2,…}, and normalize them to eliminate the influence of the dimension and value range, and obtain the updated F{f1,f2,…}, B{b1,b2,…} and M{m1,m2,…}; S402: Perform feature fusion through attention network and weighted fusion method to obtain a unified feature representation set FBM.

[0050] In a specific example of the present invention, S402 is to use an attention network to calculate the weight of each modality, perform weighted fusion according to the weight, and obtain a fused feature vector; the fused feature vector is aggregated into a unified feature representation set FBM{fbm1,fbm2,…}; Scaled dot product attention is a more computationally efficient attention mechanism. It directly calculates the dot product of the query and the key, but in order to prevent the dot product result from being too large and causing the gradient of the softmax function to disappear, the dot product result is usually divided by a scaling factor. The scoring function of scaled dot product attention can be expressed as:

[0051] Q is the query matrix, with dimension , where n is the number of queries, is the dimension of the query and key.

[0052] K is the key matrix, with dimension , where m is the number of keys, is the dimension of the key.

[0053] V is the value matrix, with dimension , where m is the number of values, is the dimension of the value.

[0054] It is the dot product operation of the query and the key, and the result is an n×m matrix representing the relevance of the query to each key.

[0055] It is a scaling factor used to prevent the dot product result from being too large, causing the gradient of the Softmax function to disappear.

[0056] Softmax is an operation that normalizes the dot product result, usually calculated along each row to ensure that the sum of all attention weights is 1.

[0057] The final result is a weighted sum, where the weights are determined by the attention scores of the Softmax output, and the output dimension is .

[0058] This attention mechanism is simpler to implement and is often more computationally efficient than additive attention because it directly computes the dot product.

[0059] In the attention mechanism, the masked Softmax operation is an important step for processing sequences of variable length or sequences containing invalid tokens. Masked Softmax sets the value of invalid positions to a very small negative number before softmax calculation, thus ensuring that the output of these positions after softmax is close to 0, and thus they are effectively ignored.

[0060] S5: Use the long short-term memory network to process the feature representation set FBM to obtain the time series feature vector: S501: Sequence input: Input the feature representation set FBM {fbm1, fbm2, ...} into the long short-term memory network, where each feature {fbm1, fbm2, ...} corresponds to a time step structure; S502: Information transmission: In each time step, the LSTM network updates its internal memory cells and hidden state through the input gate, forget gate and output gate according to the current input and the hidden state of the previous time step; S503: Hidden state output: After each time step, a hidden state is obtained, and the hidden states of all time steps are obtained as a hidden state set YC{yc1,yc2,…}; S504: Aggregate time steps: Perform an average pooling operation on the output hidden state sets of all time steps to obtain a time series feature vector.

[0061] Long short-term memory network (LSTM) is a variant of recurrent neural network (RNN), which is specially used to process sequence data and capture long-term dependencies. Each unit in LSTM has a memory unit and three gates (input gate, forget gate and output gate), which can effectively handle information transfer and forgetting in the sequence. The input gate determines the degree of influence of the input of the current time step on the memory unit, the forget gate determines whether to forget information from the memory unit, and the output gate determines the degree of influence of the memory unit state of the current time step on the output. In the processing of time steps, the LSTM model processes the input sequence time step by time step. For each time step, the LSTM unit updates the hidden state and memory unit of the current time step according to the current input, the hidden state and memory unit of the previous time step. After processing the entire sequence, you can choose to use the hidden state of the last time step as the output of the model according to your needs, or you can use the hidden state of all time steps for further processing. In a specific embodiment of the present application, the hidden state of all time steps is output.

[0062] S6: Get the global information feature vector: By linearly projecting the feature representation set FBM{fbm1,fbm2,…}, the embedding vector TR is obtained, and the embedding vector is input into the encoder trained for the appearance and sound characteristics of mice to generate a global information feature vector.

[0063] In a specific example of the present invention, the encoder is a Transformer encoder with an integrated self-attention mechanism. The encoder is a device or module that converts input data into data in a specific format. It is composed of multiple layers of encoders stacked together, each layer consisting of a multi-head self-attention mechanism and a feedforward neural network. In each layer, the input data is subjected to feature extraction and representation by the self-attention mechanism, and then subjected to nonlinear transformation of the features by the feedforward neural network. After multiple layers of stacking, features at different levels can be learned.

[0064] In a specific example of the present invention, the Transformer encoder of the self-attention mechanism uses 8 attention heads, each of which independently calculates the attention weight, and concatenates the results to obtain the final output through linear transformation. In the Transformer, the hidden unit dimension d_model is set to 512, which means that each vector in the model (including query vectors, key vectors, and value vectors) has 512 dimensions.

[0065] The commonly used activation function in the feedforward neural network is ReLU (Rectified Linear Unit), which can set all negative values ​​to 0 and keep positive values ​​unchanged. The ReLU function is simple and computationally efficient, which helps the model converge quickly. The hidden layer dimension is 2048.

[0066] S7: Feature Fusion: The time series feature vector of S504 and the global information feature vector of S602 are fused by weighted summation to obtain a multimodal feature vector of mice.

[0067] S8: Classify the species of mice based on the multimodal feature vector of mice and determine the type of mice: In a specific example of the present invention, step S8 specifically includes: S801: Input the mouse multimodal feature vector into the classifier to obtain a classification array with a length equal to the confidence score of the mouse species.

[0068] S802: Determine the rat species category, and the category corresponding to the maximum confidence score is the pest rat.

[0069] A classifier is a machine learning model or algorithm that is used to classify input data into different categories or labels. A classifier is part of supervised learning and performs classification tasks by learning the mapping relationship from input data to output categories. In the technical solution of the present application, the labels of the classifier include that the rat species in the video is a pest (first label), and that the rat species in the video is not a pest (second label).

[0070] Specifically, in a specific example of the present invention, the S8 mouse species classification module determines to which classification label the fused multimodal feature vector belongs through a soft maximum function (Softmax).

[0071] The Softmax classification function is a commonly used activation function used for classification problems. It converts each element of the input vector into a probability value between 0 and 1, and the sum of these probability values ​​is 1. The Softmax function does not change the characteristics of the input vector, but only converts it into a probability distribution form.

[0072] Example 2: A mouse species classification system based on multimodal data T1: Mouse species video data, gait video data and audio data input module, the video data includes video content information and video frame images; the gait video data includes gait video content information and gait video frame images; T2: Data preprocessing module, used for preprocessing video frame images, gait video frame images and audio data to obtain high-quality video image frame data, gait image frame data and high-quality audio data; T3: Feature extraction module, using a deep convolutional neural network optimized for rodent features to extract features of high-quality video image frame data, obtaining a video feature atlas F; using a deep convolutional neural network optimized for rodent features to extract features of high-quality gait image frame data, obtaining a gait feature atlas B; extracting features of high-quality audio data, obtaining a spectrum atlas M; T4: Multimodal feature fusion module, which performs multimodal feature fusion on the video feature atlas F, gait feature atlas B and spectrum atlas M to obtain the feature representation set FBM; T5: Time series modeling module, which uses long short-term memory network to process the feature representation set FBM to obtain the time series feature vector; T6: Global information modeling module, which performs linear projection on the feature representation set FBM to obtain the embedding vector TR, and inputs the embedding vector TR into the Transformer encoder trained for the appearance and sound characteristics of mice to generate a global information feature vector; In a specific example of the present invention, the encoder is a Transformer encoder integrating a self-attention mechanism.

[0073] T7: Feature fusion module, which uses weighted summation to fuse the time series feature vector and the global information feature vector to obtain the mouse multimodal feature vector; T8: Rat species classification module, which classifies rat species based on the multimodal feature vectors of rats and determines the type of rat species.

[0074] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.

[0075] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

Claims

1. A method for classifying mouse species based on multimodal data, characterized in that: The method comprises the following steps: S1: Acquire mouse species video data, gait video data and audio data, wherein the video data includes video content information and video frame images, and the gait video data includes gait video content information and gait video frame images; S2: preprocessing the video frame image, gait video frame image and audio data to obtain high-quality video image frame data, gait image frame data and audio data; S3: extracting features of high-quality video image frame data using a deep convolutional neural network optimized for rodent features to obtain a video feature atlas F; extracting features of high-quality gait image frame data using a deep convolutional neural network optimized for rodent features to obtain a gait feature atlas B; extracting features of high-quality audio data to obtain a spectrum atlas M; S4: Perform multimodal feature fusion on the video feature atlas F, the gait feature atlas B and the spectrum atlas M to obtain the feature representation set FBM; S5: Use long short-term memory network to process the feature representation set FBM to obtain the time series feature vector; S6: Obtain a global information feature vector, obtain an embedding vector TR by linearly projecting the feature representation set FBM, and input the embedding vector TR into an encoder trained for the appearance and sound characteristics of mice to generate a global information feature vector; S7: feature fusion, using weighted summation to fuse the time series feature vector and the global information feature vector to obtain the mouse multimodal feature vector; S8: Classify the species of mice based on their multimodal feature vectors and determine the type of the species.

2. The method for classifying rat species according to claim 1, characterized in that: The video content information includes frame rate.

3. The method for classifying rat species according to claim 2, characterized in that: The preprocessing of the video frame image, gait video frame image and audio data in S2 includes: S201: unifying the frame rates of video frame images according to video content information and a reference frame rate to obtain a set of video images with unified frame rates; S202: unifying the frame rates of gait video frame images according to gait video content information and a reference frame rate to obtain a set of gait images with unified frame rates; S203: enhancing the video image set with unified frame rates to obtain high-quality video image frame data; S204: enhancing the gait image set with unified frame rates to obtain high-quality gait image frame data; S205: Standardizing the audio data; S206: Perform noise reduction and enhancement processing on the standardized audio data to obtain high-quality audio data.

4. The method for classifying rat species according to claim 1, characterized in that: The spectrum atlas M obtained in S3 includes: S301: extracting high-quality audio data features using a logarithmic Mel-spectrogram to obtain a spectrum graph M1; S302: using Mel frequency cepstral coefficients to process high-quality audio data to obtain a spectrum graph M2; S303: Process the spectrum graph M1 using a second-order difference method to obtain a spectrum graph M3; S304: Unify the dimensions of the spectrogram M1, the spectrogram M2, and the spectrogram M3 and concatenate them to obtain a feature fusion spectrogram M4; S305: Crop the feature fusion spectrum graph M4 according to the time interval of the video feature atlas F to obtain the spectrum atlas M.

5. The method for classifying rat species according to claim 1, characterized in that: The multimodal feature fusion in S4 includes: S401: performing dimension unification and normalization processing on the video feature atlas F, the gait feature atlas B, and the spectrum atlas M; S402: Use the attention network and weighted fusion method to perform multimodal feature fusion on the results of S401 to obtain a unified feature representation set FBM.

6. The method for classifying rat species according to claim 1, characterized in that: The S5 also includes averaging the hidden state set of each time step structure obtained by long short-term memory network processing to obtain a time series feature vector.

7. The method for classifying rat species according to claim 1, characterized in that: The encoder in S6 includes a Transformer encoder with an integrated self-attention mechanism.

8. The method for classifying rat species according to claim 1, characterized in that: The classification of mouse species based on the multimodal feature vector in S8 includes: S801: Input the multimodal feature vector into the classifier to obtain a classification array with a length equal to the confidence score of the mouse species; S802: Determine the rat species category, and the category corresponding to the maximum confidence score is the pest rat.

9. A mouse species classification system based on multimodal data, characterized in that: The system comprises: T1: a module for inputting mouse species video data, gait video data and audio data, wherein the video data includes video content information and video frame images, and the gait video data includes gait video content information and gait video frame images; T2: Data preprocessing module, used for preprocessing video frame images, gait video frame images and audio data to obtain high-quality video image frame data, gait image frame data and high-quality audio data; T3: Feature extraction module, using a deep convolutional neural network optimized for rodent features to extract features of high-quality video image frame data, obtaining a video feature atlas F; using a deep convolutional neural network optimized for rodent features to extract features of high-quality gait image frame data, obtaining a gait feature atlas B; extracting features of high-quality audio data, obtaining a spectrum atlas M; T4: Multimodal feature fusion module, which performs multimodal feature fusion on the video feature atlas F, gait feature atlas B and spectrum atlas M to obtain the feature representation set FBM; T5: Time series modeling module, which uses long short-term memory network to process the feature representation set FBM to obtain the time series feature vector; T6: Global information modeling module, which performs linear projection on the feature representation set FBM to obtain the embedding vector TR, and inputs the embedding vector TR into the encoder trained for the appearance and sound characteristics of mice to generate a global information feature vector; T7: Feature fusion module, which uses weighted summation to fuse the time series feature vector and the global information feature vector to obtain the mouse multimodal feature vector; T8: Rat species classification module, which classifies rat species based on the multimodal feature vectors of rats and determines the type of rat species.

10. The rat species classification system according to claim 9, characterized in that: The T5 time series modeling module includes averaging the hidden states of each time step structure obtained by long short-term memory network processing to obtain a time series feature vector.

Images