A method and system for identifying birds in substations based on video and audio recognition

Abstract

The present invention discloses a method and system for identifying birds in a substation based on video and audio recognition. The method comprises: detecting key feature points of bird video data collected around the substation by using the FAST algorithm, marking the detected key feature points by using the BRIEF algorithm, obtaining bird video data marked with key feature points, partitioning the bird video data screen, identifying the large change area to be re-determined, and cutting these large change areas with a rectangular frame as a separate image; performing Mosaic data enhancement on the bird image data in the large change area cut by the rectangular frame; filtering out image data that meets the characteristics of birds from the data-enhanced bird image data by using the TDAN algorithm, and generating a video sequence with a higher resolution; filtering out audio data that meets the characteristics of birds from the bird audio data corresponding to the bird image data after feature recognition by using the M-GRNN model; using the AVSR method to perform modal feature fusion on the video data and audio data that meet the characteristics of birds, and then inputting the modal feature into the Transformer decoder for bird information recognition, wherein LM is embedded in the Transformer decoder to perform joint decoding to obtain the bird recognition result. The present invention can significantly improve the accuracy of bird recognition.

Description

Technical Field

[0001] The invention relates to biometric identification, and in particular to a method and system for identifying birds in a substation based on video and audio identification. Background Art

[0002] Birds often nest, stay or fly within the substation, which can easily cause short circuits, equipment damage and other problems. In serious cases, it can even trigger large-scale power outages. Identifying and driving away birds around substations is an effective way to avoid such accidents. Therefore, accurate bird identification is of great significance to ensure the stable operation of substations.

[0003] At present, bird recognition methods are roughly divided into recognition methods based on probability models and recognition methods based on template matching. They are either based on separate video data for bird recognition or based on separate audio data for bird recognition. There are generally a large number of equipment and buildings around substations, and birds are integrated with the background, making it difficult to accurately identify them. In addition, due to the strong changes in light, birds in the image will appear shadowed and exposed, resulting in the loss of bird feature information, making bird images appear uncertain, and increasing the difficulty of bird recognition. Audio recognition is prone to mismatching in large-scale non-specific bird sound recognition.

[0004] In summary, the existing bird identification methods still need to be improved in terms of identification accuracy. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method and system for identifying birds in a substation based on video and audio recognition with higher recognition accuracy.

[0006] Technical solution: The substation bird identification method based on video and audio recognition of the present invention comprises:

[0007] (1) The key feature points of the collected bird video data around the substation are detected by the FAST algorithm, and the detected key feature points are marked by the BRIEF algorithm to obtain the bird video data marked with key feature points. The bird video data screen is partitioned to identify the large change areas that need to be re-determined, and these large change areas are cut out using rectangular frames as a separate image;

[0008] (2) Mosaic data enhancement is performed on bird image data in the large change area cut by the rectangular frame;

[0009] (3) Using the TDAN algorithm, image data that meets bird characteristics is selected from the data-enhanced bird image data to generate a higher-resolution video sequence;

[0010] (4) From the bird audio data corresponding to the bird image data after feature recognition, the audio data that meets the bird characteristics is screened out through the M-GRNN model;

[0011] (5) The AVSR method is used to fuse the modal features of the video data and audio data that meet the characteristics of birds, and then the data is input into the Transformer decoder for bird information recognition. LM is embedded in the Transformer decoder to perform joint decoding to obtain the bird recognition result.

[0012] Further, step (1) comprises:

[0013] (101) Extract video frames from bird video data, and eliminate duplicate video frame data by comparing feature points between consecutive frames; while analyzing the k-1th frame, introduce the kth frame as a reference; use the FAST algorithm to detect feature points in the picture, randomly select a decision point, and arrange several detection points around it. When there are more than R points around the decision point and the grayscale difference with the decision point is greater than the acceptable range, the decision point is identified as a key feature point; R is the set threshold;

[0014] The identification of key feature points is based on the following formula:

[0015]

[0016] Where N represents the number of feature points; circl(p) represents the circular area centered on pixel p; I(p) and I(x) represent the grayscale values ​​of two pixels; ε d is the threshold; p represents the decision point at the sampling center; x represents a pixel point selected around the feature point; f CRZ (I(p),I(x)) is 1, indicating that the difference between the gray values ​​of the two pixels is greater than the acceptable range, f CRZ (I(p),I(x)) is 0, which means the difference between the gray values ​​of the two pixels is within the acceptable range;

[0017] If the N calculated at a certain point is greater than R, then the point is a key feature point;

[0018] (102) After all key feature points are identified in the video frame, these key feature points are described using the BRIEF algorithm;

[0019] The BRIEF eigenvalue of key feature point I is calculated by the following formula:

[0020]

[0021] Among them, x i represents the pixel value of a random point x; y irepresents the pixel value of a random point y; T(x i ,y i ) represents the relative intensity of these two pixel values;

[0022] When calculating the BRIEF value of each key feature point, 128 pairs of sampling pixels are randomly selected around the point based on Gaussian distribution;

[0023] Calculate the key feature point I of the k-1th frame i and the key feature point T of the kth frame j Fuzzy similarity μ(I i ,T j ):

[0024]

[0025] Among them, f(I i ) represents the BRIEF feature value of the key feature point in the k-1th frame; f(T j ) represents the BRIEF feature value of the key feature point of the kth frame; Sum(n0 represents the value obtained by converting the number n into binary and summing it bit by bit; Θ represents the XOR operation;

[0026] The BRIEF value of the key feature point contains only 128 bits, and the fuzzy similarity between any two points varies between 0 and 1, where μ(I i ,T j )=1 represents the complete correspondence between key feature points, and μ(I i ,T j ) = 0 means that the features are completely different and the possibility of matching points is extremely small;

[0027] Establish a set to measure the similarity between key feature points, which contains points I1, I2, I3, ... I of the original bird image feature point set I n and the points T1, T2, T3, ... T of the target image feature point set T n The corresponding relationship;

[0028] A={ i ,T j ,μ(I i ,T j )>|I i ∈I,T j ∈T}

[0029] The total number of key feature points extracted from the k-1th frame image is M, and the total number of key feature points identified in the kth frame is N, M ≥ N; the goal is to screen out a unique feature point X1, X2, X3...X in the k-1th frame n ​Combine to form a set X, and select the same number of feature points Y1, Y2, Y3...Y in the kth frame n Form a set Y such that pairs of points in sets X and Y match each other, subject to the following criteria:

[0030] ① In two sets, the fuzzy similarity between two points with the same sequence number should be greater than 0.5;

[0031] ② Set X and set Y are of the same size;

[0032] ③The feature points in set X and set Y are non-repetitive and unique;

[0033] The set X and the set Y that meet the above criteria are called a matching scheme for the k-1th frame and the kth frame, and the total matching degree S of the two point sets is calculated. match :

[0034]

[0035] Where n represents the length of the set; μ(X i ,Y i ) represents the feature point X i and feature point Y i The matching degree of X i is the feature point in set X; Y i is the feature point in set Y;

[0036] There are many possible pairing combinations of the midpoints of the kth frame and the k-1th frame. Among all these possible pairing combinations, select the total matching degree S match The optimal feature point pairing combination is used as output.

[0037] Furthermore, step (1) further comprises:

[0038] (103) According to the optimal feature point matching strategy, the video frame is segmented. In this process, the target frame is converted into a grid structure to form a segmented grid, and the grids except the outermost column and the bottom row are all squares with a side length of D.

[0039] Calculation formula for grid side length D:

[0040]

[0041] Wherein, w represents the overall width of the image, h represents the overall height of the image, and D represents the side length of each rectangular grid after the image is cut into segmented grids;

[0042] (104) partitioning the image to form multiple D*D grid images, and evaluating the magnitude of changes in the content of each grid; performing serial evaluation on the proportion of unsuccessfully matched feature points in each grid, and if the proportion exceeds a preset limit value, determining that the grid has shown significant changes in the current frame and needs to be re-identified;

[0043] The calculation formula for the proportion of mismatched feature points P is as follows:

[0044]

[0045] Among them, P represents the proportion of unmatched feature points, m represents the total number of feature points in the D*D grid being calculated, and C i Indicates the matching status of the i-th point in the grid;

[0046]

[0047] After the P value reaches the established limit, the region is identified as a large change region;

[0048] After identifying all the large change areas that need to be re-determined, these adjacent areas are grouped together; the size of the group of grids, namely the width and height, is measured; these large change areas are cut out using a rectangular frame as a separate image, and the cut image is used for subsequent image recognition.

[0049] Further, step (2) comprises:

[0050] (201) Create a new image sample and randomly select a point within this range as the center point of the image;

[0051] (202) Randomly select four images from the data set, and perform data augmentation operations including scaling, flipping, rotating, random cropping, and color transformation on these images;

[0052] (203) placing the processed image around the center point in four positions: the upper left corner, the lower left corner, the upper right corner, and the lower right corner;

[0053] (204) The four image regions are cut out and stitched into a new image; the portion exceeding the image sample range will be cut off.

[0054] Further, step (3) comprises:

[0055] Utilize low-resolution LR support frames and reference frame Apply TDAN algorithm to realize time dimension and Precise alignment:

[0056] First, the enhanced image is feature extracted, and a shared feature extraction network is used to extract the features from the parallel input and Get the visual feature F i LR and F t LR ; The network structure includes a convolutional layer and k1 residual blocks, and Re-LU is used as the activation function;

[0057] Then the deformation alignment module transforms the visual features F i LR and F t LR As input, to predict F i LR The convolution kernel offset Θ:

[0058] Θ=f θ (F i LR ,F t LR )

[0059] f θ (·) is the feature extraction function;

[0060] Using the convolution kernel offset Θ and visual features F i LR , and then calculate the alignment feature F of the support frame through deformation convolution i LR' :

[0061] F i LR' =f dc (F i LR ,Θ)

[0062] Among them, f dc (·) is the deformable convolution operator; Θ={Δp n |n=1,...,R},Δp n is the convolution kernel position offset, |R| is the number of convolution kernel parameters;

[0063] For the alignment feature F i LR' Each position p0 has the following relationship:

[0064]

[0065] Among them, w(p n ) is the convolution kernel position p n The learnable parameters of n F iLR The nth convolution kernel position;

[0066] The convolution operation will be applied to non-integer positions p n +Δp n On, where Δp n It may represent a decimal. To solve this problem, the operation is realized by using the bilinear interpolation method.

[0067] Then, a convolutional layer with kernel size is used to predict the sampling parameter |R|, which will be the number of output channels; finally, the deformable convolution operation is performed from Θ and F i LR Get alignment feature F i LR' ; In order to improve the transformation ability and adaptability of the module, dc Three additional deformation convolutional layers are added in both the previous and next stages;

[0068] Finally, a 3×3 convolutional layer is used to reconstruct the low-resolution LR frame from the aligned feature maps. H×W is the frame size, C is the number of channels;

[0069] In the LR reference frame and support frames After being fed back to the TDAN network, the aligned low-resolution image is finally obtained, and the super-resolution reconstruction network is further used to predict the high-resolution video frame.

[0070] Further, step (4) comprises:

[0071] (401) using MFCC to obtain key feature vectors of the audio signal;

[0072] (402) The key feature vector processed by MFCC is input into GRNN for training and classification to realize the recognition and classification of bird audio data.

[0073] Further, step (401) includes:

[0074] First, pre-emphasis processing is applied to the audio signal. The pre-emphasis formula is:

[0075] y[n]=x[n]-αx[n-1]

[0076] Wherein, y[n] represents the pre-emphasized signal; x[n] is the nth sampling point of the original audio input signal; α represents the pre-emphasis coefficient, which is 0.97 or 0.98;

[0077] The audio signal is framed. Framing is achieved by smoothly moving the selected window function on the audio signal and weighting it. The calculation formula is as follows:

[0078] y w [n]=ω[n]×y[n]

[0079] Among them, ω[n] represents the window function, and y[n] represents the frame signal;

[0080] In the process of framing the audio signal, in order to ensure the smoothness of the acoustic feature parameters, a section of audio information is retained between two adjacent frames;

[0081] In order to avoid spectrum leakage caused by using rectangular window, Hanning window is used to process the audio signal;

[0082] The calculation formula of the window function of the Hanning window is as follows:

[0083]

[0084] Among them, ω han [n] represents the window function of the Hanning window, and N represents the length of the window.

[0085] Further, step (402) includes:

[0086] The GRNN neural network structure includes an input layer, a pattern layer, a summation layer, and an output layer; the input vector received by the network and the output vector provided are X = [x1, x2, ..., x n ] T and Y=[y1,y2,...,y k ] T ; In the translation of input information, the output layer transmits information through neurons. The function of this layer is to process the bird audio signal transmitted by the input layer. The formula of the GRNN neural network output layer is as follows:

[0087]

[0088] Where X is the input vector; X i is the value of the input vector training sample corresponding to the i-th neuron; σ is the width coefficient of the Gaussian function, that is, the smoothing factor;

[0089] The GRNN neural network summation layer mainly calculates the sum of the input corresponding to its internal neurons and the input corresponding to other neurons, that is, the input audio single signal is compared and fused; the calculation formula of the GRNN neural network summation layer is as follows:

[0090]

[0091] Among them, SR S is the weighted sum of the first type of neurons over all pattern layers; Ng is the weighted sum of the second type of neurons on all pattern layers; X is the input vector; X i is the value of the input vector training sample corresponding to the i-th neuron; σ is the width coefficient of the Gaussian function, that is, the smoothing factor; the summation layer mainly sums the output results of all neurons in the previous layer, and the connection weight of the first type of neurons and each neuron in the pattern layer is 1; the second type of neurons weighted sum the neurons of all pattern layers, and the connection weight is Y i ;

[0092] After the neurons in all pattern layers perform weighted summation, the recognition results of the bird audio data are obtained.

[0093] Further, step (5) comprises:

[0094] (501) The AVSR method is used to realize the task of audio and video modality fusion, the LM joint decoder is embedded in the joint decoding reasoning to achieve character-level prediction, and the LM decoding reasoning runs through the entire beam search;

[0095] The position encoding PE is used to embed the position information in the input sequence of the encoder, and the position encoding is performed through sine and cosine functions. The specific method is as follows:

[0096]

[0097] Among them, pos represents the absolute position of the word in the sentence; i represents the dimension of the word vector, i = 0, 1, ..., 225; d = 512;

[0098] (502) The audio features f extracted by the front-end model are encoded through a dual encoding channel. audio and video features f video The audio and video encoding structures are the same, and both use the Transformer self-attention encoding layer to encode the audio and video channels separately to obtain the audio stream encoding features:

[0099]

[0100] in, Represents audio data, i = 1, 2, 3, ..., m; m is the number of bird audio coding features; R d represents a d-dimensional vector;

[0101] Video stream encoding characteristics:

[0102]

[0103] in, Represents audio data, i = 1, 2, 3, ..., n; n is the number of bird video coding features;

[0104] The dimension of the encoded feature vector of the audio and video streams is 512; the audio and video joint features are output through the multimodal feature fusion network, where the attention weight distribution is obtained by the softmax function; the calculation formula of the multimodal feature fusion network is as follows:

[0105]

[0106] in, Represents audio coding features, Represents the video encoding characteristics, represents multimodal joint features; Q a , V a , Q v , V v The query matrix, key matrix, and value matrix represent audio and video respectively; represents the dot product scaling of the attention score function; Joint{·} represents the fusion of different modal features; cat(·) function is used to concatenate strings;

[0107] Encoding audio features and video coding features It is sent to the multimodal feature fusion network, and after feature cascade processing, it is passed to the joint convolutional network to generate a multimodal joint feature. Its feature dimension is consistent with the encoding feature dimension of the input sound and image; finally, the comprehensive encoding features of audio and video will be sent to the decoding module;

[0108] (503) The LM decoder is embedded in the Transformer decoder for joint decoding. The Transformer decoder is composed of 6 layers like the encoder. The LM is constructed by a multi-layer stacked LSTM network and is trained using the training set data.

[0109] In AVSR, the original decoding method consists of the joint encoding feature vector at the current time t and the previous prediction result y t-1 The probability is calculated through the Softmax layer; LM is embedded in the original decoding method, and the joint decoding is performed with the Transformer decoder as follows:

[0110]

[0111] Among them, p decProvided by Transformer decoder;p LM Provided by LM; t is the predicted output at time t; β is a hyperparameter.

[0112] The substation bird identification system based on video and audio recognition of the present invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the substation bird identification method based on video and audio recognition is implemented. The substation bird identification system has the advantages of the substation bird identification method.

[0113] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0114] (1) The present invention uses the FAST algorithm to perform feature detection on the collected substation bird video data, and uses the BRIEF algorithm to describe the feature points, wherein the FAST algorithm completes the discrimination effect of a certain number of pixels in the video data and their point pixel values ​​being different, so that the BRIEF algorithm can respond randomly, speed up the establishment of binary pixel points in the video, and make the generated binary pixel points convenient for BRIEF to extract feature values; the video data marked by the BRIEF algorithm is convenient for the subsequent TDAN algorithm to identify;

[0115] (2) The present invention uses Mosaic data enhancement to improve the richness of the bird image information set, and then uses the TDAN algorithm to identify bird information. In view of the fact that the image information in the collected bird video data may be missing and have abnormal pixel values, it is proposed to use Mosaic data enhancement to randomly use, for example, 4 to 5 pictures in the video, randomly scale them, and then randomly distribute them for splicing to enrich the bird video data set. In particular, random scaling increases the number of bird targets in the video data set, making the TDAN algorithm more robust.

[0116] (3) The present invention improves the understanding and classification accuracy of audio data through the M-GRNN model; the M-GRNN model first obtains the key features of the audio signal through the Mel-frequency cepstral coefficient (MFCC), so that the GRNN can accelerate the recognition and classification of bird audio features; finally, the AVSR method is used to realize the fusion of audio and video modalities, and the LM is embedded in the Transformer decoder to perform joint decoding to obtain the final bird recognition result output.

[0117] In summary, the present invention can achieve accurate extraction of audio and video features, and then perform modal fusion of the audio and video features, which can significantly improve the accuracy of bird recognition. The present invention can also be used to identify other organisms and has a certain degree of universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 It is a flowchart of a method for identifying birds in a substation based on video and audio recognition provided by an embodiment of the present invention;

[0119] Figure 2 It is a flow chart of the M-GRNN model in an embodiment of the present invention. DETAILED DESCRIPTION

[0120] The present invention will be further described below in conjunction with the accompanying drawings.

[0121] like Figure 1 As shown, the embodiment of the present invention provides a method for identifying birds in a substation based on video and audio recognition, which specifically includes the following steps:

[0122] (1) The key feature points of the bird video data collected around the substation are detected using the FAST algorithm, and the detected key feature points are marked using the BRIEF algorithm to obtain the bird video data marked with key feature points. The bird video data screen is partitioned to identify the large change areas that need to be re-determined, and these large change areas are cut out using rectangular frames as a separate image, so that the subsequent TDAN algorithm can more accurately realize bird feature recognition;

[0123] The bird video data can be collected by a fixed camera or a camera mounted on a drone. The data collected by the camera includes both bird video data and bird audio data.

[0124] (101) Extract video frames from bird video data, and eliminate duplicate video frame data in the video by comparing feature points between consecutive frames; while analyzing the k-1th frame, introduce the kth frame as a reference; use the FAST algorithm to detect feature points in the picture, randomly select a decision point, and arrange 16 detection points around it, set the threshold to R, and when there are more than R points around the decision point and the grayscale difference with the decision point is greater than the acceptable range, the decision point is identified as a key feature point;

[0125] The identification of key feature points is based on the following formula:

[0126]

[0127]

[0128] Where N represents the number of feature points; circl(p) represents the circular area centered on pixel p; I(p) and I(x) represent the grayscale values ​​of two pixels; ε d is the threshold; p represents the decision point at the sampling center; x represents a pixel point selected around the feature point; f CRZ(I(p),I(x)) is 1, indicating that the difference between the gray values ​​of the two pixels is greater than the acceptable range, f CRZ (I(p),I(x)) is 0, which means that the grayscale value difference between the two pixels is within an acceptable range.

[0129] If the N calculated at a certain point is greater than R, then this point is a key feature point.

[0130] (102) After all key feature points are identified in the video frame, these key feature points are described (marked) using the BRIEF algorithm;

[0131] The BRIEF algorithm randomly selects n pairs of pixels in the vicinity of the key point for intensity comparison and records their relative intensities.

[0132] The BRIEF eigenvalue of key feature point I is calculated by the following formula:

[0133]

[0134] Among them, x i represents the pixel value of a random point x; y i represents the pixel value of a random point y; T(x i ,y i ) represents the relative intensity of these two pixel values.

[0135] When calculating the BRIEF eigenvalue of each key feature point, 128 pairs of sampling pixels are randomly selected around the point based on Gaussian distribution;

[0136] Calculate the key feature point I of the k-1th frame i and the key feature point T of the kth frame j Fuzzy similarity μ(I i ,T j ):

[0137]

[0138] Among them, f(I i ) represents the BRIEF feature value of the key feature point in the k-1th frame; f(T j ) represents the BRIEF feature value of the key feature point of the kth frame; Sum(n) represents the value obtained by converting the number n into binary and summing it bit by bit; Θ represents the exclusive OR operation;

[0139] Since the BRIEF value of the key feature point contains only 128 bits, the fuzzy similarity between any two points varies between 0 and 1, where μ(I i ,T j )=1 represents the complete correspondence between key feature points, and μ(I i ,Tj ) = 0 means that the features are completely different and the possibility of matching points is extremely small;

[0140] Establish a set to measure the similarity between key feature points, which contains points I1, I2, I3, ... I of the original bird image feature point set I n and the points T1, T2, T3, ... T of the target image feature point set T n The corresponding relationship;

[0141] A={ i ,T j ,μ(I i ,T j )>|I i ∈I,T j ∈T}

[0142] The total number of key feature points extracted from the k-1th frame image is M, and the total number of key feature points identified in the kth frame is N, M ≥ N; the goal is to screen out a unique feature point X1, X2, X3...X in the k-1th frame n Combine to form a set X, and select the same number of feature points Y1, Y2, Y3...Y in the kth frame n Form a set Y such that pairs of points in sets X and Y match each other, subject to the following criteria:

[0143] ① In two sets, the fuzzy similarity between two points with the same sequence number should be greater than 0.5;

[0144] ② Set X and set Y are of the same size;

[0145] ③The feature points in set X and set Y are non-repetitive and unique;

[0146] The set X and the set Y that meet the above criteria are called a matching scheme for the k-1th frame and the kth frame, and the total matching degree S of the two point sets is calculated. match :

[0147]

[0148] Where n represents the length of the set; μ(X i ,Y i ) represents the feature point X i and feature point Y i The matching degree of X i is the feature point in set X; Y i is the feature point in set Y;

[0149] Obviously, there are many possible pairing combinations of the midpoints of the kth frame and the k-1th frame. Among all these possible pairing combinations, we select the total matching degree S​match The optimal feature point pairing combination is used as output.

[0150] (103) According to the optimal feature point matching strategy, the video frame is segmented. In this process, the target frame is converted into a grid structure to form a segmented grid, and the grids except the outermost column and the bottom row are all squares with a side length of D.

[0151] Calculation formula for grid side length D:

[0152]

[0153] Where D represents the side length of each rectangular grid after the image is cut into segmented grids, w represents the overall width of the image, and h represents the overall height of the image;

[0154] (104) The image is partitioned to form multiple D*D grid images, and the extent of change in the content of each grid is evaluated; the proportion of unsuccessfully matched feature points in each grid is serially evaluated. If the proportion exceeds a preset limit value, it is determined that the grid shows significant changes in the current frame and needs to be re-identified; the setting of the limit value can be adjusted according to the specific needs of the operator, because various recognition technologies have different sensitivities to this value.

[0155] The calculation formula for the proportion of mismatched feature points P is as follows:

[0156]

[0157] Among them, P represents the proportion of unmatched feature points, m represents the total number of feature points in the D*D grid being calculated, and C i Indicates the matching status of the i-th point in the grid;

[0158]

[0159] If the i-th point has not been matched yet, it is assigned a value of 1; if it has been matched, it is set to 0.

[0160] After the P value reaches the established cutoff value, the region is identified as a large change region;

[0161] After identifying all the large change areas that need to be re-determined, these adjacent areas are grouped together; the size of the group of grids, namely the width and height, is measured; these large change areas are cut out using a rectangular frame as separate images, and the cut images are used for subsequent image recognition.

[0162] (2) Mosaic data enhancement is performed on bird image data in the large change area cut by the rectangular frame;

[0163] By extracting the features of bird videos, the cut-off images still show the limitations of monotonous landscape and insufficient target samples. Before using the bird recognition algorithm to identify the bird features, the bird image data cut off from the rectangular frame is enhanced. The present invention proposes to introduce Mosaic data enhancement technology to increase the richness of the bird image data set, so as to enhance the algorithm's ability to recognize bird features.

[0164] By performing various transformations on the cut bird image data, the applicability of the bird recognition algorithm is improved and the tendency of the network to over-fit the training data is reduced. Mosaic data enhancement randomly cuts and deforms four pictures that may contain bird features and splices them into one picture as the data set for the bird recognition algorithm.

[0165] Mosaic data enhancement includes the following steps:

[0166] (201) Create a new image sample and randomly select a point within this range as the center point of the image;

[0167] (202) randomly selecting four images from the cut bird image data, and performing data augmentation operations including scaling, flipping, rotating, random cropping, and color transformation on these images;

[0168] (203) placing the processed image around the center point in four positions: the upper left corner, the lower left corner, the upper right corner, and the lower right corner;

[0169] (204) The four image regions are cut out and stitched into a new image; the portion exceeding the image sample range will be cut off.

[0170] By using Mosaic data augmentation, the training effectiveness of a single image is equivalent to multi-image training, and once the images are synthesized, the target size is reduced, which is equivalent to introducing many small targets in the input data, helping the TDAN algorithm to more accurately identify the visual feature information of birds, including the overall appearance, color, texture, shape, size, behavioral habits, etc.

[0171] (3) Using the TDAN (video super-resolution) algorithm, image data that meets bird characteristics is selected from the data-enhanced bird image data to generate a higher-resolution video sequence;

[0172] The present invention proposes to use the TDAN algorithm to perform feature recognition on the bird image data after data enhancement. and reference frame Apply TDAN algorithm to realize time dimension and Precise alignment;

[0173] The specific implementation process of the TDAN algorithm is as follows:

[0174] First, the enhanced image is feature extracted, and a shared feature extraction network is used to extract the features from the parallel input and Get the visual feature F i LR and F t LR ; The network structure includes one convolutional layer and k1 residual blocks, and Re-LU is used as the activation function.

[0175] Then the deformation alignment module transforms the visual features F i LR and F t LR As input, to predict F i LR The convolution kernel offset Θ:

[0176] Θ=f θ (F i LR ,F t LR )

[0177] f θ (·) is the feature extraction function.

[0178] Using the convolution kernel offset Θ and visual features F i LR , and then calculate the alignment feature F of the support frame through deformation convolution i LR' :

[0179] F i LR' =f dc (F i LR ,Θ)

[0180] Among them, f dc (·) is the deformable convolution operator; Θ={Δp n |n=1,...,R},Δp n is the convolution kernel position offset, |R| is the number of convolution kernel parameters.

[0181] For the alignment feature F i LR' Each position p0 has the following relationship:

[0182]

[0183] Among them, w(p n ) is the convolution kernel position pn The learnable parameters of n F i LR The nth convolution kernel position.

[0184] The convolution operation will be applied to non-integer positions p n +Δp n On, where Δp n It may represent a decimal. To solve this problem, the operation is realized by using the bilinear interpolation method.

[0185] Then, a convolutional layer with kernel size is used to predict the sampling parameter |R|, which will be the number of output channels; finally, the deformable convolution operation is performed from Θ and F i LR Get alignment feature F i LR' ; In order to improve the transformation ability and adaptability of the module, dc Three additional deformation convolutional layers are added in both the previous and next stages;

[0186] Reference frame F t LR The characteristic data is only used to determine the offset and will not be passed on to the support system F i LR' The adaptively learned offsets implicitly capture motion cues while also exploring neighboring features within the same image for temporal alignment.

[0187] Finally, a 3×3 convolutional layer is used to reconstruct the low-resolution LR frame from the aligned feature maps. H×W is the frame size, and C is the number of channels.

[0188] In the LR reference frame and support frames After the continuous low-resolution frames are fed back to the TDAN network, the aligned low-resolution images are finally obtained, and the super-resolution reconstruction network is further used to predict the high-resolution video frames. Get higher resolution video sequences that match bird features.

[0189] In the above steps (2) and (3), considering the complex real environment of the substation, the present invention first uses the Mosaic data enhancement method to enhance the recognition of bird image information, and then completes the recognition of bird information features in the image through the TDAN algorithm.

[0190] (4) From the bird audio data corresponding to the bird image data after feature recognition, the audio data that meets the bird characteristics is screened out through the M-GRNN model;

[0191] The data collected by the camera also includes audio components, which also need to be analyzed. In order to achieve efficient processing of audio data and process time-series audio signals of unequal length, the first step is to perform preprocessing to generate specific feature vectors. In order to improve the recognition of bird information around the substation, the present invention proposes to use the M-GRNN model to identify audio features of the audio data corresponding to the bird video data after feature recognition. In the feature extraction link, the key features of the audio signal are obtained through the Mel-frequency cepstral coefficient (MFCC), and the generalized regression neural network (GRNN) algorithm is used to identify and classify according to the bird category.

[0192] (401) using MFCC to obtain key feature vectors of the audio signal;

[0193] like Figure 2 As shown in FIG. 1 , the audio feature extraction process of MFCC is: pre-emphasis, framing, and windowing to obtain acoustic features suitable for further analysis.

[0194] In order to compensate for the high-frequency energy loss in the speech signal due to the lip radiation suppression, a pre-emphasis processing is first applied to the audio signal, which helps to compensate for the high-frequency component of the audio signal in the subsequent analysis and processing process.

[0195] The formula for pre-emphasis is:

[0196] y[n]=x[n]-αx[n-1]

[0197] Wherein, y[n] represents the pre-emphasized signal; x[n] is the nth sampling point of the original audio input signal; α represents the pre-emphasis coefficient, which is 0.97 or 0.98;

[0198] Considering that the speech signal in the bird video is a non-stationary signal, it needs to be framed to improve the stability of the signal output. Framing is to smoothly move the selected window function on the audio signal and weight it at the same time. The calculation formula is as follows:

[0199] y w [n]=ω[n]×y[n]

[0200] Among them, ω[n] represents the window function, and y[n] represents the frame signal;

[0201] In the process of framing the audio signal, in order to ensure the smoothness of the acoustic feature parameters, a section of audio information is retained between two adjacent frames;

[0202] Since the use of rectangular windows is prone to serious spectrum leakage, the present invention uses a Hanning window to process the audio signal;

[0203] The calculation formula of the window function of the Hanning window is as follows:

[0204]

[0205] Among them, w han [n] represents the window function of the Hanning window, and N represents the length of the window.

[0206] (402) The feature vector processed by MFCC is input into GRNN for training and classification. GRNN uses the features extracted by MFCC to learn the patterns and features of the audio signal to realize the recognition and classification of bird audio data.

[0207] Generalized regression neural network (GRNN) is a branch of radial basis function neural network, and its mathematical basis is nonlinear regression analysis. The GRNN neural network structure includes input layer, pattern layer, summation layer and output layer; the input received by the network and the output vector provided are X = [x1, x2, ..., x n ] T and Y=[y1,y2,...,y k ] T ; In the translation of input information, the output layer transmits information through neurons. The function of this layer is to process the bird audio signal transmitted by the input layer. The formula of the GRNN neural network output layer is as follows:

[0208]

[0209] Where X is the input vector; X i is the value of the input vector training sample corresponding to the i-th neuron; σ is the width coefficient of the Gaussian function, that is, the smoothing factor;

[0210] The GRNN neural network summation layer mainly calculates the sum of the input corresponding to its internal neurons and the input corresponding to other neurons, that is, the input audio single signal is compared and fused; the calculation formula of the GRNN neural network summation layer is as follows:

[0211]

[0212]

[0213] Among them, S R S is the weighted sum of the first type of neurons over all pattern layers; Ng is the weighted sum of the second type of neurons on all pattern layers; X is the input vector; X iis the value of the input vector training sample corresponding to the i-th neuron; σ is the width coefficient of the Gaussian function, that is, the smoothing factor; the summation layer mainly sums the output results of all neurons in the previous layer, and the connection weight of the first type of neurons and each neuron in the pattern layer is 1; the second type of neurons weighted sum the neurons of all pattern layers, and the connection weight is Y i ;

[0214] After the neurons in all pattern layers perform weighted summation, the audio data that matches the characteristics of birds is obtained.

[0215] (5) The AVSR method is used to fuse the modal features of the video data and audio data that meet the characteristics of birds, and then the data is input into the Transformer decoder for bird information recognition. LM is embedded in the Transformer decoder to perform joint decoding to obtain the bird recognition result.

[0216] The present invention proposes to adopt a multimodal dual-stream coding model and a multimodal fusion network to solve the problem of audio and video network fusion and enhance the complementarity between the two.

[0217] (501) The fusion of audio and video modalities is realized through the AVSR (audio-visual speech recognition) method, the LM joint decoder is embedded in the joint decoding reasoning to achieve character-level prediction, and the LM decoding reasoning runs through the entire beam search;

[0218] AVSR mainly consists of three parts: video and audio front-end dual-stream encoding model, multimodal feature fusion network and joint decoding. The audio and video modal data are processed by the visual and audio modal streams respectively.

[0219] Since the self-attention mechanism cannot capture the order of the input element sequence, the position information is integrated into the input to distinguish elements at different positions. The present invention uses position encoding PE to embed the position information in the input sequence of the encoder and performs position encoding through sine and cosine functions. The specific method is as follows:

[0220]

[0221] Among them, pos represents the absolute position; i represents the dimension of the vector, i=0,1,...,225; d=512;

[0222] The image and audio input format of the model input is in a serialized form, and its length is not fixed. In order to adapt to the dynamic changes in the sequence length, an adjustable sine and cosine position encoding method is used. The vector generated by this position encoding is consistent with the feature vector of the audio and video in dimension.

[0223] (502) Multimodal features can effectively use auditory and visual features to complement and connect each other. Through the dual encoding channel, the audio features f extracted by the front-end model are audio and video features f video The audio and video encoding structures are the same, and both use the Transformer self-attention encoding layer to encode the audio and video channels separately to obtain the audio stream encoding features:

[0224]

[0225] in, represents the audio coding feature, i = 1, 2, 3, ..., m; m is the number of audio coding features that match the bird; R d Represents a d-dimensional vector.

[0226] Video stream encoding characteristics:

[0227]

[0228] in, Represents the video coding features, i = 1, 2, 3, ..., n; n is the number of bird video coding features.

[0229] The dimension of the encoded feature vector of the audio and video streams is 512; the audio and video joint features are output through the multimodal feature fusion network, where the attention weight distribution is obtained by the softmax function; the calculation formula of the multimodal feature fusion network is as follows:

[0230]

[0231] in, Represents audio coding features, Represents the video encoding characteristics, represents multimodal joint features; Q a , V a , Q v , V v The query matrix, key matrix, and value matrix represent audio and video respectively; represents the dot product scaling of the attention score function; Joint{·} represents the fusion of different modal features; cat(·) function is used to concatenate strings.

[0232] Encoding audio features and video coding features It is sent to the multimodal feature fusion network, and after feature cascade processing, it is passed to the joint convolutional network to generate a multimodal joint feature. Its feature dimension is consistent with the encoding feature dimension of the input sound and image; finally, the comprehensive encoding features of audio and video will be sent to the decoding module;

[0233] (503) In the dual-stream multimodal coding network architecture, two coding networks are used to process audio features and video features respectively to obtain corresponding coding features. The coding network of each data stream consists of 6 coding layers. The audio and video modal features are integrated through a fusion network, which produces a more enhanced joint feature by learning the correlation between the two modalities. Realize advanced fusion of audio and video modal features.

[0234] The LM decoder is embedded in the Transformer decoder for joint decoding. The Transformer decoder is composed of 6 layers like the encoder. The LM is constructed by a multi-layer stacked LSTM network and is trained using the training set data.

[0235] In AVSR, the original decoding method consists of the joint encoding feature vector at the current time t and the previous prediction result y t-1 The probability is calculated through the Softmax layer; LM is embedded in the original decoding method, and the joint decoding is performed with the Transformer decoder as follows:

[0236]

[0237] Among them, p dec Provided by Transformer decoder;p LM Provided by LM; t is the predicted output at time t; β is a hyperparameter.

[0238] This method can not only realize the decoding task of audio and video fusion, but also complete the decoding task of audio or video only in a single mode, and obtain the final bird recognition result output.

[0239] An embodiment of the present invention further provides a substation bird identification system based on video and audio recognition, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the substation bird identification method based on video and audio recognition described in the embodiment of the present invention is implemented.

Claims

1. A method for identifying birds in a substation based on video and audio recognition, characterized in that: include: (1) The key feature points of the collected bird video data around the substation are detected by the FAST algorithm, and the detected key feature points are marked by the BRIEF algorithm to obtain the bird video data marked with key feature points. The bird video data screen is partitioned to identify the large change areas that need to be re-determined, and these large change areas are cut out using rectangular frames as a separate image; (2) Mosaic data enhancement is performed on bird image data in the large change area cut by the rectangular frame; (3) Using the TDAN algorithm, image data that meets bird characteristics is selected from the data-enhanced bird image data to generate a higher-resolution video sequence; (4) From the bird audio data corresponding to the bird image data after feature recognition, the audio data that meets the bird characteristics is screened out through the M-GRNN model; (5) The AVSR method is used to fuse the modal features of the video data and audio data that meet the characteristics of birds, and then the data is input into the Transformer decoder for bird information recognition. LM is embedded in the Transformer decoder to perform joint decoding to obtain the bird recognition result; Step (1) comprises: (101) Extract video frames from bird video data, and eliminate duplicate video frame data by comparing feature points between consecutive frames; while analyzing the k-1th frame, introduce the kth frame as a reference; use the FAST algorithm to detect feature points in the picture, randomly select a decision point, and arrange several detection points around it. When there are more than R points around the decision point and the grayscale difference with the decision point is greater than the acceptable range, the decision point is identified as a key feature point; R is the set threshold; The identification of key feature points is based on the following formula: Where N represents the number of feature points; circl(p) represents the circular area centered on pixel p; I(p) and I(x) represent the grayscale values ​​of two pixels; ε d is the threshold; p represents the decision point at the sampling center; x represents a pixel point selected around the feature point; f CRZ (I(p),I(x)) is 1, indicating that the difference between the gray values ​​of the two pixels is greater than the acceptable range, f CRZ (I(p),I(x)) is 0, which means the difference between the gray values ​​of the two pixels is within the acceptable range; If the N calculated at a certain point is greater than R, then the point is a key feature point; (102) After all key feature points are identified in the video frame, these key feature points are described using the BRIEF algorithm; The BRIEF eigenvalue of key feature point I is calculated by the following formula: Among them, x i represents the pixel value of a random point x; y i represents the pixel value of a random point y; T(x i ,y i ) represents the relative intensity of these two pixel values; When calculating the BRIEF value of each key feature point, 128 pairs of sampling pixels are randomly selected around the point based on Gaussian distribution; Calculate the key feature point I of the k-1th frame i and the key feature point T of the kth frame j Fuzzy similarity μ(I i ,T j ): Among them, f(I i ) represents the BRIEF feature value of the key feature point in the k-1th frame; f(T j ) represents the BRIEF feature value of the key feature point of the kth frame; Sum(n) represents the value obtained by converting the number n into binary and summing it bit by bit; Θ represents the XOR operation; The BRIEF value of the key feature point contains only 128 bits, and the fuzzy similarity between any two points varies between 0 and 1, where μ(I i ,T j )=1 represents the complete correspondence between key feature points, and μ(I i ,T j ) = 0 means that the features are completely different and the possibility of matching points is extremely small; Establish a set to measure the similarity between key feature points, which contains points I1, I2, I3, ... I of the original bird image feature point set I n and the points T1, T2, T3, ... T of the target image feature point set T n The corresponding relationship; A={<I i ,T j ,μ(I i ,T j )>|I i ∈I,T j ∈T} The total number of key feature points extracted from the k-1th frame image is M, and the total number of key feature points identified in the kth frame is N, M ≥ N; the goal is to screen out a unique feature point X1, X2, X3...X in the k-1th frame n Combine to form a set X, and select the same number of feature points Y1, Y2, Y3...Y in the kth frame n Form a set Y such that pairs of points in sets X and Y match each other, subject to the following criteria: ① In two sets, the fuzzy similarity between two points with the same sequence number should be greater than 0.5; ② Set X and set Y are of the same size; ③The feature points in set X and set Y are non-repetitive and unique; The set X and the set Y that meet the above criteria are called a matching scheme for the k-1th frame and the kth frame, and the total matching degree S of the two point sets is calculated. match : Where n represents the length of the set; μ(X i ,Y i ) represents the feature point X i and feature point Y i The matching degree of X i is the feature point in set X; Y i is the feature point in set Y; There are many possible pairing combinations of the midpoints of the kth frame and the k-1th frame. Among all these possible pairing combinations, select the total matching degree S matc h optimal feature point pairing combination and use it as output.

2. The method for identifying birds in a substation based on video and audio recognition according to claim 1 is characterized in that: Step (1) also includes: (103) According to the optimal feature point matching strategy, the video frame is segmented. In this process, the target frame is converted into a grid structure to form a segmented grid, and the grids except the outermost column and the bottom row are all squares with a side length of D. Calculation formula for grid side length D: Wherein, w represents the overall width of the image, h represents the overall height of the image, and D represents the side length of each rectangular grid after the image is cut into segmented grids; (104) partitioning the image to form multiple D*D grid images, and evaluating the magnitude of changes in the content of each grid; performing serial evaluation on the proportion of unsuccessfully matched feature points in each grid, and if the proportion exceeds a preset limit value, determining that the grid has shown significant changes in the current frame and needs to be re-identified; The calculation formula for the proportion of mismatched feature points P is as follows: Among them, P represents the proportion of unmatched feature points, m represents the total number of feature points in the D*D grid being calculated, and C i Indicates the matching status of the i-th point in the grid; After the P value reaches the established limit, the region is identified as a large change region; After identifying all the large change areas that need to be re-determined, these adjacent areas are grouped together; the size of the group of grids, namely the width and height, is measured; these large change areas are cut out using a rectangular frame as a separate image, and the cut image is used for subsequent image recognition.

3. The method for identifying birds in a substation based on video and audio recognition according to claim 1, characterized in that: Step (2) comprises: (201) Create a new image sample and randomly select a point within this range as the center point of the image; (202) Randomly select four images from the data set, and perform data augmentation operations including scaling, flipping, rotating, random cropping, and color transformation on these images; (203) placing the processed image around the center point in four positions: the upper left corner, the lower left corner, the upper right corner, and the lower right corner; (204) The four image regions are cut out and stitched into a new image; the portion exceeding the image sample range will be cut off.

4. The method for identifying birds in a substation based on video and audio recognition according to claim 1, characterized in that: Step (3) includes: Utilize low-resolution LR support frames and reference frame Apply TDAN algorithm to realize time dimension and Precise alignment: First, the enhanced image is feature extracted, and a shared feature extraction network is used to extract the features from the parallel input and Get visual features from and The network structure includes one convolutional layer and k1 residual blocks, using Re-LU as the activation function; The deformation alignment module then transforms the visual features and As input, to predict The convolution kernel offset Θ: f θ (·) is the feature extraction function; Using the convolution kernel offset θ and visual features Then use deformation convolution to calculate the alignment features of the support frame Among them, f dc (·) is the deformable convolution operator; Θ={Δp n |n=1,...,|R|},Δp n is the convolution kernel position offset, |R| is the number of convolution kernel parameters; For alignment features Each position p0 has the following relationship: Among them, w(p n ) is the convolution kernel position p n The learnable parameters of n for The nth convolution kernel position; The convolution operation will be applied to non-integer positions p n +Δp n On, where Δp n It may represent a decimal. To solve this problem, the operation is realized by using the bilinear interpolation method. Then, a convolutional layer with kernel size is used to predict the sampling parameter |R|, which will be the number of output channels; finally, the deformable convolution operation is performed from Θ and Get Alignment Features In order to improve the transformation ability and adaptability of the module, dc Three additional deformation convolutional layers are added in both the previous and next stages; Finally, a 3×3 convolutional layer is used to reconstruct the low-resolution LR frame from the aligned feature maps. H×W is the frame size, C is the number of channels; In the LR reference frame and support frames After being fed back to the TDAN network, the aligned low-resolution image is finally obtained, and the super-resolution reconstruction network is further used to predict the high-resolution video frame.

5. The method for identifying birds in a substation based on video and audio recognition according to claim 1, characterized in that: Step (4) comprises: (401) using MFCC to obtain key feature vectors of the audio signal; (402) The key feature vector processed by MFCC is input into GRNN for training and classification to realize the recognition and classification of bird audio data.

6. The method for identifying birds in a substation based on video and audio recognition according to claim 5, characterized in that: Step (401) includes: First, pre-emphasis processing is applied to the audio signal. The pre-emphasis formula is: y[n]=x[n]-αx[n-1] Wherein, y[n] represents the pre-emphasized signal; x[n] is the nth sampling point of the original audio input signal; α represents the pre-emphasis coefficient, which is 0.97 or 0.98; The audio signal is framed. Framing is achieved by smoothly moving the selected window function on the audio signal and weighting it. The calculation formula is as follows: y w [n]=ω[n]×y[n] Among them, ω[n] represents the window function, and y[n] represents the frame signal; In the process of framing the audio signal, in order to ensure the smoothness of the acoustic feature parameters, a section of audio information is retained between two adjacent frames; In order to avoid spectrum leakage caused by using rectangular window, Hanning window is used to process the audio signal; The calculation formula of the window function of the Hanning window is as follows: Among them, ω han [n] represents the window function of the Hanning window, and N represents the length of the window.

7. The method for identifying birds in a substation based on video and audio recognition according to claim 6, characterized in that: Step (402) includes: The GRNN neural network structure includes an input layer, a pattern layer, a summation layer, and an output layer; the input received by the network and the output vector provided are X = [x1, x2, ..., x n ] T and Y=[y1,y2,...,y k ] T ; In the translation of input information, the output layer transmits information through neurons. The function of this layer is to process the bird audio signal transmitted by the input layer. The formula of the GRNN neural network output layer is as follows: Where X is the input vector; X i is the value of the input vector training sample corresponding to the i-th neuron; σ is the width coefficient of the Gaussian function, that is, the smoothing factor; The GRNN neural network summation layer mainly calculates the sum of the input corresponding to its internal neurons and the input corresponding to other neurons, that is, the input audio single signal is compared and fused; the calculation formula of the GRNN neural network summation layer is as follows: Among them, S R S is the weighted sum of the first type of neurons over all pattern layers; Ng is the weighted sum of the second type of neurons on all pattern layers; X is the input vector; X i is the value of the input vector training sample corresponding to the i-th neuron; σ is the width coefficient of the Gaussian function, that is, the smoothing factor; the summation layer mainly sums the output results of all neurons in the previous layer, and the connection weight of the first type of neurons and each neuron in the pattern layer is 1; the second type of neurons weighted sum the neurons of all pattern layers, and the connection weight is Y i ; After the neurons in all pattern layers perform weighted summation, the recognition results of the bird audio data are obtained.

8. The method for identifying birds in a substation based on video and audio recognition according to claim 6, characterized in that: Step (5) comprises: (501) The AVSR method is used to realize the task of audio and video modality fusion, the LM joint decoder is embedded in the joint decoding reasoning to achieve character-level prediction, and the LM decoding reasoning runs through the entire beam search; The position encoding PE is used to embed the position information in the input sequence of the encoder, and the position encoding is performed through sine and cosine functions. The specific method is as follows: Among them, pos represents the absolute position of the word in the sentence; i represents the dimension of the word vector, i = 0, 1, ..., 225; d = 512; (502) The audio features f extracted by the front-end model are encoded through a dual encoding channel. audio and video features f video The audio and video encoding structures are the same, and both use the Transformer self-attention encoding layer to encode the audio and video channels separately to obtain the audio stream encoding features: in, Represents audio data, i = 1, 2, 3, ..., m; m is the number of bird audio coding features; R d represents a d-dimensional vector; Video stream encoding characteristics: in, Represents audio data, i = 1, 2, 3, ..., n; n is the number of bird video coding features; The dimension of the encoded feature vector of the audio and video streams is 512; the audio and video joint features are output through the multimodal feature fusion network, where the attention weight distribution is obtained by the softmax function; the calculation formula of the multimodal feature fusion network is as follows: in, Represents audio coding features, Represents the video encoding characteristics, represents multimodal joint features; Q a , V a , Q v , V v The query matrix, key matrix, and value matrix represent audio and video respectively; represents the dot product scaling of the attention score function; Joint{·} represents the fusion of different modal features; cat(·) function is used to concatenate strings; Encoding audio features and video coding features It is sent to the multimodal feature fusion network, and after feature cascade processing, it is passed to the joint convolutional network to generate a multimodal joint feature. Its feature dimension is consistent with the encoding feature dimension of the input sound and image; finally, the comprehensive encoding features of audio and video will be sent to the decoding module; (503) The LM decoder is embedded in the Transformer decoder for joint decoding. The Transformer decoder is composed of 6 layers like the encoder. The LM is constructed by a multi-layer stacked LSTM network and is trained using the training set data. In AVSR, the original decoding method consists of the joint encoding feature vector at the current time t and the previous prediction result y t-1 The probability is calculated through the Softmax layer; LM is embedded in the original decoding method, and the joint decoding is performed with the Transformer decoder as follows: Among them, p dec Provided by Transformer decoder;p LM Provided by LM; t is the predicted output at time t; β is a hyperparameter.

9. A substation bird identification system based on video and audio recognition, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by a processor, the substation bird identification method based on video and audio recognition as described in any one of claims 1 to 8 is implemented.

Images