A method and device for tracking and identifying flying insects based on audio-visual collaboration

Abstract

The application discloses a flying insect tracking and identifying method and device based on audio-visual cooperation, relates to the flying insect tracking and identifying technical field, and aims to solve the problem that the prevention and treatment of harmful flying insects has been a thorny challenge, especially for greenhouse planting, if a small amount of sporadic pests enter the greenhouse, the breeding capacity of the pests will make the small amount of pests rapidly become a pest disaster, thereby bringing huge losses to breeders; the TCN network can effectively extract the characteristics of time sequence data, has better robustness and identification rate compared with artificial selection of characteristics or traditional machine learning methods; compared with traditional non-machine learning methods, the method drives the TCN network to learn characteristics through training data, and avoids incomplete utilization of data information caused by the artificial characteristic screening process.

Description

Technical Field

[0001] This invention relates to the field of flying insect identification and tracking technology, specifically to a flying insect tracking and identification method and device based on audio-visual collaboration. Background Technology

[0002] With the continuous advancement of agricultural production technology, more and more intelligent technologies are being applied to various aspects of agricultural planting and breeding. Among these, the control of harmful flying insects has always been a thorny challenge, especially for greenhouse and polytunnel cultivation. If a small number of sporadic pests enter the greenhouse and are not promptly removed and captured, their reproductive capacity can quickly turn a small number of pests into an infestation, leading to huge losses for farmers. Therefore, how to effectively detect sporadic harmful flying insects as early as possible has become an important issue. Currently, the main solution to this problem relies on the experience of growers and timely observation, which is very inefficient.

[0003] Existing target recognition methods using deep neural networks typically employ time-frequency graphs of signals as network input data. However, time-frequency graphs are a general algorithm and do not form features entirely based on data, which can lead to information loss. The method proposed in this invention directly extracts features from sound time series.

[0004] By combining image recognition and sound recognition, image recognition makes an initial judgment, and the decision on whether to initiate a second sound recognition judgment is based on the initial judgment result, thus obtaining a more accurate monitoring and judgment result.

[0005] By weighted fusion of image-based target object localization and sound-based target object localization, more accurate localization and tracking results can be obtained. Summary of the Invention

[0006] In view of the problems existing in the prior art, the present invention discloses a flying insect tracking and recognition device based on audio-visual collaboration. The technical solution adopted includes a high-definition camera and a microphone array. The high-definition camera and the microphone array are respectively equipped with a data preprocessing module, a CNN recognition network, a TCN classification network, and a trajectory fusion calculation.

[0007] As a preferred embodiment of the present invention, the data preprocessing module includes image data preprocessing and sound data preprocessing modules.

[0008] As a preferred embodiment of the present invention, the preprocessing of the image data includes convolutional layers and fully connected networks.

[0009] As a preferred embodiment of the present invention, the convolutional layer has multiple layers.

[0010] The present invention also discloses a method for tracking and identifying flying insects using the above-mentioned audio-visual collaborative method, which includes the following steps:

[0011] Step 1, Image data preprocessing

[0012] The image data captured by the high-definition camera is preprocessed. Based on the image's chromaticity components, it is determined whether the image is entirely white or black, and appropriate processing is performed. If so, the image is considered unrecognizable and is enhanced before use; otherwise, the image is considered usable directly. Step 2: CNN network.

[0013] Foreign object detection based on image data employs a CNN network to achieve relatively accurate foreign object identification. The CNN network outputs two states: normal and foreign object present. Then, it performs a detection and judgment process on the target region. The model identifies the image captured in the target direction region. When the model outputs an abnormal state, a foreign object is considered to be present in the current field of view. The CNN network's fault judgment involves several convolutional layers and a fully connected network in series. When the time-frequency distribution map is input to the first convolutional layer, it performs feature extraction. The extracted features are then input to the next convolutional layer for pooling, and this process is repeated several times to reduce dimensionality. After pooling in the last convolutional layer, the multiple sets of extracted two-dimensional features are split and concatenated into a vector, which is then input to the fully connected network. The fully connected network processes the vector and outputs the desired result. The two output states correspond to two nodes output by the fully connected network. When the output state is "foreign object present," the coordinates of the foreign object can be obtained based on its position in the image, and its trajectory can be determined based on the time.

[0014] Step 3, Audio Data Preprocessing Module

[0015] The audio signal is sampled at a sampling rate of 24kHz. N sampling points are extracted from the input audio data stream as a single frame of input data. N is a large value, usually requiring several seconds or more of data. The sampling time used for the N sampling points is denoted as T, which is the recording time of one frame of data. The mean and variance of these N data points are calculated, and the data frame is standardized as follows:

[0016]

[0017] Where s is the input time series, m is the mean, and σ is the standard deviation of the series;

[0018] Step 4, TCN network

[0019] The audio signal input is the output of the TCN module. The sampling rate of the revolution data only needs to satisfy that the number of sampling points within time T is consistent with the length of the feature sequence output by the TCN module 2 from the network.

[0020] TCN modules 1-3 have the same structure. The dilated convolution is formed by stacking two layers of causal convolution with a kernel size of 2. The BN layer, ReLU layer, and Dropout layer are the conventional batch normalization layer, ReLU activation function layer, and dropout layer, respectively. The 1x1 convolution layer is directly connected to the ReLU activation function layer at the module output to form a residual connection, which can avoid network degradation.

[0021] Feature fusion: The output of TCN module 2 is cross-combined with the foreign object size sequence to form a new feature sequence as the output of feature fusion. The cross-combination method is that the sequence input by TCN module 2 is the odd sequence of the output sequence, and the foreign object size data sequence is the even sequence of the output sequence.

[0022] First, the labeled dataset with a sampling rate of 24KHz and a length of N points is normalized, and then the TCN network designed in this invention is trained. The sound signal to be recognized is input into the TCN network designed in this invention to output the coordinates of the flying insect, thereby forming the flight trajectory of the flying insect.

[0023] Step 5, Trajectory Fusion

[0024] After obtaining the trajectory of the foreign object using two methods, CNN network image recognition and TCN network sound recognition, they are then weighted with a weight of 0.5 to obtain the final trajectory of the foreign object.

[0025] The beneficial effects of this invention are as follows: 1. The TCN network designed in this invention can effectively extract features from time series data, and has better robustness and recognition rate compared with manual feature selection or traditional machine learning methods; compared with traditional non-machine learning methods, the method of this invention drives the TCN network to learn features through training data, avoiding the incomplete utilization of data information caused by the manual feature selection process.

[0026] 2. The TCN network design incorporates the fusion of the target object's size data sequence and the intermediate abstract features of the sound signal, and further extracts features from the fused information. This design fully considers the correlation between the size of the target object and its flight sound signal, which is more conducive to the mining of target features and can improve recognition efficiency.

[0027] 3. The motion trajectory of the target object is calculated by using a CNN network for image recognition and a TCN network for sound recognition. The estimated values ​​of the two motion trajectories are weighted and fused to obtain a more accurate motion trajectory. Attached Figure Description

[0028] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, the elements or parts are not necessarily drawn to actual scale.

[0029] Figure 1 This is a schematic diagram illustrating the steps and principles of the present invention;

[0030] Figure 2 This is a schematic diagram of the CNN network structure of the present invention;

[0031] Figure 3 This is a schematic diagram of the overall structure of the TCN network of the present invention;

[0032] Figure 4 This is a schematic diagram of the convolutional layer structure of the present invention;

[0033] Figure 5 This is a schematic diagram of the output and input structure of the present invention;

[0034] Figure 6 This is a schematic diagram of the feature fusion structure of the present invention. Detailed Implementation

[0035] The technical solution of the present invention will now be clearly and completely described in conjunction with the accompanying drawings. In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0036] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0037] Example 1

[0038] like Figures 1 to 6As shown, this invention discloses a method and device for tracking and identifying flying insects based on audio-visual collaboration. The technical solution adopted is a fusion decision mechanism using dual deep learning neural networks designed for the special application scenario of tracking and identifying flying insects. The method first uses a pre-trained CNN recognition network model to make an initial decision on the image within the field of view captured by the high-definition camera. When a foreign object appears in the image, a second decision is initiated. The second decision uses a pre-trained TCN classification network model to extract and identify the deep features of the sound signal picked up by the microphone array, and then identifies whether the foreign object is the target flying insect and calculates its real-time orientation. At the same time, the image data preprocessing is driven to calculate the orientation and trajectory of the foreign object based on its position in the image, and then weighted and fused with the trajectory calculated by the sound data preprocessing module to obtain a more accurate orientation and trajectory result.

[0039] By fusing the insect size data variable with the abstract features of the sound signal extracted from the TCN network, i.e., the output of TCN module 2, the features are fused, and then the fused features are further extracted.

[0040] The motion trajectory of the target object is calculated by using a CNN network for image recognition and a TCN network for sound recognition. The estimated values ​​of the two motion trajectories are then weighted and fused to obtain a more accurate motion trajectory.

[0041] The present invention also discloses a method for tracking and identifying flying insects using the above-mentioned audio-visual collaborative method, which includes the following steps:

[0042] Step 1, Image data preprocessing

[0043] The image data captured by the high-definition camera is preprocessed. Based on the color components of the image, it is determined whether the image is close to white or black. If so, the image is considered unrecognizable and is enhanced before use. If not, the image is considered usable directly.

[0044] Step 2, CNN network

[0045] Foreign object detection based on image data employs a CNN network to achieve relatively accurate foreign object identification. The CNN network outputs two states: normal and foreign object present. Then, it performs target region detection, performing model recognition on images captured in the target direction region. When the model outputs an abnormal state, a foreign object is considered to be present in the current field of view. The CNN network's fault judgment involves several convolutional layers and a fully connected network in series. When the time-frequency distribution map is input to the first convolutional layer, it performs feature extraction. The extracted features are then input to the next convolutional layer for pooling, and this process is repeated several times to reduce dimensionality. After pooling in the last convolutional layer, the final output of multiple sets of two-dimensional extracted features is split and concatenated into a vector, which is then input to the fully connected network. The fully connected network processes the data and outputs the desired result.

[0046] Furthermore, the two output states correspond to the two nodes output by the fully connected network; when the output state is "there is a foreign object", the coordinates of the foreign object can be obtained based on its position in the image, and the trajectory of the foreign object can be obtained based on the time.

[0047] Each convolutional layer includes three steps: convolution, activation, and pooling. There are four convolutional layers: a first convolutional layer, a second convolutional layer, a third convolutional layer, and a fourth convolutional layer. The first convolutional layer has 16x16 kernels (64 kernels total) with a stride of 2, and uses ReLU as its activation function. Its pooling function is 2x2 with a stride of 2. The second convolutional layer has 8x8 kernels (128 kernels total) with a stride of 2. The activation function of the second convolutional layer... The activation function of the third convolutional layer also uses ReLU; the pooling of the second convolutional layer also uses 2x2 with a stride of 2; the kernel of the third convolutional layer is 4x4 with 128 kernels and a stride of 1; the activation function of the third convolutional layer also uses ReLU; the pooling of the third convolutional layer also uses 2x2 with a stride of 2; the kernel of the fourth convolutional layer is 2x2 with 128 kernels and a stride of 1; the activation function of the fourth convolutional layer is also ReLU; the pooling of the fourth convolutional layer uses 2x2 with a stride of 2.

[0048] A fully connected network consists of two fully connected layers connected in series, and each fully connected layer uses 2048 neurons.

[0049] The formula for convolution is:

[0050]

[0051] Where, x i,j c represents the time-frequency distribution of the input participating in the convolution. i,j Represents the convolution kernel;

[0052] The kernel size is calculated by the backpropagation algorithm during the training phase. In this embodiment, the kernel size of the first convolutional layer is 16*16, the kernel size of the second convolutional layer is 8*8, the kernel size of the third layer is 4*4, and the kernel size of the fourth layer is 2*2.

[0053] Step 3, Audio Data Preprocessing Module

[0054] The audio signal is sampled at a sampling rate of 24kHz. N sampling points are extracted from the input audio data stream as a single frame of input data. N is a large value, usually requiring several seconds or more of data. The sampling time used for the N sampling points is denoted as T, which is the recording time of one frame of data. The mean and variance of these N data points are calculated, and the data frame is standardized as follows:

[0055]

[0056] Where s is the input time series, m is the mean, and σ is the standard deviation of the series;

[0057] Step 4, TCN network

[0058] The audio signal input is the output of the TCN module. The sampling rate of the revolution data only needs to satisfy that the number of sampling points within time T is consistent with the length of the feature sequence output by the TCN module 2 from the network.

[0059] TCN modules 1-3 have the same structure, such as Figure 4 As shown, the dilated convolution is formed by stacking two layers of causal convolution, with a kernel size of 2, as... Figure 4 The BN layer, ReLU layer, and Dropout layer shown are the conventional batch normalization layer, ReLU activation function layer, and dropout layer, respectively; the 1x1 convolutional layer is directly connected to the ReLU activation function layer at the module output by the input, forming a residual connection, which can avoid network degradation.

[0060] Feature fusion, such as Figure 5 As shown, the output of TCN module 2 is cross-combined with the foreign object size sequence to form a new feature sequence as the output of feature fusion. The cross-combination method is that the sequence input by TCN module 2 is the odd sequence of the output sequence, and the foreign object size data sequence is the even sequence of the output sequence.

[0061] First, the labeled dataset with a sampling rate of 24KHz and a length of N points is normalized, and then the TCN network designed in this invention is trained. The sound signal to be recognized is input into the TCN network designed in this invention to output the coordinates of the flying insect, thereby forming the flight trajectory of the flying insect.

[0062] The Softmax layer is essentially a softmax function, as shown in the following formula:

[0063]

[0064] Where y i The input vector is the softmax layer. The purpose of softmax is to map a vector whose elements can take any value to a vector where each element belongs to (0,1) and the sum of all values ​​is 1. This can be understood as a probability. The fully connected layer is where each node is connected to all nodes in the previous layer, and it is used to combine the features extracted from the previous layer. Due to its fully connected nature, the fully connected layer usually has the most parameters.

[0065] Normalization involves dividing the signal by a fixed value, such as the peak value. The calculation formula is as follows:

[0066]

[0067] The training process of TCN network is similar to that of other neural networks. The basic network structure consists of a network identical to the recognition process and an error backpropagation mechanism. During training, the training dataset is sequentially input into the network input. The network calculates and obtains the recognition result. The error between the recognition result and the label of the input training sample is then used to correct the parameters of each layer of the network in reverse order. This process continues until the error between the recognition result and the label of the input sample reaches a predetermined threshold. The parameters of each layer of the network constitute the trained TCN network model.

[0068] Step 5, Trajectory Fusion

[0069] After obtaining the trajectory of the foreign object using two methods, CNN network image recognition and TCN network sound recognition, they are then weighted with a weight of 0.5 to obtain the final trajectory of the foreign object.

[0070] For time t, the coordinates of the flying insect obtained from the image data through a CNN network are: The coordinates of the flying insect obtained by using sound data through a TCN network are as follows in This represents the horizontal angle of the insect's position at time t, obtained from the image data. The pitch angle represents the position of the flying insect at time t, obtained from the image data. This represents the horizontal angle of the insect's position obtained using sound data through a TCN network. This represents the pitch angle of the flying insect's position obtained using sound data through a TCN network; the weighting formula is:

[0071]

[0072]

[0073] Since t represents time, having the insect's position coordinates at each time point naturally yields its trajectory.

[0074] The working principle of this invention is as follows: A high-definition camera and microphone array are used to collect high-definition images and sound signals within the monitored field of view. Flying insects that secretly enter the monitored field of view are recorded by the camera, and the sound signals they generate during flight are recorded by the microphone array. Convolutional neural networks are used to initially identify the foreign object within the target field of view, obtaining a signal indicating whether secondary sound confirmation is needed. If necessary, a TCN network combined with causal convolution and dilated convolution is used to extract features from the picked-up sound signals. The correlation between the sound signal and the flight sound of the target flying object is specifically considered. The abstract features learned by the network are combined with rotational speed information for further feature learning, further confirming whether it is the target flying object and obtaining its trajectory. This method can fully utilize data information and avoid information loss caused by manual feature selection.

[0075] The circuit connection involved in this invention is a common method used by those skilled in the art, and technical inspiration can be obtained through a limited number of experiments. It is common knowledge.

[0076] Components not described in detail in this article are existing technologies.

[0077] While the specific embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention, and modifications or variations without creative effort are still within the protection scope of the present invention.

Claims

1. A method for tracking and identifying flying insects based on audio-visual collaboration, characterized in that: Includes the following steps: Step 1, Image data preprocessing The image data captured by the high-definition camera is preprocessed, and the color components of the image are used to determine whether the overall image is close to white or black, and then appropriate processing is performed. Step 2, CCN network Foreign object detection based on image data employs a CNN network to achieve relatively accurate foreign object recognition. The output of the CNN network has two states: normal and foreign object present. Then, the target area is judged and detected. Step 3, Audio Data Preprocessing Module The audio signal is sampled at a sampling rate of 24kHz. N sampling points are extracted from the input audio data stream as a single frame of input data. N is a large value, requiring several seconds or more of data. The sampling time used for N sampling points is denoted as T, which is the recording time of one frame of data. Calculate the mean and variance of these N data points, and then standardize the data frame as follows: ； Where s is the input time series, and m is the mean. The standard deviation of the sequence; Step 4, TCN network The audio signal input is the output of the TCN module. The sampling rate of the revolution data only needs to satisfy that the number of sampling points within time T is consistent with the length of the feature sequence output by the TCN module 2 from the network. Feature fusion: The output of TCN module 2 is cross-combined with the foreign object size sequence to form a new feature sequence as the output of feature fusion. The cross-combination method is that the sequence input by TCN module 2 is the odd sequence of the output sequence, and the foreign object size data sequence is the even sequence of the output sequence. First, the labeled dataset with a sampling rate of 24KHz and a length of N points is normalized, and then the designed TCN network is trained. The sound signal to be recognized is input into the designed TCN network to output the coordinates of the flying insect, thereby forming the flight trajectory of the flying insect. Step 5, Trajectory Fusion After obtaining the trajectory of the foreign object using both CNN image recognition and TCN sound recognition methods, a weighted sum is applied to obtain the final trajectory. For time t, the coordinates of the flying insect obtained from the image data via the CNN network are: The coordinates of the flying insects obtained using sound data through a TCN network are: ,in This represents the horizontal angle of the insect's position at time t, obtained from the image data. The pitch angle represents the position of the flying insect at time t, obtained from the image data. This represents the horizontal angle of the insect's position obtained using sound data through a TCN network. This represents the pitch angle of the flying insect's position obtained using sound data through a TCN network; the weighting formula is: ， ； t represents time, and the weights are 0.

5.

2. A device for tracking and identifying flying insects based on audio-visual collaboration, used to implement the method for tracking and identifying flying insects based on audio-visual collaboration as described in claim 1, characterized in that: It includes a high-definition camera and a microphone array. The high-definition camera and the microphone array are respectively equipped with a data preprocessing module, a CNN recognition network, a TCN classification network, and a trajectory fusion calculation module. The data preprocessing module includes image data preprocessing and sound data preprocessing modules. The high-definition camera captures images within its field of view, which are initially judged using a pre-trained CNN recognition network. When a foreign object appears in the image, a secondary judgment is initiated. The secondary judgment uses a pre-trained TCN classification network model to extract and recognize the deep features of the sound signals picked up by the microphone array. Then, it identifies whether the foreign object is a target flying insect and calculates its real-time location. At the same time, it drives the image data preprocessing to calculate the location and trajectory of the foreign object based on its position in the image. This is then weighted and fused with the trajectory calculated by the sound data preprocessing module to obtain a more accurate location and trajectory result.

3. The insect tracking and identification device based on audio-visual coordination according to claim 2, characterized in that: The preprocessing of the image data includes convolutional layers and fully connected networks.

4. The insect tracking and identification device based on audio-visual coordination according to claim 3, characterized in that: The convolutional layer has multiple layers.

Images