Crop pest detection and early warning system based on fruit fly perception neural network

Abstract

The invention discloses a crop pest detection and early warning system based on a fruit fly perception neural network in the field of pest monitoring and early warning, the crop pest detection and early warning system comprises a pre-synaptic nervous system network and a post-synaptic nervous system network, the pre-synaptic nervous system network is used for processing video information, and the post-synaptic nervous system network is used for constructing an early warning monitoring model. The visual field information is processed based on biological inspiration of a fruit fly visual system network. The APDNNM visual neural network model is used for solving the technical problem of pest detection in intelligent video monitoring. According to the model, spike neuronal cells are constructed and synthesized on the basis of a response characteristic mechanism of feedforward neuronal cells and LPTC neurons of a fruit fly visual nervous system to sense pest information existing in crops, and early warning detection is achieved.

Description

Technical Field

[0001] The invention belongs to the field of pest monitoring and early warning, and in particular relates to a crop pest detection and early warning system based on a fruit fly perception neural network. Background Art

[0002] During the growth period of crops, pests are sudden and cause great harm to a large area. Timely detection and control of pests and active prevention and control measures can ensure the yield and quality of crops. At present, pests are mainly identified manually, and the degree of intelligent detection is low. Therefore, an effective pest early warning detection system is needed to detect pests in crops in a timely manner.

[0003] In recent years, domestic and foreign researchers have applied machine vision algorithms to the field of agricultural pest identification and detection. One method is to place sticky insect boards at fixed positions to capture pests and automatically identify them, while using machine vision methods such as automatic threshold segmentation, mark adhesion processing, target recognition, and biological feature interference removal to provide a basis for pest control. There are also clustering and filtering methods to identify pests, or multiple PCR methods to detect and identify the insect stages of grain borers, grain weevils, rice weevils, corn weevils, and wheat moths.

[0004] There are several pest detection models: 1. An improved RetinaNet automatic detection model for rice canopy pests. ResNeXt101 is used as the feature extraction network in the model. The average accuracy of the model for detecting two pest-damaged areas is 93.76%; 2. The Convolutional Neural Networks (CNN) model is used to identify dozens of pests and diseases in crops, achieving good detection results; 3. The terahertz spectrometer is equipped with a reflective imaging module to perform terahertz layer imaging on the internal structure of wheat grains. This method can effectively identify the insect-infested areas inside the wheat grains, but it fails to achieve online monitoring of insect-infested damage to accumulated wheat grains; 4. A real-world granary pest image dataset SGI-6 is established, and the clustering algorithm is used to improve the region proposal network of the FasterR-CNN model to identify six common stored grain pests, with an average accuracy (mAP) of 96.63%.

[0005] The above methods for identifying and detecting pests are based on digital image processing technology and deep learning methods. They have the disadvantages of relying on prior rules, sample training rules, single scenes and large computing resources. Research on such problems requires seeking new technological breakthroughs. The visual nervous system of animals has adapted to the actual scene in the long-term evolution of response detection neurons with special preferences for targets. For example, the visual system of fruit flies perceives visual information in the external environment and can obtain the behavioral changes in the intensity of light and dark in the field of view. It uses the lobula plate tangential cells (LPTC) in the neural layer to detect the direction of movement of the target and realize the detection and tracking of the target. However, there is no related research report on the construction of crop pest target detection technology model with the help of biological inspiration. Therefore, it is particularly important to explore and design an early warning detection model for the appearance of pests in crops in visual scenes with the help of the fruit fly visual nervous system. Summary of the invention

[0006] The present invention aims to provide a crop pest detection and early warning system based on a fruit fly perception neural network to solve the problems of existing pest detection methods, such as reliance on prior rules, sample training rules, single scenarios and large computing resources.

[0007] The crop pest detection and early warning system based on the fruit fly perception neural network in this scheme includes a presynaptic neural network for processing video information and a postsynaptic neural network for building an early warning monitoring model;

[0008] The presynaptic neural network comprises the following steps:

[0009] Step 1: Set up Photoreceptor and Retina neural layers. The Photoreceptor neural layer is used to receive video information, and the Retina neural layer performs noise reduction on the information received by the Photoreceptor neural layer. The specific calculation formula is as follows:

[0010]

[0011] Step 2: A Lamina neural layer is provided, and the information output amount obtained by the Lamina neural layer using shunt inhibition processing is calculated as follows:

[0012]

[0013] Step 3: A Medulla neural layer is provided. The Medulla neural layer mainly summarizes the information output of each node of the Lamina neural layer. The calculation formula is as follows:

[0014] m t (u,v)=χt (3r+1,3s+1)×χ t-1 (3r+u,3s+v)-χ t-1 (3r+1,3s+1)×χ t (3r+u,3s+v)

[0015] The presynaptic neural network comprises the following steps:

[0016] Step 4: Provide an LPTC neural layer, the LPTC neural layer receives the output of information in the presynaptic neural system, and designs an adaptive threshold according to the LPTC output. The calculation formula is as follows:

[0017]

[0018] Step 5: An FFC neural layer is provided, which receives the change of the light intensity of the target information in the Photoreceptor neural layer and eliminates the noise interference in the original input information. The calculation formula is as follows:

[0019]

[0020] Step 6: An SSC neural layer is provided. The SSC neural layer constructs an early warning model based on the information output by LPTC and FFC, and then decides when to issue an early warning signal. The calculation formula is as follows:

[0021]

[0022] Further, in step 1, Z t is the output membrane potential of the R layer cell (i, j) at the tth frame; P t , P t-1 and P t-2 The pixel values ​​of the frame at time t, the frame at time t-1, and the frame at time t-2 are stored respectively.

[0023] Further, in step 4, σ t (th) is the threshold value, t is the current frame image. When pests appear in agriculture, the output of the leaflet plate tangential cells will be triggered. When it reaches a certain level, a spike signal will appear:

[0024]

[0025] 1 indicates a spike signal occurs.

[0026] Further, in step 5, ij The smoothing filter kernel can be expressed as:

[0027] The average grayscale value of frame t can be obtained by the following formula:

[0028]

[0029] Then, the average gray value obtained by the above formula is smoothed by the following formula:

[0030]

[0031] Furthermore, in the formula, n c is the time step of the image frame, k l =Sigmoid(n c xl) is the persistence coefficient of the time step, and finally the output of FFC is normalized:

[0032] S t (FFC) = 2(sigmoid(F t (FFC))-0.5)

[0033] The spike signal sent by the feedforward cell at frame time t can be obtained by the following formula:

[0034]

[0035] Among them, T FFC is a dynamic threshold, T FFC =T FO +α FFC ×S t (FFC), T FO and α FFC is a given constant factor.

[0036] Further, in step six, T SSC The dynamic detection threshold is obtained by using the above detection threshold scheme, SSC t The output behavior of FFC neurons is obtained:

[0037]

[0038] Where n s The number of consecutive spike signals in a period of time on the table. t (m) is obtained from the output of the M neural node cells, and its output is normalized:

[0039]

[0040] Where 1 indicates that a warning detection signal is issued.

[0041] Beneficial effects: The system is based on the biological inspiration of the fruit fly visual system network. After processing the video image sequence taken by the monocular camera to obtain the target information, it is sent to the system network model to process the changes in the spatiotemporal position information membrane potential excitation, and the visual field information is processed to judge the information of the pests.

[0042] This model constructs synthetic spike neurons based on the feedforward neurons and LPTC neurons response characteristics of the fruit fly visual nervous system to perceive pest information in crops and realize early warning detection. The system can effectively detect the presence of pests, greatly reducing the risk of crop damage, with small errors and high overall stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is the internal structure diagram of the APDNNM of the crop pest detection and early warning system based on the fruit fly perception neural network of the present invention;

[0044] Figure 2 This is a screenshot of a video actually shot during the performance test of the present invention;

[0045] Figure 3 This is a screenshot of a video actually shot during the performance test of the present invention;

[0046] Figure 4 This is a screenshot of a video actually shot during the performance test of the present invention;

[0047] Figure 5 This is a screenshot of a video actually shot during the performance test of the present invention;

[0048] Figure 6 This is a screenshot of a video actually shot during the performance test of the present invention;

[0049] Figure 7 This is a screenshot of a video actually shot during the performance test of the present invention;

[0050] Figure 8 This is a screenshot of a video actually shot during the performance test of the present invention;

[0051] Fig. 9 This is a screenshot of a video actually shot during the performance test of the present invention;

[0052] Fig.10 for Figure 2 The performance test corresponding to the video shown;

[0053] Fig.11 for Figure 3 Performance tests corresponding to the video screenshots shown;

[0054] Fig.12 for Figure 4 Performance tests corresponding to the video screenshots shown;

[0055] Fig.13 for Figure 5 Performance tests corresponding to the video screenshots shown;

[0056] Fig.14 for Figure 6 Performance tests corresponding to the video screenshots shown;

[0057] Fig.15 for Figure 7 Performance tests corresponding to the video screenshots shown;

[0058] Fig.16 for Figure 8 Performance tests corresponding to the video screenshots shown;

[0059] Fig.17 for Fig. 9 Performance tests corresponding to the video screenshots shown. DETAILED DESCRIPTION

[0060] The following is further described in detail through specific implementation methods:

[0061] Crop pest detection and early warning system based on fruit fly perception neural network:

[0062] 1. Drosophila visual nervous system pathway

[0063] The neurophysiological theories of the functional structure, organ organization, etc. of the fruit fly visual nervous system were reported. The fruit fly visual nervous system consists of two compound eyes, each with more than 3,500 small eyes.

[0064] The specific contents of the visual nervous system pathway of Drosophila are as follows:

[0065] 1. Fruit flies use the perception layer of the visual nervous system (photoreceptor, P) to perceive changes in the intensity of light of external target information;

[0066] 2. Transmit the information difference to the retinal layer (Retina, R), eliminate the redundant information noise, and transmit the signal to the thin film layer (Lamina, L) after depolarization processing;

[0067] 3. The thin film layer (Lamina, L) uses convolutional projection to obtain the output neural information of the cart node. The oo node receives the information of the cart node in the corresponding layer and uses shunt inhibition or lateral inhibition to obtain the output behavior quantity.

[0068] 4. The m1 node in the medulla (Medulla, M) uses a symmetrical lateral inhibition mechanism and a motion direction detector to obtain the target local motion output for the behavior of the oo node, and the m2 node obtains the target motion output behavior by accumulation;

[0069] 5. Lobula plate tangential cells (LPTC) have the function of detecting target direction, including spike transmission neurons H2, small target motion detection neurons LC11 and LC16, etc. Each LPTC neuron has a response characteristic to a specific motion pattern and direction, and transmits its output behavior to the brain nervous system, providing an important reference for fruit flies to detect targets.

[0070] 2. Artificial Vision System Design

[0071] Agricultural pests mainly appear on the roots and leaves of plants, biting the roots and leaves of crops with slow movements, causing the plants to die within a certain period of time. Timely discovery and early treatment are necessary. Therefore, based on the biological mechanism that the fruit fly visual nervous system can detect the changing behavior of moving targets in real time, this paper designs and implements a bionic agricultural pest detection neural network model (APDNNM).

[0072] APDNNM processes the video image sequence taken by the monocular camera to obtain the target information, and then sends it to the system network model for the changes in the spatiotemporal position information membrane potential excitation. The internal structure of the visual neural system network model proposed in this paper is as follows: Figure 1 shown.

[0073] APDNNM consists of presynaptic and postsynaptic neural systems respectively. The design details of the model are described as follows.

[0074] 2.1 Presynaptic neural network

[0075] The presynaptic neural network of APDNNM includes four neural layers: Photoreceptor, Retina, Lamina and Medulla, which simulate the structural characteristics of the visual nervous system of fruit flies. The video image sequence is input, and the membrane potential excitation generated by the target information is processed layer by layer. The functional design of each neural layer is as follows.

[0076] 2.1.1 Photoreceptor and Retina neural layer

[0077] The photoreceptor and retina neural layers (hereinafter referred to as P layer and R layer) are composed of M×N photoreceptor cells, each of which corresponds to a pixel of the input image. The P layer uses the target detection and recognition algorithm to obtain the target information P(i,j), and divides the M×N photoreceptor cells in the R layer into M=8m and N=8n. The target information is smoothed and filtered to eliminate the influence of noise, thereby obtaining the output of the corresponding sub-block.

[0078]

[0079] In the formula, Z t is the output membrane potential of the R layer cell (i, j) at the tth frame; P t , P t-1 and P t-2 The pixel values ​​of the frame at time t, the frame at time t-1, and the frame at time t-2 are stored respectively.

[0080] 2.1.2 Lamina neural layer

[0081] The Lamina neural layer (hereinafter referred to as the L layer) is composed of the cartridge node (cart) and the switch node (On-Off, oo) layer, and the input of the (p, q) cells in the cart node layer sub-block.

[0082]

[0083] Among them, w uv is a 3x3 Gaussian convolution kernel:

[0084] Where 1≤p, 1≤q, 1≤i≤m, 1≤j≤n, the output of the cart node cell at position (r,s) is output via the following formula:

[0085] E t (r,s)=X(r,s)-X T-1 (r,s) (3)

[0086] In the formula, 1≤r≤6m, 1≤s≤6n. The oo node cell receives the input from the cart node cell, and then obtains the output of the oo node cell through shunt inhibition:

[0087]

[0088] In the formula, A l is a positive attenuation coefficient, B l is the positive excitatory activity, γ t (r,s) is the tilt control of cart node cell (r,s).

[0089] 2.1.3 Medulla neural layer

[0090] The Medulla neural layer (hereinafter referred to as the M layer) is composed of 2m×2n sub-block node cells arranged in sequence. The oo node cell (r, s) obtains the output of the node cell (r, s) of the 2mx2n sub-block with the help of the associated motion detector EMD.

[0091] m t (u,v)=χt (3r+1,3s+1)×χ t-1 (3r+u,3s+v)-χ t-1 (3r+1,3s+1)×χ t (3r+u,3s+v) (5)

[0092] Where, 0≤(u,v)≤2, and (u,v)≠(1,1), the Sigmoid function is used for normalization to obtain the output behavior of the node cell (r,s):

[0093]

[0094] 2.2 Postsynaptic neural network

[0095] The postsynaptic neural system of APDNNM is composed of lobula plate tangential cells (LPTC), feedforward cells (FFC) and synthesized spike cells (SSC) to form a detection neural network model. The functional design is as follows.

[0096] 2.2.1LPTC neural layer

[0097] like Figure 1 As shown, the lobule plate tangential cell LPTC receives the output of the cell in the presynaptic nervous system, and the threshold designed according to the LPTC output needs to change adaptively.

[0098]

[0099] In the formula, σ t (th) is the threshold value, t is the current frame image, Y t and Calculated by the following formula:

[0100]

[0101] Where l is the time step of the image frame. When pests appear in agriculture, the output of the leaflet tangential cells will be triggered, and a spike signal will appear when it reaches a certain level:

[0102]

[0103] 1 indicates a spike signal occurs.

[0104] 2.2.2 FFC neural layer

[0105] The FFC neural layer receives the change of the light intensity of the target information in the P neural layer, suppresses the noise interference of the target information caused by the environment, and the information Ft(i,j) after smoothing filtering by FFC is:

[0106]

[0107] In the formula, w ij The smoothing filter kernel can be expressed as:

[0108] The average grayscale value of frame t can be obtained by the following formula:

[0109]

[0110] Then, the average gray value obtained by the above formula is smoothed by the following formula:

[0111]

[0112] Where n c is the time step of the image frame, k l =Sigmoid(n c xl) is the persistence coefficient of the time step, and finally the output of FFC is normalized:

[0113] S t (FFC) = 2(sigmoid(F t (FFC))-0.5) (14)

[0114] The spike signal sent by the feedforward cell at frame time t can be obtained by the following formula:

[0115]

[0116] Among them, T FFC is a dynamic threshold, T FFC =T FO +α FFC ×S t (FFC), T FO and α FFC is a given constant factor.

[0117] 2.2.3 SSC cells

[0118] The SSC generates its own output and spike signal with the help of the output and spike signal of the FFC neuron cells, and then determines when the APDNNM issues a warning signal. The spike signal of the synthetic spike cell is generated by the following formula:

[0119]

[0120] Where, T SSCThe dynamic detection threshold is obtained by using the above detection threshold scheme, SSC t The output behavior of FFC neurons is obtained:

[0121]

[0122] Where n s The number of consecutive spike signals in a period of time on the table. t (m) is obtained from the output of the M neural node cells, and its output is normalized:

[0123]

[0124] In the formula, 1 indicates that APDNNM issues a warning detection signal.

[0125] 3. Numerical Experiments

[0126] The experiment was implemented on a computer with CPU / 3.60GHz, RAM / 8.00GB, and Win10. The program code was written in C++ and VisualStudio2019. The test video image sequences were all 35frame / s, and the image sequence size of the input neural network was an 8-bit grayscale image of 128×128 pixels. The parameter settings of APDNNM are shown in Table 1.

[0127] Table 1 APDNNM parameter settings

[0128]

[0129] 3.1 Experimental analysis

[0130] 3.1.1 Test Video

[0131] To verify the effectiveness of APDNNM, crops such as sorghum, corn, and camellia were involved, and video image sequences were taken to record the behavioral activities of pests; Figure 2-9 The video screenshots shown include 4 pest scenes and 4 pest-free scenes. Comparative experimental analysis is carried out to verify that the APDNNM proposed in this paper has superiority in early warning detection tasks in different scenarios.

[0132] 3.1.2 Performance Analysis of APDNNM

[0133] Will Figure 2-9 The actual video captured in the video screenshots is used for performance testing. Figure 2 , 3 , 6, 7 Videos contain information about pests, Figure 4 , 5 , 8, and 9 contain no pest information.

[0134] in Figure 2 The performance test corresponding to the video shown is as follows Fig.10 As shown;

[0135] in Figure 3 The performance test corresponding to the video shown is as follows Fig.11 As shown;

[0136] in Figure 4 The performance test corresponding to the video shown is as follows Fig.12 As shown;

[0137] in Figure 5 The performance test corresponding to the video shown is as follows Fig.13 As shown;

[0138] in Figure 6 The performance test corresponding to the video shown is as follows Fig.14 As shown;

[0139] in Figure 7 The performance test corresponding to the video shown is as follows Fig.15 As shown;

[0140] in Figure 8 The performance test corresponding to the video shown is as follows Fig.16 As shown;

[0141] in Fig. 9 The performance test corresponding to the video shown is as follows Fig.17 shown.

[0142] From the above, we know that the APDNNM early warning detection model of videos 2 and 3 and videos 6 and 7 also has omissions and false alarms. This is because the biological characteristics have delayed characteristics, which leads to misjudgment due to the delayed output of the neural network; videos 4, 5 and 9 have no pests, and the APDNNM early warning detection model can accurately detect them; video 8 has no pests, but due to the high light intensity and noise interference, the APDNNM early warning detection model has a short-term false alarm phenomenon, which is much lower than the warning state when there are pests. It can be seen that the APDNNM early warning detection model has a certain stability.

[0143] IV. Conclusion

[0144] Based on the biological inspiration of the fruit fly visual system network, the visual field information is processed. An APDNNM visual neural network model is proposed to solve the technical problem of pest detection in intelligent video surveillance. The model is based on the feedforward neuron cells and LPTC neuron response characteristics of the fruit fly visual nervous system to construct synthetic spike neuron cells to perceive the pest information in crops and realize early warning detection.

[0145] Further research work will be carried out in the future. In order to apply it in a practical environment, APDNNM will be embedded in the ARMv8 development board, and a drone equipped with the development board will be used to perform early warning detection of pests.

[0146] The above is only an embodiment of the present invention, and the common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. Crop pest detection and early warning system based on fruit fly perception neural network, Features: It includes a presynaptic neural network for processing video information and a postsynaptic neural network for building an early warning monitoring model; The presynaptic neural network comprises the following steps: Step 1: Set up Photoreceptor and Retina neural layers. The Photoreceptor neural layer is used to receive video information, and the Retina neural layer performs noise reduction on the information received by the Photoreceptor neural layer. The specific calculation formula is as follows: Step 2: A Lamina neural layer is provided, and the information output amount obtained by the Lamina neural layer using shunt inhibition processing is calculated as follows: Step 3: A Medulla neural layer is provided. The Medulla neural layer mainly summarizes the information output of each node of the Lamina neural layer. The calculation formula is as follows: <h2 style=";text-align:left;direction:ltr">m<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (u,v)=χ<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (3r+1,3s+1)×χ<h2 style=";text-align:left;direction:ltr"> t-1 <h2 style=";text-align:left;direction:ltr"> (3r+u,3s+v)-χ<h2 style=";text-align:left;direction:ltr"> t-1 <h2 style=";text-align:left;direction:ltr"> (3r+1,3s+1)×χ<h2 style=";text-align:left;direction:ltr"> t <h2 style=";text-align:left;direction:ltr"> (3r+u,3s+v) The presynaptic neural network comprises the following steps: Step 4: Provide an LPTC neural layer, the LPTC neural layer receives the output of information in the presynaptic neural system, and designs an adaptive threshold according to the LPTC output. The calculation formula is as follows: Step 5: An FFC neural layer is provided, which receives the change of the light intensity of the target information in the Photoreceptor neural layer and eliminates the noise interference in the original input information. The calculation formula is as follows: Step 6: An SSC neural layer is provided. The SSC neural layer constructs an early warning model based on the information output by LPTC and FFC, and then decides when to issue an early warning signal. The calculation formula is as follows:

2. The crop pest detection and early warning system based on fruit fly perception neural network according to claim 1, Features: In step 1, Z t is the output membrane potential of the R layer cell (i, j) at the tth frame; P t , P t-1 and P t-2 The pixel values ​​of the frame at time t, the frame at time t-1, and the frame at time t-2 are stored respectively.

3. The crop pest detection and early warning system based on fruit fly perception neural network according to claim 1, Features: In step 4, σ t (th) is the threshold value, t is the current frame image. When pests appear in agriculture, the output of the leaflet plate tangential cells will be triggered. When it reaches a certain level, a spike signal will appear: 1 indicates a spike signal occurs.

4. The crop pest detection and early warning system based on fruit fly perception neural network according to claim 1, Features: In step five, ij The smoothing filter kernel can be expressed as: The average grayscale value of frame t can be obtained by the following formula: Then, the average gray value obtained by the above formula is smoothed by the following formula:

5. The crop pest detection and early warning system based on fruit fly perception neural network according to claim 4, Features: Where n c is the time step of the image frame, k l =Sigmoid(n c xl) is the persistence coefficient of the time step, and finally the output of FFC is normalized: S t (FFC)=2(sigmoid(F t (FFC))-0.5) The spike signal sent by the feedforward cell at frame time t can be obtained by the following formula: Among them, T FFC is a dynamic threshold, T FFC =T FO +α FFC ×S t (FFC), T FO and α FFC is a given constant factor.

6. The crop pest detection and early warning system based on fruit fly perception neural network according to claim 1, Features: In step 6, T SSC The dynamic detection threshold is obtained by using the above detection threshold scheme, SSC t The output behavior of FFC neurons is obtained: Where n s The number of consecutive spike signals in a period of time on the table. t (m) is obtained from the output of the M neural node cells, and its output is normalized: Where 1 indicates that a warning detection signal is issued.

Images