Crop disease and pest identification method and system based on mobilenetv2 and storage medium

Abstract

Embodiments of the present application disclose a crop disease and pest identification method and system based on MobileNetV2 and a storage medium. The method belongs to the field of image recognition and comprises the following steps: receiving a crop video stream sent by a camera, extracting a frame of crop image from the crop video stream as a to-be-identified crop image; inputting the to-be-identified crop image into a current crop identification model for identification to obtain a crop disease and pest identification result, and different crop identification models can be switched for identification, wherein the crop identification model is a model obtained by deriving a preset algorithm module from a MobileNetV2 network and constructing the model through the preset algorithm module. The embodiments of the present application not only can improve the crop disease and pest identification rate and reduce the identification delay, but also are suitable for being deployed into embedded or FPGA devices to improve the system flexibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to a crop disease and pest identification method and system based on MobileNetV2 and a storage medium. BACKGROUND

[0002] With the gradual industrialization and technological development of agriculture, the prevention and control of agricultural diseases and pests has also gradually realized automation and technologicalization. Through image processing technology, automatic identification of crop diseases and pests can solve the shortcomings of text description and manual identification of crop diseases and pests, and provide real-time and accurate identification results for managers.

[0003] At present, there are mainly two ways to automatically identify diseases and pests through image processing technology: one is to deploy a deep learning model on the cloud, collect crop images through an edge device, send them to a crop identification model on the cloud server through the network for identification, and then return the identification result to the edge device for display. This method not only has high deployment cost, but also relies heavily on network transmission signals, and the identification result will be delayed. The other way is to directly load a deep learning network model on the CPU or GPU of the edge device for identification. However, due to the high power consumption of CPU or GPU, the recognition effect of the edge device is poor. SUMMARY

[0004] The embodiments of the present application provide a crop disease and pest identification method and system based on MobileNetV2 and a storage medium, aiming to solve the problem of high recognition delay and high power consumption of existing crop disease and pest identification methods.

[0005] In a first aspect, the embodiments of the present application provide a crop disease and pest identification method based on MobileNetV2, which includes:

[0006] Receiving a crop video stream sent by a camera, and extracting a frame of crop image from the crop video stream as a to-be-identified crop image;

[0007] Inputting the to-be-identified crop image into a current crop identification model for identification to obtain a crop disease and pest identification result, and different crop identification models can be switched for identification, wherein the crop identification model is a model constructed by a preset algorithm module derived from a MobileNetV2 network.

[0008] In a second aspect, the embodiments of the present application also provide a crop disease and pest identification system based on MobileNetV2, which comprises a display device, a camera, a terminal device and an electronic device, the camera is used to collect crop images, convert the crop images into crop video streams, and send the crop video streams to the electronic device; the display device is used to receive the crop images sent by the electronic device, and display the crop images; the terminal device is used to receive the crop disease and pest identification results sent by the electronic device, and display the crop disease and pest identification results; and the electronic device is used to execute the crop disease and pest identification method based on MobileNetV2.

[0009] In a third aspect, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program can realize the crop disease and pest identification method based on MobileNetV2 when executed by a processor.

[0010] The embodiments of the present application provide a crop disease and pest identification method, system and storage medium based on MobileNetV2. The method comprises the following steps: receiving a crop video stream sent by a camera, and extracting a frame of crop image from the crop video stream as a to-be-identified crop image; inputting the to-be-identified crop image into a current crop identification model for identification to obtain a crop disease and pest identification result, and switching different crop identification models for identification, wherein the crop identification model is a model constructed by a preset algorithm module derived from a MobileNetV2 network. The technical scheme of the embodiments of the present application directly identifies the crop image captured by the camera by using the crop identification model constructed by the preset algorithm module to obtain the crop disease and pest result, without relying on network transmission for identification, thereby reducing the identification delay and improving the identification efficiency; and the crop identification model is constructed by the preset algorithm module in the MobileNetV2 network, thereby reducing the power consumption of the device carrying the crop identification model. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical scheme of the embodiments of the present application, the drawings required in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 A hardware architecture diagram of a crop disease and pest identification system based on MobileNetV2 provided by the embodiments of the present application;

[0013] Figure 2 A flowchart of a crop disease and pest identification method based on MobileNetV2 provided by an embodiment of the present application is shown in FIG. 1.

[0014] Figure 3 A structural diagram of a MobileNetV2 network provided by an embodiment of the present application is shown in FIG. 2.

[0015] Figure 4 A sub-flowchart of a crop disease and pest identification method based on MobileNetV2 provided by an embodiment of the present application is shown in FIG. 3.

[0016] Figure 5 A sub-flowchart of a crop disease and pest identification method based on MobileNetV2 provided by an embodiment of the present application is shown in FIG. 4.

[0017] Figure 6 A schematic diagram of a crop identification model based on MobileNetV2 provided by an embodiment of the present application is shown in FIG. 5.

[0018] Figure 7 A flowchart of a crop disease and pest identification method based on MobileNetV2 provided by another embodiment of the present application is shown in FIG. 6.

[0019] Figure 8 A schematic block diagram of a crop disease and pest identification system based on MobileNetV2 provided by an embodiment of the present application is shown in FIG. 7.

[0020] Figure 9 A schematic block diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0022] It should be understood that, when used in the present specification and the appended claims, the terms “comprise” and “include” indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0023] It should also be understood that the terms used herein in the specification and the appended claims are for the purpose of describing particular embodiments only and are not intended to be limiting, as the scope of the present application will be limited only by the appended claims. As used herein in the specification and in the claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0024] It should also be further understood that the term "and / or" used in the specification and the appended claims herein is to be taken as a specific disjunctive in the sense of "one or the other" or "one or more than one" as well as in the sense of an "inclusive or" (i.e., as meaning "one or more than one" or "any combination of one or more of the associated listed items").

[0025] As used herein in the specification and in the claims, the term "if can be construed to mean "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be construed to mean "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0026] The crop disease and pest identification method based on MobileNetV2 of the embodiments of the present application can be applied to electronic devices, specifically, can be applied to electronic devices equipped with SOC (System-on-Chip) chips of the ZYNQ series, and optionally, can be applied to SOC chips of the xc7z035ffg676-2 type. The ZYNQ series chip is an ARM+FPGA architecture, where ARM (Advanced RISC Machine) and FPGA (Field Programmable Gate Arrays) can be further divided into PS (Processing System) and PL (Programmable Logic). More specifically, please refer to Figure 1 , Figure 1A hardware architecture diagram of a crop disease and pest identification system based on MobileNetV2 is provided in the embodiments of the present application. The hardware of the crop disease and pest identification system based on MobileNetV2 specifically comprises a display device 10, a camera 20, and an electronic device 40, wherein the electronic device 40 comprises a receiving processing module 401, a transmission module 402, a storage module 403, a processing module 404, a crop identification module 405, and a display module 406. In the embodiments of the present application, the PS end corresponds to the ARM part, that is, the processing module 404, which specifically comprises an ARM, an HP interface (high-performance purpose), and a GP interface (general purpose), and the ARM is specifically an ARM Cortex-A9 CPU processor; the PL end corresponds to the FPGA part, which specifically comprises the receiving processing module 401, the transmission module 402, the crop identification module 405, and the display module 406. Further, the transmission module 402 comprises a VDMA (Variable Destination Multiple Access) and an AXI bus (Advanced eXtensible Interface), wherein the VDMA comprises a VDMA0 and a VDMA1. The display module 406 is an HDMI display module, which is used to transmit high-definition pictures to the display device 10, for example, a "j" instruction can be input to call up the crop image captured by the camera 20 to display on the display screen of the display device 10. The storage module 403 comprises an SD card (Secure Digital Memory Card) and a DDR (Double Data Rate Synchronous Dynamic Random Access Memory), which is used to store crop images and crop disease and pest identification results.

[0027] In the embodiment of the present application, for the convenience of understanding, the working process of the crop disease and pest identification system based on MobileNetV2 will be introduced as follows: the camera 10 collects crop images, and converts the crop images into video stream type data to obtain crop video streams, and then sends the crop video streams to the receiving processing module 401, which writes the video streams into the DDR for storage through the VDMA0 and the AXI bus. The video streams stored in the DDR can be read by the VDMA1 through the AXI bus and output to the HDMI display module 406, which processes and encodes the video streams to obtain crop images and then outputs the crop images to the display device 10 for display through the display screen of the display device 10. At the same time, the ARM Cortex-A9 CPU processor extracts a frame of crop image from the video streams stored in the DDR as a to-be-identified crop image, and sends the to-be-identified crop image to the crop identification module 405 for crop disease and pest identification to obtain a crop disease and pest identification result. The crop disease and pest identification result can be output to the host computer through the UART interface or sent to the terminal device through the Bluetooth module for display.

[0028] In the embodiment of the present application, it should be noted that the ZYNQ series chip is an ARM+FPGA architecture, that is, a dual-core ARM Cortex-A9 processor and a field programmable gate array (FPGA) logic component are combined, so that the electronic device of the embodiment of the present application not only can utilize the powerful parallel processing capability of the FPGA to accelerate the convolution calculation in image recognition classification and improve the recognition efficiency, but also can conveniently perform embedded development on the ARM processor, add more peripherals to expand the system function, and increase the rich function design.

[0029] Please refer to Figure 2 , Figure 2 is a flowchart of a crop disease and pest identification method based on MobileNetV2 provided by the embodiment of the present application. As shown in Figure 2 , the method comprises the following steps S100-S110.

[0030] S100, receiving a crop video stream sent by a camera, and extracting a frame of crop image from the crop video stream as a to-be-identified crop image.

[0031] In the embodiment of the present application, a plurality of crop images are captured by the camera, for example, more than 100 crop images. Understandably, the camera can capture a plurality of types of crop images, for example, a plurality of types of crop images such as apples, cherries, corn, grapes, and potatoes. The camera converts the captured crop images into video stream type data, and then sends the video stream to a receiving processing module in the electronic device for receiving. The receiving processing module writes the video stream into the storage module of the electronic device through a transmission module for storage. It should be noted that in the embodiment of the present application, the camera uses an OV5640 camera. The reason for using the OV5640 camera is that the crop images captured by the OV5640 camera are clearer, support high resolution, and have high image acquisition rate and good processing performance. The processor of the electronic device can initialize and configure the OV5640 camera through an I2C bus, configure the working mode of the OV5640 camera as image output with resolution 224x224 and data format RGB888, so that the camera sends the crop images in a format that conforms to the electronic device for disease and pest identification.

[0032] In the embodiment of the present application, when it is necessary to identify crop images, the corresponding video stream of the crop type is found according to the identification instruction sent by the user. Since the storage module of the electronic device stores data of the crop video stream type, and the crop identification model identifies pictures, it is necessary to extract a frame of the crop image from the crop video stream as a to-be-identified crop image, and send the to-be-identified crop image to the crop identification model through the AXI bus for identification. It should be noted that in the embodiment of the present application, the extracted crop image is an image of a crop leaf.

[0033] S110, input the to-be-identified crop image into the current crop identification model for identification to obtain a crop disease and pest identification result, and different crop identification models can be switched for identification. The crop identification model is a model obtained by constructing a preset algorithm module derived from a MobileNetV2 network and the preset algorithm module.

[0034] In this embodiment of the invention, based on the MobileNetV2 network, crop recognition models for various crop types are trained using a CPU / GPU. The parameters of these trained models are obtained and stored in the storage module of the electronic device. For example, recognition models for various crops such as apples, cherries, corn, grapes, potatoes, raspberries, soybeans, and tomatoes are trained, and their parameters are stored on an SD card (Secure Digital Memory Card). If a model switching command is received from the user, the model parameters corresponding to the command are obtained as the target model parameters. The parameters of the current crop recognition model are then set as the target model parameters to obtain the target crop recognition model. This target model is then used as the current crop recognition model, and the image of the crop to be recognized is input into the current model for identification to obtain the crop disease / pest recognition result. If no model switching command is received, the image of the crop to be recognized is input into the current model for identification to obtain the crop disease / pest recognition result. It should be noted that in this embodiment of the invention, the user can switch between different crop recognition models by inputting commands according to actual needs. For example, if the user inputs the letter "a", the apple crop recognition model will be called; if the user inputs the letter "p", the potato crop recognition model will be called. It should also be noted that in this embodiment of the invention, the MobileNetV2 network is a lightweight convolutional neural network that uses depthwise separable convolution. Compared with conventional convolution, depthwise separable convolution can reduce parameters and improve running speed to a certain extent. Specifically, as... Figure 3 As shown, Figure 3 The diagram illustrates the structure of the MobileNetV2 network. The MobileNetV2 network first employs a 2D convolutional layer connected by seven bottleneck layers, then inputs it into a fully connected layer with dimensions of 1×1×1280. The output is a vector with dimensions 1 × the number of specific categories; the category corresponding to the largest element in the vector is the final recognition result. In this embodiment of the invention, traditional convolutional neural networks require large amounts of memory and computation, making them inefficient for use on electronic devices. However, the crop recognition model in this solution utilizes the MobileNetV2 network, which significantly reduces the number of model parameters and computational load while only slightly lowering accuracy. This not only reduces recognition latency but also improves recognition speed.

[0035] In some embodiments, such as this embodiment, as Figure 4As shown, the preset algorithm module is derived from the MobileNetV2 network, and the crop recognition model constructed through the preset algorithm module can include the following steps S111-S112.

[0036] S111, derive a general convolution algorithm module, a depth classifiable convolution algorithm module, a shortcut branch algorithm module and a full connection algorithm module through a high-level language synthesis tool;

[0037] S112, construct the crop recognition model through cyclic reuse of the general convolution algorithm module, the depth classifiable convolution algorithm module, the shortcut branch algorithm module and the full connection algorithm module.

[0038] In the embodiment of the application, since the MobileNetV2 network is developed based on the high-level language C / C++, and the FPGA chip of the electronic device is developed based on the RTL (Register Transfer Level) language, the MobileNetV2 network can be deployed in the FPGA chip by synthesizing the MobileNetV2 network into the RTL code through the high-level language synthesis tool (such as Vivado HLS). Specifically, first, the MobileNetV2 network is built using a high-level language (such as C / C++), and array expansion, loop pipelining and other synthesis optimization strategies are reasonably used to improve the algorithm performance of the MobileNetV2 network, then the HLS tool is used to fix the process, including allocation, scheduling and binding, to simulate and verify the algorithm of the MobileNetV2 network, and finally, the preset algorithm module is derived from the MobileNetV2 network. The preset algorithm module includes four algorithm modules, namely Conv, Pwdwconv, Shortcut and Fc. Further, the Conv is a general convolution module that can extract convolution information; the PwdwConv is a depth classifiable convolution module that integrates PwConv and DwConv to reduce the number of parameters and operation cost; the Shortcut is a shortcut branch module that can alleviate the gradient divergence problem in the deep network; and the Fc is a full connection module that acts as a "classifier" in the entire network. The four algorithm modules derived are cyclically reused to construct a crop recognition model based on the MobileNetV2 network. In the embodiment of the application, compared with the traditional method of directly deploying the entire MobileNetV2 network in the electronic device in sequence, since the entire MobileNetV2 network calls PwConv and DwConv convolution multiple times, the network is too large, resulting in a large amount of hardware consumption, which is difficult for the electronic device to bear, and the present scheme achieves the effect of reducing the hardware consumption of the electronic device by cyclically and repeatedly using the four algorithm modules.

[0039] In some embodiments, for example in the present embodiment, as shown in Figure 5 the step 112 includes steps S1121-S1124.

[0040] S1121, based on the ordinary convolution algorithm module, the point-by-point convolution and the deep convolution, a top layer network is constructed;

[0041] S1122, based on the ordinary convolution algorithm module, the deep classifiable convolution algorithm module and the shortcut branch algorithm module, a bottleneck layer network is constructed;

[0042] S1123, based on the deep convolution and the full connection algorithm module, a bottom layer network is constructed;

[0043] S1124, the crop recognition model is constructed through the top layer network, the bottleneck layer network and the bottom layer network.

[0044] In the present embodiment, please refer to Figure 6 , Figure 6 is a schematic diagram of the architecture of the crop recognition model based on MobileNetV2 constructed by the top layer network, the bottleneck layer network and the bottom layer network. Specifically, the crop recognition model is divided into three parts: Head (top layer network), Bottleneck (bottleneck layer network) and Tail (bottom layer network). It should be noted that in the present embodiment, the deep classifiable convolution algorithm module integrates the point-by-point convolution and the deep convolution. The Head part is constructed by the ordinary convolution algorithm module, the point-by-point convolution and the deep convolution, and is the starting input part of the entire crop recognition model. The crop image to be identified is input into the Head part for convolution operation to obtain a first convolution result. It should also be noted that in the present embodiment, the input is a crop image to be identified of 224x224x3, and the output is an intermediate convolution result of 112x112x6, i.e. the first convolution result.

[0045] The Bottleneck part is constructed by the common convolution algorithm module, the deep classifiable convolution algorithm module and the shortcut branch algorithm module, specifically, the Bottleneck part is composed of 16 Bottleneck residual blocks, each Bottleneck residual block is composed of multiple layers of DwConv, PwConv, Shortcut and Relu activation function, and the Shortcut only plays a role when the stride is 1. The first convolution result is input into the Bottleneck for convolution operation to obtain a second convolution result. It should be noted that in the embodiment of the application, the 112*112*6 first convolution result is input into the Bottleneck for convolution operation to obtain a 7*7*1280 second convolution result.

[0046] The Tail part is constructed by the deep convolution and the full connection algorithm module, and is the final output part of the entire crop identification model. Specifically, the 7*7*1280 second convolution result is input into the Tail part for operation to obtain a crop disease and pest identification result.

[0047] Please refer to Figure 7 , Figure 7 A flowchart of a crop disease and pest identification method based on MobileNetV2 provided by another embodiment of the application is shown in FIG. 6. The method further includes steps S120-S130 after step S110.

[0048] S120, if a photograph saving instruction is received, an interrupt instruction is triggered to stop the current crop identification model from identifying the to-be-identified crop image;

[0049] S130, the to-be-identified crop image, the disease and pest category and the disease probability of the multiple disease and pest categories are saved.

[0050] In the embodiment of the present application, when the crop recognition model recognizes the to-be-recognized crop image, the user can control the current crop recognition model to save the to-be-recognized crop image currently recognized and the recognition result by inputting an instruction. Specifically, it is judged whether a photograph saving instruction input by the user is received, for example, the user presses a photograph button, and then the photograph saving instruction is sent. If the photograph saving instruction is received, an interrupt instruction is triggered to control the crop recognition model to trigger an interrupt, stop recognizing the to-be-recognized crop image currently recognized, and save the to-be-recognized crop image currently recognized and the corresponding pest and disease recognition result. The pest and disease recognition result includes a pest and disease category and a disease probability of a plurality of pest and disease categories, and more specifically, the to-be-recognized crop image currently recognized, the recognized pest and disease category, and the disease probability of the plurality of pest and disease categories are cached in the DDR of the electronic device and saved in the SD card, so that the pest and disease condition of the to-be-recognized crop image can be further studied in detail subsequently.

[0051] In the embodiment of the present application, after the to-be-recognized crop image, the pest and disease category, and the disease probability of the plurality of pest and disease categories are saved, the pest and disease category and the disease probability of the plurality of pest and disease categories can be transmitted to a terminal device for display. Specifically, a Bluetooth module can be externally connected through a UART (Universal Asynchronous Receiver / Transmitter) serial port, wireless transmission function can be realized between the Bluetooth module and the terminal device (such as a mobile phone, a tablet computer, or a smart watch), or wireless communication function can be realized by establishing a connection between the wireless communication module and the terminal device. When the transmission instruction sent by the terminal device is received, the pest and disease category and the disease probability of the plurality of pest and disease categories are transmitted to the display screen of the terminal device for display, so that the user can quickly and conveniently know the pest and disease condition of the crop, the portability and mobility are greatly increased through the terminal device, and the pest and disease condition of the crop is identified in real time and quickly. After the current crop image recognition is completed, the step of extracting a frame of the crop image from the crop video stream as a to-be-recognized crop image is returned to be executed, a new frame of the crop image is extracted to continue the subsequent pest and disease recognition task of the crop, until the user inputs a stop instruction to stop the recognition, for example, the user can input "0" as a stop recognition instruction to stop the pest and disease recognition, and energy consumption is saved.

[0052] In the embodiment of the present application, the crop pest and disease recognition system based on MobileNetV2 has greater performance improvement compared with the traditional crop recognition model. Specifically, in order to better introduce the system, the performance of the system will be introduced from the following five aspects: running speed, model accuracy, time sequence, area, and power consumption.

[0053] In terms of running speed, the running speed of the system is verified from the two ways of taking SD card to read image for recognition and camera to collect image for recognition. As shown in Table 1, Table 1 is an average time comparison table of different processing methods for recognizing one frame of apple leaf image. First, when taking the method of reading crop pictures from SD card for recognition, further, the image of crop leaves is read for recognition, and the specific process of verifying the running speed is: 100 apple leaf pictures are read from the SD card as a test set, input into the crop recognition model of the system for recognition, and the average time for recognizing one apple crop leaf picture is about 84ms, while the average time for recognizing one apple crop leaf picture by the traditional crop model based on CPU chip is 410ms. It can be seen that compared with the traditional CPU, the system has nearly 5 times improvement in running speed. The crop recognition model obtained by optimizing the MobileNetV2 network is very suitable for deployment on embedded edge devices.

[0054] Secondly, the process of verifying the running speed of the system by collecting the image of crop leaves by camera for recognition is: the apple leaf image is collected directly by the camera and input into the crop recognition model of the system for recognition, wherein the average time for recognizing one frame of apple leaf image by the crop recognition model is 89ms; the average time for the system to completely process one frame of apple leaf image (from inputting the image from the camera to outputting the recognition result) is 116ms, compared with the recognition method of reading the image from the SD card, the camera image recognition method needs to perform hardware transmission, memory reading and writing and other steps, which will introduce additional processing time. But compared with the traditional CPU-based recognition method, the system also shows obvious acceleration effect.

[0055] Table 1

[0056]

[0057]

[0058] In terms of model accuracy, the accuracy of various crop recognition models is tested respectively: the trained crop recognition model of various crops (such as including apple, cherry, corn, grape, potato, raspberry, soybean, tomato, etc.) is deployed to the system, and 100 corresponding crop leaf images are used to identify the disease and pest of each model, and the average accuracy is more than 85%. It should be noted that in this embodiment, in order to quantitatively show the recognition accuracy of the system based on the MobileNetV2 network, the system reads the crop leaf images from the SD card for recognition. As shown in Table 2, Table 2 shows the recognition results of four kinds of crop images of diseases and pests, specifically, including Apple (apple, 4 kinds of disease and pest species: Apple_scab, Black_rot, Cedar_apple_rust, Healthy), Potato (potato, 3 kinds of disease and pest species: Early_blight, Late_blight, Healthy), Peach (peach, 2 kinds of disease and pest species: Bacterial_spot, Healthy), Cherry (cherry, 2 kinds of disease and pest species: Powdery_mildew, Healthy) recognition results, the recognition accuracy is 87%, 90%, 94%, and 88% respectively. It can be seen that the network model of the system can still maintain good accuracy while being as light as possible and improving speed. And in the scene of crop disease and pest recognition, there are too many leaf samples, and the accuracy of single recognition is not too demanding, more importantly, the recognition speed and low power consumption of the algorithm model.

[0059] Table 2

[0060]

[0061] In terms of timing and area, the clock frequency of each part of the system is as follows: the running frequency of the PS end ARM processor is 767MHz; the running frequency of the PL end FPGA is 130MHz. And the system has no timing violation, which can ensure the stable operation of the system for a long time and in multiple environments. Through testing, the highest clock frequency that the system can run is above 150MHz, but considering the best match of power consumption and performance, the system is selected to run at 130MHz to realize the high-performance crop disease and pest identification function with relatively low power consumption. Secondly, the area is the hardware resource consumption of the system. According to experimental verification, in terms of hardware resource consumption, the overall resource consumption of the system and the resource consumption of the crop recognition model are lower than those of the traditional crop recognition model, and the overall hardware overhead is not high, which is suitable for application to various embedded mobile or wearable edge devices.

[0062] In terms of power consumption, the detailed resource consumption of the processor and the four algorithm modules constituting the crop recognition model is shown in Table 3. In Table 3, Module represents a module, Power represents power, Proportion represents proportion, Processing System represents a processor, Conv, PwdwConv, Shortcut, and Fc represent ordinary convolution algorithm module, depth classifiable convolution algorithm module, shortcut branch algorithm module, and full connection algorithm module, respectively. As can be seen from Table 3, the power consumption of the four algorithm modules of the system is reasonable, which also ensures the possibility of deploying the system in low-power edge devices. Alternatively, if the application scenario has extremely stringent power requirements, the running frequency of the ARM processor and the four algorithm modules can be appropriately reduced, sacrificing a small amount of speed for lower power consumption. For example, the running frequency of the PL end can be reduced to 100MHz, and the system dynamic power consumption can be reduced to 2.805W, but the performance of 125ms average per frame image inference time can still be achieved.

[0063] Table 3

[0064]

[0065] Figure 8 is a schematic block diagram of a crop disease and pest identification system 200 based on MobileNetV2 provided by an embodiment of the present application. As shown in Figure 8 , the crop disease and pest identification system 200 based on MobileNetV2 includes a display device 10, a camera 20, a terminal device 30, and an electronic device 40 for executing the above-mentioned crop disease and pest identification method based on MobileNetV2.

[0066] , the camera 20 is configured to collect crop images, convert the crop images into crop video streams, and send the crop video streams to the electronic device 40; the display device 10 is configured to receive crop images sent by the electronic device 40 and display the crop images; the terminal device 30 is configured to receive crop disease and pest identification results sent by the electronic device 40 and display the crop disease and pest identification results; and the electronic device 40 is configured to execute the above-mentioned crop disease and pest identification method based on MobileNetV2.

[0067] In some embodiments, for example in the present embodiment, the electronic device 40 includes a receiving processing module 401, a transmission module 402, a storage module 403, a processing module 404, and a crop recognition module 405.

[0068] The receiving processing module 401 is configured to receive the crop video stream sent by the camera 20 and save the crop video stream to the storage module 403. The processing module 404 is configured to acquire the crop video stream from the storage module 403, extract a frame of crop image from the crop video stream as a to-be-identified crop image, and transmit the to-be-identified crop image to the crop identification module 405 for identification via the transmission module 402. The crop identification module 405 is configured to identify the to-be-identified crop image to obtain a crop disease and pest identification result. Further, the electronic device 40 further includes a display module 406 configured to acquire the crop video stream from the storage module 403 via the transmission module 402, perform stream processing coding on the crop video stream to obtain a crop image, and display the crop image via the display device 10.

[0069] In some embodiments, for example in the present embodiment, the processing module 404 includes a derivation unit and a construction unit.

[0070] The derivation unit is configured to derive a general convolution algorithm module, a depth classifiable convolution algorithm module, a shortcut branch algorithm module, and a full connection algorithm module via a high-level language synthesis tool. The construction unit is configured to construct the crop identification model by circularly multiplexing the general convolution algorithm module, the depth classifiable convolution algorithm module, the shortcut branch algorithm module, and the full connection algorithm module.

[0071] In some embodiments, for example in the present embodiment, the construction unit includes a first construction subunit, a second construction subunit, a third construction subunit, and a fourth construction subunit.

[0072] The first construction subunit is configured to construct a top layer network based on the general convolution algorithm module, the point-by-point convolution, and the depth convolution. The second construction subunit is configured to construct a bottleneck layer network based on the general convolution algorithm module, the depth classifiable convolution algorithm module, and the shortcut branch algorithm module. The third construction subunit is configured to construct a bottom layer network based on the depth convolution and the full connection algorithm module. The fourth construction subunit is configured to construct the crop identification model via the top layer network, the bottleneck layer network, and the bottom layer network.

[0073] In some embodiments, for example in the present embodiment, the crop identification module 405 includes a first operation unit, a second operation unit, and a third operation unit.

[0074] The first operation unit is configured to input the to-be-identified crop image into the top-layer network for convolution operation to obtain a first convolution result; the second operation unit is configured to input the first convolution result into the bottleneck-layer network for convolution operation to obtain a second convolution result; and the third operation unit is configured to input the second convolution result into the bottom-layer network to obtain the crop disease and pest identification result.

[0075] In some embodiments, for example in the present embodiment, the crop identification module 405 further includes a switching unit and an identification unit.

[0076] The switching unit is configured to, if the model switching instruction is received, obtain model parameters corresponding to the model switching instruction as target model parameters, and set the parameters of the current crop identification model to the target model parameters to obtain a target crop identification model; and the identification unit is configured to use the target crop identification model as the current crop identification model, and input the to-be-identified crop image into the current crop identification model for identification to obtain the crop disease and pest identification result.

[0077] In some embodiments, for example in the present embodiment, the processing module 404 further includes an interrupt unit and a saving unit.

[0078] The interrupt unit is configured to, if a photograph saving instruction is received, trigger an interrupt instruction to stop the current crop identification model from identifying the to-be-identified crop image; and the saving unit is configured to save the to-be-identified crop image, the disease and pest category, and the disease probability of the plurality of disease and pest categories.

[0079] In some embodiments, for example in the present embodiment, the processing module 404 further includes a transmission unit configured to transmit the disease and pest category and the disease probability of the plurality of disease and pest categories to the terminal device 30 for display, and return to the step of extracting a frame of the crop image from the crop video stream as a to-be-identified crop image.

[0080] The crop disease and pest identification system based on MobileNetV2 described above can be implemented in the form of a computer program, which can run on an electronic device as shown in Figure 9 .

[0081] Please refer to Figure 9 , Figure 9 is a schematic block diagram of an electronic device provided by an embodiment of the present application. Please refer to Figure 9 , the electronic device 900 includes a processor 902, a memory, and an interface 907 connected through a system bus 901, wherein the memory can include a storage medium 903 and an internal memory 904.

[0082] The storage medium 903 can store an operating system 9031 and a computer program 9032. The computer program 9032, when executed, can cause the processor 902 to perform a crop disease and pest identification method based on MobileNetV2.

[0083] The processor 902 is configured to provide computing and control capabilities to support the operation of the entire electronic device 900.

[0084] The memory 904 provides an environment for the execution of the computer program 9032 in the storage medium 903. The computer program 9032, when executed by the processor 902, can cause the processor 902 to perform a crop disease and pest identification method based on MobileNetV2.

[0085] The interface 905 is configured to communicate with other devices. Those skilled in the art can understand that the interface 905 can be configured to transmit and receive data according to the communication protocols used by the devices, and the specific forms of the interface 905 are not limited here. Figure 9 It should be understood that the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device 900 to which the scheme of the present application is applied. The specific electronic device 900 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0086] It should be understood that in the embodiments of the present application, the processor 902 can be a central processing unit (CPU), and the processor 902 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0087] Those of ordinary skill in the art can understand that all or part of the processes in the method of the above embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a storage medium, which is a computer readable storage medium. The computer program is executed by at least one processor in the wireless communication system to implement the process steps of the above-mentioned method embodiments.

[0088] Therefore, the present application also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program. The computer program is executed by a processor to enable the processor to perform any embodiment of the crop disease and pest identification method based on MobileNetV2.

[0089] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer-readable storage media that can store program codes.

[0090] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, wireless communication software, or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0091] In several embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are only illustrative. For example, the division of each unit is only a logical functional division, and actual implementation can have another division manner. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0092] The steps in the method embodiments of the present application can be adjusted, combined and reduced in sequence according to actual needs. The units in the system embodiments of the present application can be combined, divided and reduced according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0093] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the parts that make contributions to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The wireless communication software product is stored in a storage medium and includes a plurality of instructions for enabling a computer device (which can be a personal wireless communication terminal, a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0094] In the above embodiments, the description of each embodiment is focused on, and the part not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0095] Obviously, various modifications and changes can be made to the present application without departing from the spirit and scope of the present application. Thus, such modifications and changes are intended to be included within the scope of the present application as defined in the claims and their equivalents.

[0096] The above description is merely illustrative of the application and not restrictive thereof. The scope of the application is not limited to the above embodiments, but any modifications or replacements within the technical scope of the present application should be covered by the present application. Therefore, the scope of the present application should be defined by the scope of the claims.

Claims

1. A crop disease and pest identification method based on MobileNetV2, characterized in that, The method comprises the following steps: receiving a crop video stream sent by a camera and extracting a frame of crop image from the crop video stream as a to-be-identified crop image; inputting the to-be-identified crop image into a current crop identification model for identification to obtain a crop disease and pest identification result, and the different crop identification models can be switched for identification, wherein the crop identification model is a model obtained by deriving a preset algorithm module from a MobileNetV2 network and constructing the preset algorithm module; wherein the crop identification model derived from the MobileNetV2 network and constructed by the preset algorithm module comprises: deriving an ordinary convolution algorithm module, a depth separable convolution algorithm module, a shortcut branch algorithm module and a fully connected algorithm module through a high-level language synthesis tool; constructing the crop identification model by circularly reusing the ordinary convolution algorithm module, the depth separable convolution algorithm module, the shortcut branch algorithm module and the fully connected algorithm module; the depth separable convolution algorithm module integrates point-wise convolution and depth convolution, and the crop identification model constructed by circularly reusing the ordinary convolution algorithm module, the depth separable convolution algorithm module, the shortcut branch algorithm module and the fully connected algorithm module comprises: constructing a top layer network based on the ordinary convolution algorithm module, the point-wise convolution and the depth convolution; constructing a bottleneck layer network based on the ordinary convolution algorithm module, the depth separable convolution algorithm module and the shortcut branch algorithm module; constructing a bottom layer network based on the depth convolution and the fully connected algorithm module; constructing the crop identification model through the top layer network, the bottleneck layer network and the bottom layer network. 2.The MobileNetV2-based crop disease and pest recognition method according to claim 1, characterized in that, the method further comprises: if a model switching instruction is received, obtaining model parameters corresponding to the model switching instruction as target model parameters, and setting the parameters of the current crop identification model as the target model parameters to obtain a target crop identification model; inputting the target crop identification model as the current crop identification model and inputting the to-be-identified crop image into the current crop identification model for identification to obtain the crop disease and pest identification result. The crop disease and pest identification result comprises a disease and pest category and a disease probability of multiple disease and pest categories, and the method further comprises: 3.The MobileNetV2-based crop disease and pest recognition method according to claim 1, characterized in that, if a photograph saving instruction is received, triggering an interrupt instruction to stop the current crop identification model from identifying the to-be-identified crop image; ​ ​ 4. The crop disease and pest recognition method based on MobileNetV2 according to claim 3, characterized in that, ​ ​ Save the to-be-identified crop image, the disease and pest category, and the disease probability of the plurality of disease and pest categories. 5.The MobileNetV2-based crop disease and pest recognition method according to claim 4, characterized in that, The method further comprises: Transmit the disease and pest category and the disease probability of the plurality of disease and pest categories to the terminal device for display, and return to execute the step of extracting a frame of the crop image from the crop video stream as the to-be-identified crop image.

6. A crop disease and pest identification system based on MobileNetV2, comprising a display device, a camera, a terminal device and an electronic device, characterized in that: The camera is used to collect crop images and convert the crop images into a crop video stream, and then send the crop video stream to the electronic device; the display device is used to receive the crop images sent by the electronic device and display the crop images; the terminal device is used to receive the crop disease and pest identification result sent by the electronic device and display the crop disease and pest identification result; and the electronic device is used to execute the method for crop disease and pest identification based on MobileNetV2 according to any one of claims 1 to 5. The electronic device comprises a receiving and processing module, a transmission module, a storage module, a processing module, and a crop identification module, wherein, The receiving and processing module is used to receive the crop video stream sent by the camera and save the crop video stream to the storage module; The processing module is used to obtain the crop video stream from the storage module, extract a frame of crop image from the crop video stream as a to-be-identified crop image, and transmit the to-be-identified crop image to the crop identification module for identification through the transmission module; The crop identification module is used to identify the to-be-identified crop image to obtain a crop disease and pest identification result.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program can implement the method according to any one of claims 1 to 5 when executed by the processor.

Images