Pest situation monitoring method and apparatus based on damaged image, and device and readable medium

Abstract

Provided in the present application are a pest situation monitoring method and apparatus based on a damaged image, and a device and a readable medium. The method comprises: performing damage processing of at least one damage type on each sample insect image, and marking an insect class and a damage type corresponding to each sample insect image, so as to obtain a training insect image set; inputting the training insect image set into an adversarial network model for training, so as to obtain a classification model; and inputting into the classification model an insect image to be processed, and determining an insect class and a damage type corresponding to the insect image to be processed. In the technical solution, an insect class classification model is trained on the basis of a damaged image, so that the generalization performance and identification performance of the classification model with respect to a damaged or incomplete insect image are improved, thereby providing an accurate and reliable means for real-time monitoring and classification of an insect class.

Description

Insect monitoring method, device, equipment and readable medium based on damaged images

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 28, 2023, with application number 202311605875.X. The entire contents of this application are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and readable medium for insect monitoring based on damaged images. Background Art

[0003] Insect monitoring refers to the use of insect visual data collection equipment to automatically complete system operations such as insect trapping, insect killing, insect dispersion, photography, transportation, collection, and drainage without human supervision. The pest categories and counts are uploaded to the agricultural insect monitoring platform in real time. Combined with practical experience and historical data, predictions are made on future trends of pests and diseases, improving labor efficiency and the accuracy of monitoring results, and providing accurate and timely forecasting services to scientific researchers and growers.

[0004] However, due to the difficulty in controlling the wild environment, such as weather conditions and damage to wild organisms, insect visual data collection equipment is often easily damaged, causing the collected insect image data to be contaminated, and ultimately making the model trained based on the damaged data ineffective.

[0005] Therefore, how to provide a technical solution that can accurately identify the insect category in the damaged image is a technical problem that needs to be solved urgently by those skilled in the art.

[0006] Summary of the Invention

[0007] One objective of this application is to provide a method, apparatus, device, and readable medium for insect monitoring based on damaged images, at least to address the problem of classifying damaged or incomplete insect image data. This application provides a novel method for insect monitoring based on damaged images. This method utilizes a domain generalization algorithm to effectively address the performance degradation of insect classification models caused by degraded training image quality due to damage to insect visual data collection equipment. This method improves the insect classification model's performance for damaged or incomplete insect images, providing a precise and reliable means for real-time monitoring and classification of insects.

[0008] To achieve the above objectives, some embodiments of the present application provide the following aspects:

[0009] In a first aspect, some embodiments of the present application further provide a method for monitoring insect infestation based on damaged images, the method comprising:

[0010] Performing damage processing of at least one damage type on each sample insect image, and labeling the insect category and damage type corresponding to each sample insect image to obtain a training insect image set;

[0011] Inputting the training insect image set into the adversarial network model for training to obtain a classification model;

[0012] The insect image to be processed is input into the classification model to determine the insect category and damage type corresponding to the insect image to be processed.

[0013] In a second aspect, some embodiments of the present application further provide an insect monitoring device based on a damaged image, the device comprising:

[0014] a damaged image acquisition module configured to perform damage processing on each sample insect image according to at least one damage type, and label the insect category and damage type corresponding to each sample insect image to obtain a training insect image set;

[0015] a damaged image training module configured to input the training insect image set into an adversarial network model for training to obtain a classification model;

[0016] The damage image processing module is configured to input the insect image to be processed into the classification model to determine the insect category and damage type corresponding to the insect image to be processed.

[0017] In a third aspect, some embodiments of the present application further provide a computer device, comprising:

[0018] one or more processors; and

[0019] A memory storing computer program instructions, which, when executed, cause the processor to perform the insect monitoring method based on damaged images as described above.

[0020] In a fourth aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the insect monitoring method based on damaged images as described above.

[0021] Compared to related technologies, the solution provided in the embodiments of this application performs damage processing on each sample insect image for at least one damage type, labels each sample insect image with the corresponding insect category and damage type, and generates a training insect image set. This training insect image set is then fed into an adversarial network model for training to generate a classification model. The processed insect image is then fed into the classification model to determine the insect category and damage type corresponding to the processed insect image. This technical solution trains the insect classification model based on damaged images to improve the classification model's generalization and recognition performance for damaged or incomplete insect images, providing a precise and reliable means for real-time monitoring and classification of insect categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a flow chart of an insect monitoring method based on damaged images provided in Example 1 of the present application;

[0023] FIG2 is a schematic diagram of the structure of an adversarial network model provided in Example 2 of the present application;

[0024] FIG3 is a schematic structural diagram of an insect monitoring device based on damaged images provided in Example 3 of the present application;

[0025] FIG4 is a schematic diagram of the structure of a computer device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] The technical solutions provided by the embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0028] Example 1

[0029] FIG1 is a flow chart of the insect monitoring method based on damaged images provided in Example 1 of the present application. As shown in FIG1 , the process specifically includes the following steps:

[0030] Step S101 , performing damage processing of at least one damage type on each sample insect image, and labeling the insect category and damage type corresponding to each sample insect image to obtain a training insect image set;

[0031] The sample insect image may be an image obtained by capturing trapped insects using an insect presence visual collection device. Specifically, the insects may be first attracted and collected using an intelligent insect presence detection light, and then captured using a high-resolution CCD camera to obtain the sample insect image.

[0032] It should be noted that the sample insect images are insect images collected under undamaged conditions.

[0033] Since the insect visual collection equipment is in the wild environment, it is easily affected by weather conditions or damage by wild creatures. For example, the lens may be blocked by rain or snow in rainy or snowy weather, or the camera may not focus accurately during the focusing process, or the insect visual collection equipment may be damaged by wild creatures during image collection, causing the equipment to shake and resulting in blurred images, or strong light environment may cause the collected images to be overexposed, etc.

[0034] In view of the above problems, this application pre-damages insect images to train a classification model under image damage conditions.

[0035] Among them, the damage type can be determined based on any damage behavior that can be encountered in the wild, such as light intensity, depth of field, occlusion, jitter, etc.

[0036] In the present application, damage processing of various damage types may be performed on the same sample insect image, and damage processing of different damage degrees of the same damage type may also be performed on the same sample insect image.

[0037] For example, the resolution of a sample insect image is 100 PPI. Without changing the size of the sample insect image, the resolution is reduced to 80 PPI and 50 PPI respectively.

[0038] As another example, a region of a preset size in a sample insect image is randomly blocked to generate a plurality of blocked training insect images.

[0039] Optionally, the damage type includes at least one of image illumination change, partial image loss, or image color reset.

[0040] Among them, image illumination change can be to adjust the contrast, brightness or saturation of the sample insect image; local image loss can be to block part of the area of ​​the sample insect image; image color reset can be to adjust the pixel value of each pixel in the sample insect image according to certain rules.

[0041] In this application, the training insect image set can be expressed as: C = {(x1, y1, d1), (x1, y1, d1), ..., (x n ,y n ,dn )};

[0042] Where C represents the training insect image set, x represents the training insect image, y represents the insect category, d represents the damage type, and n represents the number of training insect images.

[0043] Step S102: inputting the training insect image set into an adversarial network model for training to obtain a classification model;

[0044] The adversarial network model may be a model that performs adversarial training on the insect categories and damage types of training insect images, so that the trained model can perform category recognition on insect images with different damage types.

[0045] Specifically, the training insect image set is input into the adversarial network model, and the insect category and damage type are predicted respectively. The predicted results are compared with the actual labels to encourage the adversarial network model to learn the inseparable features of the damage type in the training insect image set, so as to achieve domain generalization of the model and obtain a classification model.

[0046] Step S103: input the insect image to be processed into the classification model to determine the insect category and damage type corresponding to the insect image to be processed.

[0047] The insect image to be processed may be a newly collected insect image by an insect visual data collection device. Specifically, the insect image to be processed may be input into a trained classification model to output the insect category and damage type corresponding to the insect image to be processed.

[0048] For example, an insect image is input into a trained classification model, and the output is: the insect category is aphid, and the damage type is partial image loss.

[0049] The present invention provides an insect monitoring method based on damaged images. This method performs at least one damage treatment on each sample insect image, labels each sample insect image with the corresponding insect category and damage type, and generates a training insect image set. The training insect image set is then trained in an adversarial network model to generate a classification model. The processed insect image is then input into the classification model to determine the insect category and damage type corresponding to the processed insect image. This technical solution trains the insect classification model based on damaged images to improve the classification model's generalization and recognition performance for damaged or incomplete insect images, providing a precise and reliable means for real-time monitoring and classification of insect categories.

[0050] The benefit of this setup is that it can continuously learn about insect species online, enabling automatic discovery of new species while retaining accurate identification of old species.

[0051] Example 2

[0052] FIG2 is a schematic diagram of the structure of an adversarial network model provided in Example 2 of the present application. As shown in FIG2, the adversarial network model includes a feature extractor, a category classifier, a domain classifier, and; wherein,

[0053] The feature extractor is configured to perform feature extraction on the training insect image set to obtain a feature image set;

[0054] The category classifier is configured to identify the insect category of each training insect image in the feature image set, and determine the insect category corresponding to each training insect image;

[0055] The domain classifier is configured to identify the damage type of each training insect image in the feature image set, and determine the damage type corresponding to each training insect image;

[0056] The gradient reversal layer is configured to perform a negative operation on the gradient during the back propagation process of the gradient.

[0057] The feature extractor can be a convolutional neural network. Exemplarily, the convolutional neural network may include convolutional layers, pooling layers, and fully connected layers. A training insect image is input into the convolutional neural network. The convolutional layers extract local features from the insect image while preserving the spatial location information of the features, resulting in a convolutional feature map. The pooling layers reduce the spatial size of the convolutional feature map, reducing the number of model parameters and avoiding overfitting. The fully connected layers expand the features obtained by the convolution and pooling operations into a one-dimensional vector, resulting in a feature image.

[0058] The category classifier can be configured to classify and identify the insect categories of the training insect images. For example, a classification algorithm such as logistic regression, naive Bayes, decision tree, support vector machine, random forest, or gradient boosting tree can be used for classification and identification.

[0059] Among them, the domain classifier can be set to identify which damage algorithm the training bug image comes from.

[0060] Since the feature extractor aims to prevent the domain classifier from correctly determining which corruption algorithm the information comes from, the feature extractor and the domain classifier form an adversarial relationship. This adversarial relationship is achieved through adversarial training based on the gradient reversal layer.

[0061] Accordingly, the domain classifier is specifically configured to identify the damage type of each training insect image in the feature image extracted from the image by the feature extractor, and determine the damage type corresponding to each training insect image.

[0062] Backpropagation, which updates parameters, involves passing the loss value backward layer by layer. Each layer calculates the gradient based on the returned loss value and updates its parameters accordingly. The feature extractor's task is to maximize bug classification accuracy while minimizing domain classification accuracy, thus maximizing the domain classifier's objective function.

[0063] Therefore, as shown in Figure 2, a gradient reversal layer (GR) is placed between the domain classifier and the feature extractor. This makes the training objectives of the network before and after the gradient reversal layer opposite to each other, thus achieving an adversarial effect, improving the classification model's recognition accuracy for different damage types, and increasing the model's reliability and robustness in practical applications.

[0064] In some embodiments of the present application, the process of constructing the loss function of the adversarial network model includes: determining the loss function of the adversarial network model based on category classification loss, domain classification loss and gradient reversal loss.

[0065] Among them, the loss function is set as a learning criterion to optimize the adversarial network model during the training process of the adversarial network model to improve the generalization ability and robustness of the adversarial network model.

[0066] Specifically, the category classification loss, domain classification loss, and gradient reversal loss can be weighted to obtain the loss function of the adversarial network model.

[0067] Optionally, determining a loss function of the adversarial network model based on the category classification loss, the domain classification loss, and the gradient reversal loss includes:

[0068] The loss function of the adversarial network model is determined using the following formula: L = L class +αL domain ;

[0069] Where L represents the total loss of the adversarial network model, L class represents the category classification loss, L domain represents the domain classification loss, GR represents the gradient reversal loss, and α is a hyperparameter.

[0070] An embodiment of the present application provides an adversarial network model comprising a feature extractor, a category classifier, and a domain classifier. The feature extractor is configured to extract features from a training insect image set to obtain a feature image set; the category classifier is configured to identify the insect category of each training insect image in the feature image set and determine the insect category corresponding to each training insect image; and the domain classifier is configured to identify the damage type of each training insect image in the feature image set and determine the damage type corresponding to each training insect image. This technical solution, by performing adversarial learning on the feature extractor and the domain classifier, enables the classification model to accurately identify insect categories across multiple damage types, thereby increasing the reliability and robustness of the classification model in practical applications and providing a precise and reliable means for real-time monitoring and classification of insect species.

[0071] Example 3

[0072] FIG3 is a schematic diagram of the structure of the insect monitoring device based on damaged images provided in Example 3 of the present application. As shown in FIG3 , it specifically includes the following:

[0073] The damaged image acquisition module 310 is configured to perform damage processing of at least one damage type on each sample insect image, and label the insect category and damage type corresponding to each sample insect image to obtain a training insect image set;

[0074] a damaged image training module 320 configured to input the training insect image set into an adversarial network model for training to obtain a classification model;

[0075] The damaged image processing module 330 is configured to input the insect image to be processed into the classification model to determine the insect category and damage type corresponding to the insect image to be processed.

[0076] The insect monitoring device based on damaged images in the embodiments of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc., and the embodiments of the present application do not specifically limit this.

[0077] The insect monitoring device based on damaged images in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0078] The insect monitoring device based on damaged images provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, they will not be described here.

[0079] Example 4

[0080] In addition, an embodiment of the present application further provides a computer device. FIG4 is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present application. The structure of the device is shown in FIG4 . The device includes a memory 41 configured to store computer-readable instructions and a processor 42 configured to execute the computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor is triggered to execute the method described.

[0081] The methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code configured to perform the methods shown in the flowcharts. When the computer program is executed by a processing unit, the functions defined in the methods of the present application are performed.

[0082] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. Computer-readable media may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0083] In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program configured to be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0084] Computer program code that can perform the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0085] The flow chart or block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code comprises one or more executable instructions configured to implement the logical function of the specification. It should also be noted that in some alternative implementations, the function marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs the function or operation specified, or can be implemented with a combination of dedicated hardware and computer instructions.

[0086] As another aspect, embodiments of the present application further provide a computer-readable medium, which may be included in the device described in the above embodiments, or may exist independently and not incorporated into the device. The computer-readable medium carries one or more computer-readable instructions, which can be executed by a processor to implement the steps of the methods and / or technical solutions of the above embodiments of the present application.

[0087] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0088] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0089] Computer-readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc-read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be configured to store information that can be accessed by a computing device.

[0090] In addition, an embodiment of the present application further provides a computer program, which is stored in a computer device, so that the computer device executes the method for controlling code execution.

[0091] It should be noted that the present application can be implemented in a combination of software and / or software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0092] It is obvious to those skilled in the art that the present application is not limited to the details of the above-mentioned exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. Words such as first and second are used to indicate names and do not indicate any particular order.

Claims

1. A pest situation monitoring method based on damaged images, wherein, the method includes: Performing damage processing on each sample pest image with at least one type of damage, and annotating the pest category and damage type corresponding to each sample pest image to obtain a training pest image set; Inputting the training pest image set into an adversarial network model for training to obtain a classification model; Inputting the pest image to be processed into the classification model to determine the pest category and damage type corresponding to the pest image to be processed.

2. The method according to claim 1, wherein, the damage type includes at least one of image illumination change, local image loss, or image color reset.

3. The method according to claim 1, wherein, the adversarial network model includes a feature extractor, a category classifier, and a domain classifier; wherein, the feature extractor is configured to perform feature extraction on the training pest image set to obtain a feature image set; the category classifier is configured to identify the pest category of each training pest image in the feature image set to determine the pest category corresponding to each training pest image; the domain classifier is configured to identify the damage type of each training pest image in the feature image set to determine the damage type corresponding to each training pest image.

4. The method according to claim 3, wherein, the adversarial network model further includes a gradient reversal layer; wherein, the gradient reversal layer is configured to perform gradient reversal on each training pest image in the feature image set to obtain a gradient reversal feature image set.

5. The method according to claim 4, wherein, the domain classifier is specifically configured to identify the damage type of each training pest image in the gradient reversal feature image set to determine the damage type corresponding to each training pest image.

6. The method according to claim 4, wherein, the construction process of the loss function of the adversarial network model includes: Determining the loss function of the adversarial network model according to the category classification loss, the domain classification loss, and the gradient reversal loss.

7. The method according to claim 5, wherein, determining the loss function of the adversarial network model according to the category classification loss, the domain classification loss, and the gradient reversal loss includes: Using the following formula to determine the loss function of the adversarial network model: L = L class + αL domain - λGR; where L represents the total loss of the adversarial network model, L class represents the class classification loss, and L domain represents the domain classification loss. GR represents the gradient reversal loss, and α and λ are hyperparameters.

8. A pest situation monitoring device based on damaged images, wherein, the device includes: A damaged image acquisition module configured to perform damage processing on each sample pest image with at least one type of damage, and annotating the pest category and damage type corresponding to each sample pest image to obtain a training pest image set; A damaged image training module configured to input the training pest image set into an adversarial network model for training to obtain a classification model; A damaged image processing module configured to input the pest image to be processed into the classification model to determine the pest category and damage type corresponding to the pest image to be processed.

9. A computer device, wherein, the device includes: One or more processors; and A memory storing computer program instructions, which when executed cause the processor to execute the damaged-image-based pest monitoring method according to any one of claims 1-7.

10. A computer-readable medium, wherein, computer program instructions are stored thereon, and the computer program instructions can be executed by a processor to implement the damaged-image-based pest monitoring method according to any one of claims 1-7.

Images