Intelligent mosquito trap for mosquito classification

Abstract

An insect trap includes a combination of one or more components for classifying insects by genus and species. The trap includes an imaging device, a digital microphone, and a passive infrared sensor at an entrance of the trap to sense wing beat frequency and size of the insects (to identify mosquito entry). A lamb skin membrane filled with an insect attractant such as carbon dioxide mixed with internal air gases mimics human skin so that the insects can land on the membrane and even puncture the membrane as if to enjoy a blood feast. An imaging device such as a passive infrared sensor or camera collects image data of the insects. The insects can be mosquitoes.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 198,254, filed October 7, 2020, and is incorporated herein by reference. Background Technology

[0003] Taxonomy is the process of classifying organisms in nature. Entomology is the study of insect organisms. Taxonomy within the context of entomology is a relatively obscure and difficult discipline in modern scientific times. Few people are willing to spend their entire careers staring at a microscope for hours trying to identify the genus and species of an insect. Mosquitoes and other insects are considered "vectors" because they carry viruses, bacteria, and disease strains and transmit them to humans. Therefore, the term "vector" has been given its broadest meaning in the field of infectious diseases.

[0004] Modern entomology focuses on eliminating or minimizing human involvement in mosquito classification during disease outbreaks. Public health experts can now identify which mosquito species they are looking for. Once they have identified the correct mosquito, they can take them to a laboratory for DNA testing and other methods to check for pathogens (i.e., viruses) within the trapped mosquitoes. Of course, if they find a significant number of mosquitoes carrying viruses, a public health crisis will occur, requiring corrective action.

[0005] Other efforts focus on detecting invasive mosquitoes at borders. This is a problem of global concern—the need to identify whether mosquitoes at a country's borders (land, sea, airspace, or roads) are invasive. For example, consider a scenario where mosquitoes, for instance, are both domestic and non-U.S.-native vectors, are on vehicles entering the U.S. border.

[0006] Assuming mosquito traps do exist at the border, this "new" species of mosquito would likely be caught along with other native mosquitoes. The problem here is how public health authorities can identify "foreign" mosquitoes in such a trap. Current entomological classification systems require regular trips to these traps to collect and examine the specimens under a microscope, identifying each sample individually. This would be extremely cumbersome if the goal is to detect only a specific type of "foreign" mosquito.

[0007] Current disease models rely on the accurate classification of vectors. Improvements are needed in entomological classification systems for specialized and detail-intensive instances, such as the hypotheses mentioned above. The field of entomological classification needs to include algorithms suitable for solving important but difficult-to-define problems, such as identifying hypothetical “exotic” mosquitoes that have actually been trapped. Updated algorithms are needed to provide researchers with options for entomological classification in specialized situations, such as the hypothetical random emergence of new insects affecting local populations.

[0008] Continuing with the example of "invasive" mosquitoes, the field of entomological taxonomy needs improved techniques and models that have been trained with images of invasive mosquitoes (provided by international partners) to identify genera and species directly from initial observations. Alternatively, the need should be met to enable invasive mosquitoes to conform to models trained with other mosquitoes. These techniques would allow researchers to notify public health officials that new mosquitoes previously unknown in a given location have been trapped. In either case, there would be significant public health benefits at border areas.

[0009] As detailed in this disclosure, to address the aforementioned shortcomings, digitizing the anatomy of mosquito specimens globally (with the participation of citizens and experts) would facilitate the creation of a large repository of mosquito anatomical images labeled with genus and species. This repository could then be used to train personnel and also for automatic identification using the algorithms described in this disclosure (when uploading photos). For example, but not limited to this disclosure, the Florida Medical Entomology Lab in Vero Beach trains a very small number of personnel annually (both international and domestic military personnel) in specific insect taxonomy areas. Previous investigations have shown that space is very limited, and many people are excluded from such training programs. With a digital repository in place, training programs can be scaled globally, and there are potentially thousands more images available to train interested personnel.

[0010] In at least one example, the need for such improvements in entomological classification is evident. Many states and districts in India (especially those near the border) are already and currently willing to pay for such services. These services are provided by appropriate mosquito traps and can be deployed on international flights, ships, and buses.

[0011] Another expression of need in this area is the routine training of soldiers traveling to countries where mosquito-borne diseases are common to help local communities identify mosquitoes and other disease vectors. Digital repositories could be used to train soldiers remotely, eliminating the need for them to travel to the locations where these services are needed. Additionally, soldiers, and even personnel in government agencies traveling and residing overseas, could benefit from mosquito traps at bases and / or homes that could alert them to the types of mosquitoes being trapped nearby and their level of danger.

[0012] Finally, in another expression of need in this field, there is a lack of digital progress in current mosquito traps. Most current traps are based on light attractants or carbon dioxide (CO2) attractants (sometimes enhanced with human sweat). Traps with these attractants lure mosquitoes in. Once inside the trap, a fan typically pulls the mosquitoes into chambers from which they cannot escape. Some traps use chemicals in these chambers to actually kill the mosquitoes. The next day, taxonomists collect the mosquitoes from these chambers to identify species. Other traps can detect whether an insect that has fallen in is a mosquito—they use passive infrared sensors (“PIR sensors”) to detect mosquitoes entering the trap. PIR sensors can incorporate thermoelectric sensors that detect changes in infrared radiation levels. PIR sensors are digitally / electrically adaptable to communicate with other computers via networks; however, so far, there is no ability to capture images or videos within the trap.

[0013] There is a need for a smart mosquito catcher that uses electronic devices to collect data related to mosquito anatomy, behavior, and movement, in order to classify mosquito samples at least according to their genus and species. Summary of the Invention

[0014] An insect trap includes a combination of one or more components for classifying insects according to genus and species. The trap includes an imaging device, a digital microphone, and a passive infrared sensor at its entrance to sense the wingbeat frequency and size of the insects (to identify insect entry). A lambskin membrane filled with an insect attractant, such as carbon dioxide mixed with internal air, mimics human skin, allowing the insects to alight on the membrane and even pierce it, as if enjoying a blood feast. An imaging device, such as a passive infrared sensor or a camera, collects image data of the insects. The insects may be mosquitoes.

[0015] This disclosure presents a system for designing state-of-the-art artificial intelligence (AI) technology, namely, a technique for classifying mosquitoes using digital images. Digital images can be analyzed based on a mask-based convolutional neural network to extract anatomical components of mosquitoes from the digital images and permanently archived based on genus, species, and other taxonomic criteria. Attached Figure Description

[0016] A patent application document or a patent published therefrom contains at least one drawing presented in color. Upon request and payment of the necessary fees, the Patent Office will provide a copy of the published patent or patent application with the color drawing. Reference will now be made to the drawing, which is not necessarily drawn to scale.

[0017] Figure 1 This is a schematic diagram of an intelligent trap assembly according to an embodiment of the present disclosure.

[0018] Figure 2 According to this disclosure, the pests are segmented pixel-by-pixel using bounding boxes within a neural network.

[0019] Figure 3 This is a schematic diagram of the output of the neural network according to the present disclosure, which segments pixels of corresponding body parts of pests and annotates images of pests.

[0020] Figure 4 This is a schematic diagram of the masked dissection result of performing convolutional neural network operations on a digital image according to this disclosure.

[0021] Figure 5A This is a schematic diagram of the thoracic anatomy result obtained by cropping a digital image using a convolutional neural network operation according to this disclosure.

[0022] Figure 5B This is a schematic diagram of the abdominal anatomy result obtained by cropping a digital image using a convolutional neural network operation according to this disclosure.

[0023] Figure 5C This is a schematic diagram of the wing anatomy result obtained by cropping a digital image using a convolutional neural network operation according to this disclosure.

[0024] Figure 5D This is a schematic diagram of the anatomical result of cropping a digital image by performing convolutional neural network operations according to this disclosure.

[0025] Figure 6A It is a first screenshot of the screen displayed by the software application (“app”) used in conjunction with the systems and methods disclosed herein.

[0026] Figure 6BIt is a second screenshot of the screen displayed by the software application (“app”) used in conjunction with the systems and methods disclosed herein.

[0027] Figure 6C It is a third screenshot of the screen displayed by the software application (“app”) used in conjunction with the systems and methods disclosed herein.

[0028] Figure 7A This is a fourth screenshot of the screen displayed by the software application (“app”) used in conjunction with the systems and methods disclosed herein.

[0029] Figure 7B This is a fifth screenshot of the screen displayed by the software application (“app”) used in conjunction with the systems and methods disclosed herein.

[0030] Figure 7C This is the sixth screenshot of the software application (“app”) used in conjunction with the systems and methods disclosed herein.

[0031] Figure 8 This is a schematic diagram of a convolutional neural network (“CNN”) configured to implement the computerized methods of this disclosure in a computer environment.

[0032] Figure 9A This is a schematic diagram of the segmentation results of the corresponding mosquito species, where each species is represented by the original image, the feature map is shown by the bounding box used by the corresponding convolutional network, and the separately segmented anatomical parts are separated from the original image for display on the graphics processing unit of the computer used in this paper.

[0033] Figure 9B This is a schematic diagram of the segmentation results of the corresponding mosquito species, where each species is represented by the original image, the feature map is shown by the bounding box used by the corresponding convolutional network, and the separately segmented anatomical parts are separated from the original image for display on the graphics processing unit of the computer used in this paper.

[0034] Figure 9C This is a schematic diagram of the segmentation results of the corresponding mosquito species, where each species is represented by the original image, the feature map is shown by the bounding box used by the corresponding convolutional network, and the separately segmented anatomical parts are separated from the original image for display on the graphics processing unit of the computer used in this paper.

[0035] Figure 10This is a schematic diagram of the training and testing of a convolutional neural network that provides merged segmentation results for corresponding bumblebee species. Each species is represented by an original image, feature maps are shown using bounding boxes used by the corresponding convolutional network, and individually segmented anatomical sites are separated from the original images for display on the graphics processing unit of the computer used in this paper. The training and testing are used to illustrate that the convolutional neural network used to identify mosquito anatomy can be adapted to other pest species.

[0036] Figure 11 It is a segmented image of a mosquito, in which components of the mosquito's anatomy are identified as foreground pixels and filtered out using the convolutional neural network of this disclosure.

[0037] Figure 12 This is a schematic diagram of a computer environment according to an embodiment of the present disclosure, wherein the neural network according to the present disclosure performs processing for anatomical structure segmentation and pest identification. Detailed Implementation

[0038] This disclosure presents a hardware design for an intelligent mosquito trap that automatically captures images, videos, and wingbeat frequencies of trapped mosquitoes. Once the digital data becomes available, artificial intelligence (“AI”) algorithms, such as those described below, are executed to identify the species of the trapped specimen. This disclosure presents design details and prototype implementations of an intelligent mosquito trap that can use any attractant (CO2, light, human sweat, animal odor, lactic acid, etc.). The trap incorporates the following components that make it unique: a) digital microphones / passive infrared sensors (PIRs) at the entrance of the trap, used to sense the wingbeat frequency and size of insects (i.e., to identify the entry of mosquitoes); b) a lambskin membrane filled with CO2 and containing air, which mimics human skin, allowing mosquitoes to land on the membrane and pierce it as if they were actually sucking blood; and c) multiple embedded cameras for capturing images of mosquitoes on the lambskin membrane for post-processing. Combining these components in a mosquito trap is novel and has never been attempted before.

[0039] Identifying mosquitoes from trapped samples is a time-consuming and labor-intensive process. Mosquitoes are attracted to traps placed in strategic locations. Experts then identify each mosquito individually by visual inspection through a microscope. Around the world, this process can take hours and is inefficient. With the disclosed smart trap, images, videos, and wingbeat frequencies are captured immediately when mosquitoes are attracted. AI algorithms then automate the process of identifying mosquito samples. Images, classification results, and extracted anatomical structures can be instantly transmitted to a remote control center. Currently, no such technology can record images, videos, and wingbeat frequency audio of trapped mosquitoes.

[0040] The terms used to describe devices are given their broadest, most common meanings. For example, an imaging device can be any device that collects information and / or transmits that information to a computer by identifying physical image features of an object. Thus, an imaging device can be any kind of camera, but it can also be a passive infrared sensor and other types of data collection devices that use light transmission.

[0041] In one non-limiting embodiment, the device for collecting mosquito samples is a CO2 trap that emits warm CO2 to attract mosquitoes (which can be modified into a light trap and has the ability to add more attractants). The mosquitoes are attracted to the trap because they have been induced to believe that the CO2 comes from a potential source of blood. Figure 1 This is an example diagram. Figure 1 The device 100 for collecting mosquito samples includes a camera 150 (including, but not limited to, a Raspberry Pi (R-Pi) based imaging device), a microphone 175, and a network or direct connection to a computer processor 1200, which stores software in a computerized memory configured to receive image and audio data to determine the classification of species within the camera's field of view. The device 100 also includes a membrane body 175 that mimics animal or human skin, through which mosquitoes may attempt to pierce and suck blood. While mosquitoes perch on the membrane body, the camera 150 or any other imaging device can collect image data within the field of view. The device also includes a sealed chamber 195 for collecting mosquitoes after data capture. The mosquitoes may be alive or dead and can be used to verify data collection through other manual methods. A cover 110 prevents mosquitoes of interest from flying away.

[0042] The membrane 175 is mounted on a brass positioning disc 165, thus possessing the following non-limiting features: 1) the membrane has a gas inlet and four holes to allow the membrane to be filled with CO2 gas; 2) the brass positioning disc has a central recess 167 for placing lactic acid, or any other type of mosquito attractant; 3) the brass positioning disc is heated using a heater 180 (e.g., a canister heater) and the temperature is monitored by a thermocouple (TC); 4) a Raspberry Pi processor controls the temperature to ensure the presence of warm CO2 to further enhance mosquito attraction. In some embodiments, the skin surface 185 is a lambskin preventative membrane, as lambskin is considered an excellent landing surface for mosquitoes. The skin surface 185 of the membrane 175 is attached to the brass positioning disc 165 using a rubber band, since the inner diameter of the skin is slightly larger than the outer diameter of the positioning disc. CO2 gas permeates from the interior of the membrane housing 160 into the membrane 175 and floats to the top of the device 100 to attract mosquitoes.

[0043] Camera 150 is triggered by its built-in microphone 155, which continuously listens inside the trap. In software stored in computer memory, the audio spectrum is converted into a frequency representation of the audio using a Fast Fourier Transform. The software then generates a frequency representation of the audio (e.g., an augmax function). If the frequency falls within the range of mosquito wingbeat frequencies, the software triggers the camera. In a non-limiting embodiment, this range is currently set to 400 to 1000 Hz. Based on performance, the user can change this value to increase or decrease the overall sensitivity. In some embodiments, a passive infrared sensor connected to the camera and other electronics detects whether an insect entering the trap is a mosquito.

[0044] This disclosure presents design details and prototype implementations of a smart mosquito trap that can use any attractant (CO2, light, human sweat, lactic acid, etc.). The disclosure proposes a trap that incorporates one or more of the following components to make it unique: a) a digital microphone / passive infrared sensor at the trap's entrance, used to sense the frequency and size of insect wingbeats (to identify mosquito entry); b) a lambskin membrane filled with CO2 and containing air, mimicking human skin so that mosquitoes can land on the membrane and pierce it (to lure them into believing they can have a blood feast); c) multiple embedded cameras for capturing images and videos of mosquitoes on the lambskin membrane for post-processing; d) a hardware platform capable of executing artificial intelligence algorithms, which processes the captured data to classify mosquitoes into genera and species; e) a depth-sensing camera system. The system is used to track and record flying mosquitoes with video / audio; f) a simple funnel-shaped structure with a reduced inlet diameter, which connects to the instrument chamber; g) an ultraviolet light source for detecting the presence of fluorescence within the insect; h) multiple chambers in the trap, each of which is triggered by an intelligent actuation system that causes mosquitoes (based on how they are classified) to fall into the appropriate chamber for collection; i) an option to provide yeast and water to the trap to attract pregnant mosquitoes; j) hardware for exporting data to a cloud server via WiFi / cellular connection; k) sensors in the trap for collecting ambient environmental data, including temperature, altitude, humidity, latitude / longitude, time, and ambient light sensors; l) hardware capable of freezing mosquitoes after they are captured in the trap.

[0045] In one embodiment, the device is a mosquito trap with a digital microphone and an infrared sensor at the trap's entrance to sense wingbeat frequencies. Both sensors are used to classify incoming mosquitoes. An active infrared sensor can be used as a tripwire to trigger the microphone and camera, allowing both instruments to record, and it can be used to confirm the wingbeat frequencies reported by the microphone. This is done by applying a Fast Fourier Transform to both and obtaining frequency maps from both sensors.

[0046] In another embodiment, a lambskin membrane filled with CO2 is used to mimic human skin and respiration. Mosquitoes can land on the membrane and puncture it (to lure them into thinking they can have a blood feast). The lambskin membrane is slightly porous to allow the CO2 gas to leak out slowly, creating a slow flow of CO2 to help attract mosquitoes to the trap. The membrane will be positioned below the sensor and camera to ensure that mosquitoes pass through the sensor.

[0047] In another implementation, an embedded camera is configured to capture images of mosquitoes resting on a surface. This allows the user to capture images from multiple different angles, enabling certain machine learning algorithms, as illustrated in the example below, to provide multiple different views to the classifier. Providing the classifier with different views will allow for better classification based on the images and better extraction of insect anatomy.

[0048] In another implementation, using an embedded hardware platform allows machine learning algorithms to be executed to process the captured data for mosquito classification. To reduce inference time when using these platforms, the device will also employ a dedicated ASIC processor, such as the Google Coral Accelerator module. This will allow for reduced latency and lower network traffic.

[0049] In another implementation, a global shutter camera will also be used to capture images of mosquitoes in flight. Being able to capture images of flying mosquitoes will allow users to reduce the size of the traps and may allow avoiding keeping all landing surfaces together. A global shutter camera will also provide sharper images by reducing motion blur, which will help machine learning algorithms better classify the images.

[0050] A simple funnel shape will be used to reduce the diameter of the inlet at the top of the device. In one non-limiting embodiment, the inlet channel will be reduced by a factor of three. This funnel shape will also be attached to the top of the sensor chamber containing the infrared sensor and microphone. The funnel shape is designed to reduce the number of mosquitoes entering the trap, allowing the sensor to detect one mosquito at a time.

[0051] In other non-limiting embodiments, an ultraviolet light source will be used to detect the presence of fluorescence on the insects. With the release of genetically modified mosquitoes in South Florida, the ability to detect them is crucial. These modified mosquitoes possess fluorescent tags that identify them as genetically modified. Under suitable lighting conditions provided by ultraviolet light, a camera will be able to capture these tags. The system will be able to count these tags as they enter the trap.

[0052] In some implementations, the base of the trap will have multiple chambers. These chambers will be triggered based on the classification of a mosquito or other insects that fly into the trap. Based on the classification, an intelligent actuation system using a servo mechanism / motor allows the sample to fall for collection in the appropriate chamber. If it is a common vector mosquito, it is collected in chamber 1. If it is an Aedes aegypti, it is collected in chamber 2. If it is an Aedes aegypti and it has a fluorescent tag, it is collected in chamber 3. If it is another insect, it is collected in the general chamber.

[0053] For attractants, the trap will have several options. Users can use yeast or dry ice to generate CO2. The CO2 will be distributed using a solenoid valve in the trap to control the gas output only during specified periods. Alternatively, water can be used to attract pregnant mosquitoes that have had a blood meal and are looking to lay eggs. Users can also use commercially available solid attractants to provide a more cost-effective attractant solution for the trap.

[0054] The trap will send the categorized audio and images to a cloud server for users to access at will. To enable this, the trap will have WiFi and cellular modules to allow users to connect to networks of their choice. For remote locations, a satellite communication module can be added to allow low-bandwidth communication for obtaining counts and other critical information. Furthermore, the trap can use LoRa for long-range radio communication without requiring satellite communication.

[0055] Additional sensors on the trap will allow for the collection of ambient environmental data. This data will include temperature, air pressure, altitude, humidity, latitude, longitude, time, CO2 concentration, and ambient light conditions. The sensors we will use include a Bosch BME280 for temperature and humidity, a Bosch BMP390 for air pressure and altitude, a MediaTek MTK3339-based GPS module for latitude and longitude, a Sensirion SCD-40 for CO2 concentration, and a Rohm BH1750 for ambient light.

[0056] While the mosquitoes are inside the chamber, the trap is able to freeze them. This is accomplished using a Peltier module to lower a metal plate below freezing. The Peltier module, assisted by a fan, further reduces the plate's temperature and allows it to be maintained at a constant temperature. Freezing the mosquitoes slows down sample decomposition and allows the user to freely conduct further research on the captured mosquitoes.

[0057] As described above, device 100 is a smart trap that can be used to capture and collect data about mosquitoes. The remainder of this disclosure provides examples of various image and data analyses that can be used as described above in artificial intelligence. Accordingly, this disclosure incorporates by reference U.S. Patent No. 10,963,742, entitled "Leveraging Smart-Phone Cameras and Image Processing Techniques To Classify Mosquito Genus and Species," published March 30, 2021. Patent '742 discloses a computerized method for identifying insect samples (such as genus and species of mosquitoes) and includes collecting multiple digital images of insect samples located in a set of corresponding image backgrounds. This disclosure continues by extracting image portions from each digital image, wherein the image portion includes subject pixels of image data corresponding to the insect sample and excluding image background pixels. The method also includes converting the subject pixels into a selected color space dataset and identifying texture features of the image portion from the selected color space dataset.

[0058] This disclosure also incorporates, by reference, U.S. Patent No. 11,048,928, entitled "Systems and Methods of Entomology Classification Based on Extracted Anatomies," published June 29, 2021. Patent '928 discloses a deep neural network framework for extracting anatomical components from mosquito images, such as, but not limited to, the thorax, wings, abdomen, and legs. This technology is based on... Figure 8 The concept of Mask R-CNN 800 involves artificial intelligence iteratively learning a feature map 808 from an image 802, placing anchor points (shown as bounding boxes 812 in the figure, but can be of any shape) around the foreground components, and then performing segmentation 820 and classification 824 on the pixels corresponding to the anatomical components within the anchor points. In some embodiments, results of this disclosure show that the technique is advantageous when interpreted in the context of being able to collect descriptive morphological markers for classifying mosquitoes.

[0059] In one embodiment of U.S. Patent No. 11,048,928, a system for identifying the genus and species of insects includes an imaging device configured to generate images of insects. A computer processor is connected to a memory that stores computer-implemented commands in software, and the memory receives images, wherein the software implements the following computerized method on the corresponding images, the method beginning with the steps of: applying a first convolutional neural network to the corresponding image to generate a feature map relating to anatomical pixels in the corresponding image that correspond to body parts of the insect. The computer then applies anchors to the feature map, wherein the anchors identify portions of the corresponding layer of image data in the feature map containing the corresponding anatomical pixels of the corresponding body parts. Generating a mask allows the system to segment the corresponding anatomical pixels from the corresponding layer. The system extracts fully connected layers from the corresponding layers to which the first convolutional neural network has been applied. The fully connected layers are applied to a regressor network and a classification network, wherein generating the mask for segmentation, applying the fully connected layers to the regressor network, and applying the fully connected layers to the classification network are parallel operations performed by software.

[0060] Also in an embodiment of U.S. Patent No. 11,048,928, a system and method for extracting information relating to anatomical components of a living organism from digital images are disclosed. The method includes training a mask-based convolutional neural network with a set of training images segmented using a computerized algorithm that identifies ground-based anatomical components with a set accuracy. Training includes: classifying corresponding anatomical components in the training images and comparing the training images with ground-based images; tightening bounding boxes surrounding the anatomical components in the digital images; and generating a mask for extracting information from a second set of images. A general discussion of CNNs and related terminology can be found in the numerous references cited below. For example, Reference 35 (Shepard) explains how to convolve filters consisting of multiple kernels (weighted matrices) onto a raw image to create feature maps with multiple layers and adaptive data density. Shepard explains using feature maps to pool certain layers using techniques such as max pooling, which separates those feature maps by their maximum values ​​to reduce complexity. A Corrected Nonlinear Unit (Re-LU) dataset is added to the feature map to identify regions that should be considered but were missed when the variation from one pixel to the next is below a filtering threshold. In very simple terms, Re-LU works on the image to produce features that can be appended to... Figure 8The activation function of the layer for the feature map shown. Generally, in some non-limiting implementations, Re-LU may retain a specific filter value at the corresponding output matrix index, or insert zeros if the specific index value is negative. Thus, the overall concept of a convolutional neural network incorporates convolutional layers as feature maps of the original image, pooling layers and ReLU layers for adding details, and fully connected layers as the combined, data-rich output. As described in reference 35 (Stewart), such as Figure 8 The fully connected layers shown in the non-restrictive examples aggregate all the information into the final replicated image.

[0061] In some aspects, this disclosure relates to computerized devices, computer-implemented methods, and computerized systems for identifying insect specimens, such as but not limited to mosquitoes, using digital image analysis. This disclosure proposes a system in which a user (expert or ordinary citizen) takes a photograph of a mosquito or other pest using a smartphone and then immediately sends the image along with GPS information data from the smartphone to a central server.

[0062] The server will implement the algorithms described in this disclosure to: a) identify the genus of mosquitoes; b) identify the species of mosquitoes; c) segment the main body of an image into objects of interest, such as wings, legs, beak, abdomen, scutes, etc.; d) provide feedback to the user regarding the species, information about what diseases the species carries, and more interesting information such as flight range. A potential application is mosquito identification, as it is currently a challenging and cognitively demanding problem. School districts could also use this software application to teach children biology and other scientific fields, as such scientific analytical skills may eventually become mandatory in schools in many areas. Defense and Homeland Security agencies, as well as other government agencies, may recognize the need for the computerized applications described herein.

[0063] One non-limiting value claim of this disclosure is the ability to bypass human classification (currently through microscopic observation) and instead use digital cameras and the proposed technology for automatic classification of genera and species types. A second value claim is that a system with large-scale, labeled, citizen- and expert-generated images can begin digitizing the anatomy of mosquitoes globally. This database can bring incalculable value to mosquito, and especially vector surveillance, training, and global information sharing.

[0064] Although the exemplary embodiments of this disclosure have been explained in detail herein, it should be understood that other embodiments are contemplated. Therefore, this disclosure is not intended to be limited in its scope to the details of the construction and arrangement of the components set forth in the following description or shown in the accompanying drawings. This disclosure is capable of other embodiments and can be practiced or implemented in various ways. For example, the test results and examples all relate to identifying the genus and species of mosquitoes based on mosquito traits and characteristics extracted from digital images. However, the techniques and concepts utilized and claimed in this disclosure are not limited to mosquitoes but can be used in conjunction with identification processes for other species such as animals, humans, plants, etc.

[0065] Figure 1 A prior art system utilizing a pest trap 100 is shown, which captures pests, photographs them using a built-in camera 150, and sends the photographs to the cloud. In the cloud, algorithms implemented by computers and servers on various networks are designed to identify the type of pest. This information is fed back to farmers, who can then plan appropriate treatments for their crops. However, anatomical structures are not extracted in this product.

[0066] Figure 2 The results of a non-limiting example of the method used for digital segmentation of insects 200 (i.e., mosquitoes as shown in the figure) using a convolutional neural network (CNN) are shown. This procedure is based on the concept of Mask R-CNN as described in the following article cited below: He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick, "Mask r-cnn." from Proceedings of the IEEE international conference on computer vision, pp. 2961–2969. 201, cited as reference 33. Another example of CNN is discussed in an online resource entitled "Simple Introduction to Convolutional Neural Networks" by Dr. Matthew Stewart (cited as reference 35). Both articles are incorporated by reference as if each were fully described herein. This disclosure utilizes advances in such convolutional neural networks (CNNs) and segments pixels containing various anatomical components of interest (i.e., relative to the bounding boxes of existing CNNs) by adding a branch parallel to the branch used for recognizing bounding boxes of existing CNNs for predicting object masks. Figure 8 (Further discussion on pixel-by-pixel segmentation). Figure 2This illustrates one step of CNN anatomical segmentation, which uses bounding boxes 210, 220, 230, 240 trained by regression to separate and identify corresponding pixels on each of the corresponding chest, wing, abdomen, and leg portions of an image. As mentioned above, the term "bounding box" is merely exemplary, for example, because the contours used for segmenting anatomical structures can take any shape, and Figure 2 The frame or rectangle is not a limitation of this disclosure.

[0067] This approach involves several challenging steps. The first step involves training a model using a pre-trained convolutional neural network (CNN) to generate proposals related to certain regions within an image where objects may be present. Without limiting this disclosure, an example implementation uses ResNet101 as the backbone convolutional model. The model is initialized using pre-trained weights from the MS COCO dataset. The MS COCO dataset is disclosed in Lin, Tsung-Yi, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick, “Microsoft COCO: Common objects in context,” from the European conference on computer vision, pp. 740–755; Springer, Cham, 2014, cited as reference 34, which is incorporated herein by reference in its entirety as if fully described herein. The MS COCO dataset is a large-scale object detection dataset. It contains 1.5 million object instances and 80 object categories.

[0068] ResNet has been largely described in He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep residual learning for image recognition.” from Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. 2016, which is incorporated herein by reference in its entirety as if fully described herein and is cited as reference 33. A key feature of ResNet is its very deep network and the introduction of residual connections to obtain input from one layer to the next. Residual connections help address the vanishing gradient problem by detecting the minimum differences between convolutional layers. The next step is to design an object detector network that performs three tasks: classifying bounding boxes 210, 220, 230, and 240 according to their corresponding anatomical structures; shrinking the boxes; and generating a mask 818 for each anatomical component (i.e., pixel-wise segmentation 820). In constructing the architecture of the object detector network, the non-limiting example disclosed herein uses pixel-wise sigmoid and binary cross-entropy loss functions (to identify "k" anatomical components) and trains them rigorously.

[0069] Regarding the generation of the training dataset, the non-restricted example utilizes a tool to create masks for each anatomical component in the object dataset. To begin training, the procedure first annotates 571 mosquito images using the VGG image annotator tool, which is itself a very tedious task. Figure 3 An example of annotated image 300 is shown. Of the 571 mosquito images previously annotated by experts, 404 images were selected as training images and 167 images as validation images. The methods and systems of this disclosure then iterate the model against these known accurate training datasets to optimize the weights and hyperparameters. For example, the implementation of this paper optimizes hyperparameters such as the base feature extractor model, learning rate, momentum, optimizer, number of steps per round, number of validation steps, and number of rounds—all parameters to be set in the convolutional neural network (CNN) used herein. Non-limiting details are shown in Table 1 below, for example.

[0070] Table 1.

[0071]

[0072] The metric for measuring the accuracy of the mask R-CNN algorithm is mAP (average precision across all classes). It is calculated by averaging the average precision across all classes at all IoU thresholds, and is 0.833. The IoU metric measures the intersection of the ratios of pixels belonging to the ground truth within the bounding box and the union of the ratios of predicted pixels within the box to the ground truth. In one design, the IoU threshold is set to 0.75. Figure 3 This illustrates how training images 300 can be annotated to describe anatomical regions represented by corresponding anatomical region pixels 310, 320, 330, and 340. In a non-limiting embodiment, the computer-implemented software package will also provide additional annotations for training purposes. Figure 3 The segmented body parts shown are given written anatomical names. Taking an accurately segmented image as an example, each of the hyperparameters in Table 1 can be optimized to replicate the results of the trained dataset.

[0073] Figure 4 The masked anatomical structure 400 in the results is shown, where bounding boxes 410, 420, 430, and 440 are refined through a regression process performed by a convolutional neural network on pixels corresponding to specific parts of the anatomical structure. Figures 5A to 5D This demonstrates how convolutional neural networks work via... Figure 8 The procedure outlined in the document locates these corresponding pixel sets and, after extracting 470 background pixels, cropped the anatomical results for the sample's chest (510 pixels), abdomen (520 pixels), wings (530 pixels), and legs (540 pixels). The results are based on cloud storage implemented via the network.

[0074] The non-limiting embodiments of this disclosure bring about the development and design of smartphone applications in Android and iOS that enable users to capture images of mosquitoes or select an image from the phone's local storage. The application will classify the mosquitoes and also extract anatomical pixels corresponding to the anatomical structures of interest—the thorax, wings, abdomen, and even the legs—as shown in Figures 6 and 7.

[0075] This disclosure incorporates a framework based on a mask-region-based convolutional neural network to automatically detect and individually extract pixels corresponding to anatomical components of mosquitoes (specifically the thorax, wings, abdomen, and legs) from images. In one non-limiting embodiment, the training dataset consists of 1500 smartphone images of nine mosquito species trapped in Florida. In the proposed technique, the first step is to detect anatomical components within the mosquito images. Then, as discussed herein, the system and method of this disclosure localize and classify the extracted anatomical components while adding branches to the neural network architecture to segment pixels containing only anatomical components.

[0076] In order to evaluate generality, this disclosure, in the case where the architecture was trained using only mosquito images, is as follows. Figure 10 The example architecture for testing bumblebee images is shown. The procedures disclosed herein have the advantageous results indicated.

[0077] Mosquito-borne diseases remain a major public health problem. Currently, mosquito vector surveillance remains a manual process worldwide. The steps involve placing traps, collecting samples, and identifying each sample individually under a microscope to determine its species. Unfortunately, this process is cognitively demanding and can take hours to complete. This is partly because the mosquitoes falling into the traps include both disease vectors and many non-vector mosquitoes. Recently, AI methods have been designed to automate mosquito classification. Work similar to designing machine learning models (References 1-4) is based on manually designed features from image data generated by smartphones or digital cameras. Two recent papers designed deep neural network techniques (which do not require manually designed features) to classify mosquitoes from image data generated via smartphones (References 5, 6). Other work processes the sound of mosquito flight for classification, based on the concept that wingbeat frequencies are unique to all mosquito species (References 7-10).

[0078] In this disclosure, the work demonstrates a novel application of mosquito images when processed using AI technology. The most descriptive anatomical components of mosquitoes are the thorax, abdomen, wings, and legs, and this disclosure proposes a technique for extracting only the anatomical pixels corresponding to these specific anatomical components from any mosquito image. This technique is based on a convolutional neural network based on mask regions (Reference 11).

[0079] This disclosure utilizes procedures that employ convolutional neural networks (CNNs), including Figure 8 The feature map 808 is shown. In CNN theory, the neural network 800 applies weighted filters 806 to image 802 to achieve a specific purpose. In a very simple sense, filters 806 are weighted to extract very precise portions (called features) from image 802. In some implementations, the weights are designed based on a training scenario that explores gradient changes representing edges in certain parts of the image. The corresponding result of applying each filter and its weights to the image is feature map 808. Although Figure 8Feature map 808 is shown as a two-dimensional schematic, but this disclosure includes feature maps in which a CNN applies feature filters having kernel layers that output multi-layer feature maps. In other words, feature map 808 may have corresponding image data layers. Image data may be processed globally as a set, or individually as corresponding layers to which convolutional neural network 806 is applied. The first step of this disclosure includes extracting the corresponding feature map 808 of anatomical features of interest from a training dataset of 1500 smartphone images covering 200 mosquito samples of nine species trapped in Florida. The neural network 806 that extracts feature map 808 is a ResNet-101 with a feature pyramid network (reference 12) (which can process images at multiple scales, and one of the scales is well-suited to the architecture of our problem).

[0080] like Figure 8 As shown, the system and method of this paper include applying bounding boxes 817, which are tuned to label corresponding features of an image, such as feature map 808 of corresponding anatomical parts of a mosquito's body in an example of this disclosure. Bounding boxes 812, 817 are organized according to a region proposal network 816 for each feature of interest. As mentioned above, the examples in the figures are not limiting, as the bounding boxes can take any shape, including but not limited to rectangles. In a non-limiting example, as shown at reference 816, bounding boxes 812, 817 are proposed based on a computer-driven system that learns from a training set that if a gradient occurs in a specific region of a convolutional image layer, a gradient change in the pixels of the corresponding convolutional image layer may correspond to a specific anatomical feature. The system and method utilize regression processes, loss theory, and feedback from one feature map to the next to make the bounding boxes increasingly accurate and more adapted to a feature of interest (e.g., the thorax, legs, abdomen, and wings of an insect). The proposed pixel separations or segments correspond to these features of interest.

[0081] In some non-limiting embodiments, the process of refining bounding boxes 812, 817 for corresponding anatomical pixel groups constituting body parts is paired with an alignment process 814, which ensures that the output of the region proposal network 816 still matches the contours shown in the original feature map 808. Once this alignment is completed, and as... Figure 8As shown in the non-limiting example, the system and method disclosed herein perform masking operations or pixel extraction in a second convolutional neural network 818. The second convolutional neural network provides a segmented image 820, where some examples produce anatomical pixels corresponding to the thorax, abdomen, wings, and legs of an insect. The output of the bounding boxes 812 applied by the region proposal network 816 is also fed into a fully connected neural network layer 822. Notably, the second convolutional neural network 818 utilizes filtered convolutional layers, such that each "neuron" or matrix index within the data layer performing the convolution is computed individually and is more sparse. The fully connected layer 822 more tightly tracks each prior layer and is richer in data. The last fully connected layer is fed to both the classifier 824 and the bounding box regressor 826. The fully connected layer 822 is actually interconnected layer by layer, neuron by neuron, as indicated by the arrows. The final fully connected layer 834 is the output layer and includes all the data from all layers. In separate parallel operations, a bounding box regressor 826 and a classification processor 824 are applied to each layer of a first convolutional neural network 806 and / or a second convolutional neural network 818. The bounding box regressor 826 utilizes error function analysis to more accurately regress the bounding box 812 around the corresponding feature of interest. This feedback loop 850 ensures that the bounding boxes 812, 817 of the region proposal network 816 provide convolutional image layers that are distinct for each feature explored by the feature map 808. The classifier 824 provides an automated, computerized process to identify and label the corresponding anatomical pixel groups 828, thereby identifying each anatomical region of the object insect from the original image 802.

[0082] Subsequently, this public statement elaborates on how... Figure 2 The rectangular anchor point shown is only used for the step of detecting and locating anatomical components (represented as foreground) in the image. The term anchor point is a generalized term for the "bounding boxes" 812, 817 described above. Therefore, anchor points can take any shape, and rectangular bounding boxes 812, 817 are non-limiting examples of anchor points. Once the foreground is detected, the next step is to mask each component present in the foreground by adding branches (i.e., "extracting its pixels") for the foreground pixels (e.g., in the background pixels 470). Figure 4 Refer to 410, 420, 430, and 440 for splitting. This extra branch is... Figure 8 The second convolutional neural network 818 is shown in the middle. The second convolutional neural network runs in parallel with two other branches 824 and 826 to classify the extracted rectangular anchor points and tighten them to improve accuracy via feedback loop 850. Figure 11 The diagram illustrates how the foreground pixels corresponding to the chest 1125, abdomen 1135, wings 1145, and legs 1155 are extracted and reconstructed into an image with a string identifier label superimposed on it, as shown in the figure.

[0083] The evaluation of this technology revealed favorable results. For example... Figure 4 As shown, the anatomical pixels corresponding to the thorax, wings, abdomen, and legs are extracted with high precision (i.e., very low false positives). However, for the legs, in some non-limiting embodiments, the false negatives are higher than for other parts because the number of background pixels far exceeds the number of leg pixels in the image. Nevertheless, it can be seen that sufficient descriptive features are indeed extracted within the mosquito's legs because the mosquito's legs are long, and descriptive features such as color bands do indeed repeat throughout the legs.

[0084] This disclosure explains that extracting images of mosquito anatomy has implications for: (a) faster classification of mosquitoes in the wild; (b) new, large-scale, low-cost, digital-based breeding programs for taxonomists; and (c) new, attractive tools that stimulate broader public participation in scientific work. Furthermore, to assess general applicability, this disclosure incorporates tests conducted on architectures trained on mosquito images using images of bumblebees (which are important pollinators).

[0085] In summary, the results demonstrate excellent accuracy in wing extraction and, to a certain extent, thorax extraction, proving the versatility of this technique for many insect classes. Training has enabled a mask-based region-based convolutional neural network (Mask R-CNN) to automatically detect and individually extract anatomical pixels corresponding to the anatomical components of the mosquito's thorax, wings, abdomen, and legs from images. For this study, this disclosure describes 23 samples of *Aedes aegypti* and *Aedes infirmatus*, and 22 samples of *Aedes proboscis*, *Anopheles crucians*, *Anopheles quadrimaculatus*, *Anopheles stearnii*, *Culex coronator*, *Culex nigricans*, and *Culex salinarius*. The dataset consists of 1600 mosquito images after the samples were imaged via multiple smartphones. These mosquito images were divided into 1500 images used to train the neural network and 100 images used for validation. This dataset generated a total of 1600 chest images, 1600 abdomen images, 3109 wing images, and 6223 leg images. This data was used for training on an Nvidia graphics processing unit (GPU) cluster with four GeForce GTX TITAN X cards. Figure 8 The architecture shown has each card with 3,583 cores and 12 GB of memory. Training and validating the architecture takes 48 hours.

[0086] For the tests described in this disclosure, the study described herein trapped another group of 27 mosquitoes (i.e., three of each species) and imaged them (via a smartphone). The test dataset consists of 27 images of the thorax and abdomen, 48 images of the wings, and 105 images of the legs. One implementation presents the extraction of images 925A-925C from one sample of nine species in the test dataset. Figure 9A , Figure 9B , Figure 9C The results of the techniques for analyzing the anatomical components of the mosquito 945A-945C, 955A-955C, 965A-965C, and 975A-975C. These figures represent... Figure 9A , Figure 9B , Figure 9C All other images tested under each category are shown, including the original images 925A, 925B, and 925C, the corresponding output layers 935A, 935B, and 935C of the convolutional neural network, and the extracted anatomical pixels corresponding to the corresponding chest portions 945A, 945B, and 945C, abdominal portions 955A, 955B, and 955C, wing portions 965A, 965B, and 965C, and leg portions 975A, 975B, and 975C. The anatomical components are indeed clearly derived from image data 935A-935C, which was processed by the convolutional neural network using bounding boxes.

[0087] Next, the system and method in this paper quantify the performance of the entire dataset using four standard metrics: precision, recall, intersection-to-union (IoU), and average precision across all classes (mAP). Precision is essentially the fraction of relevant instances (pixels in this case) out of those retrieved instances (again, pixels). Recall is the fraction of actually retrieved relevant instances. IoU is a metric that evaluates the area ratio of the intersection and union between predicted pixels and the ground truth. A higher IoU means more overlap between the prediction and the ground truth, and therefore implies better classification. To define the final metric, average precision across all classes (mAP), this disclosure defines another metric, average precision (AP), which is the average of all precision values ​​for a range of recalls (0 to 100 for our problem) across a specific category in the four categories (i.e., wings, chest, legs, and abdomen) of our problem at a given preset IoU threshold. This metric essentially balances precision and recall for a class at a specific IoU value. Finally, average precision across all classes (mAP) is the average of the AP values ​​across all four categories.

[0088] For different IoU values, the precision and recall values ​​for the validation and test datasets are presented in Tables 1 and 2, respectively. As shown in the figure, the performance metrics for the validation dataset during training match the metrics for testing (i.e., unseen images) across all IoUs and during post-training. This is compelling evidence that the architecture is robust and not overfitting.

[0089] Precision was high for all categories, meaning low false positives. Recall was also high for chest, abdomen, and wings, indicating low false negatives for these categories. However, recall for the leg category was relatively poor. It turns out that in a non-restrictive architecture, a significant portion of leg pixels are classified as background. While this might seem a bit discouraging, Figure 9A , Figure 9B , Figure 9C In this study, the leg segment was still correctly identified and extracted using the disclosed architecture (attributed to high accuracy). Therefore, the goal of collecting morphological markers from all anatomical components was still achieved. Finally, Table 3 presents the class-wide average accuracy for all categories. The lower values ​​in Table 3 are due to the poorer classification performance of the leg segment compared to the thorax, abdomen, and wings.

[0090] Table 1. Precision and recall on the validation set for different IoU thresholds

[0091]

[0092]

[0093] Table 2. Precision and recall for different IoU thresholds on the test set.

[0094]

[0095] Table 3. mAP scores for masking

[0096]

[0097] This disclosure also includes Figure 10 The results of a small-scale experiment using bumblebee images are presented. The information in this paper then validates how an AI architecture trained solely on mosquito images performs when tested with bumblebee image 1025, after image 1025 has been processed by the neural network as disclosed herein, to segment bee anatomy using bounding box image 1035. Bumblebees (genus: *Bumblebee*) are important pollinators, and detecting them in nature is crucial. Figure 10Example results are presented, representing one of the three species of bumblebees, but these results only represent over 100 bumblebee images we tested. The bumblebee images are sourced from the Smithsonian National Museum of Natural History in Washington, D.C. See reference 13 for images. Figure 10 As shown, a non-limiting technique according to this disclosure is robust in detecting and extracting wing pixels 1065. While thorax pixels 1045 are mostly extracted correctly, the ability to extract abdomen pixels 1055 and leg pixels 1075 is relatively poor. With these results, the confidence level for extracting wings from a wide range of insects using the disclosed architecture is quite high. With appropriate ground-based data, only minimal fine-tuning of the architecture is required to ensure robust extraction of all anatomical components from a large range of insects.

[0098] This disclosure includes the following discussion regarding the significance of the contributions made herein.

[0099] (a) More rapid classification of trapped mosquitoes. In situations where vector-borne diseases are a problem worldwide, the standard practice is to set up mosquito traps, return the next day to collect samples, freeze them, and take them to a facility where taxonomists identify each sample individually under a microscope to classify them into genera and species. This process takes several hours each day and is highly cognitively demanding. During the rainy season and disease outbreaks, hundreds of mosquitoes are trapped, and processing a single batch from a single trap can take an entire day. Based on the technology described herein, mobile cameras can assist in taking high-quality photographs of the trapped mosquito samples, and the extracted anatomical structures can be used by experts to classify them by viewing digital monitors (rather than through a microscope). This reduces the cognitive stress on taxonomists and speeds up the monitoring process. In a non-limiting embodiment, Table 4 presents details of the morphological markers that taxonomists seek to identify mosquitoes, which were used in this study and are further discussed in reference 14.

[0100] Table 4. Anatomical components and markers that contribute to mosquito classification (References 6-30).

[0101]

[0102]

[0103] (b) AI and cloud-supported education for training the next generation of taxonomists.Currently, the process of training taxonomists worldwide involves very few training institutions that store frozen specimens of both local and non-local mosquitoes. Those interested in these programs include not only professional taxonomists but also amateur enthusiasts. The associated costs of storing frozen mosquitoes are not negligible (especially in countries with low economic levels), severely limiting access to these programs and making recruitment very costly. Technologies similar to those disclosed herein enable digital support for trainees. Benefits include, but are not limited to, the possibility of remote education, reduced institutional operating costs, lower recruitment costs, and the opportunity to recruit more trainees. These benefits, when realized in practice, will have a positive impact on taxonomy, entomology, public health, and other fields.

[0104] (c) Digital preservation of insect anatomy under the threat of extinction. Recent reports have indicated that insects are rapidly disappearing. As more citizen scientists explore nature and share data to identify species facing immediate threats, digitally preserving insect morphology itself can help conserve insects. Preserving insect images can also help cultivate future scientists in various fields.

[0105] Image processing examples

[0106] Image dataset generation and preprocessing. In the summer of 2019, the study involved collaborating with the mosquito control district of Hillsborough County, Florida, to deploy outdoor mosquito traps over several days. Each morning after the traps were deployed, the method disclosed herein involved collecting all captured mosquitoes, freezing them in portable containers, and taking them to the county laboratory, where taxonomists identified the mosquitoes. This study utilized 23 specimens of *Aedes aegypti* and *Aedes nigra*, and 22 specimens of *Aedes proboscis*, *Aedes catastrophe*, *Anopheles tetrastigma*, *Anopheles skrvii*, *Culex pipiens*, *Culex nigricans*, and *Culex salsa*. Notably, specimens of eight species were identified as being trapped in the wild. The *Anopheles skrvii* specimens were reared separately in the laboratory; their ancestors were originally trapped in India.

[0107] Each sample was then placed on a plain, flat surface and imaged using smartphones (iPhone 8, 8 Plus, and Samsung Galaxy S8, S10) under normal indoor lighting conditions. To capture the images, the smartphones were attached to a movable platform 4–5 inches above the mosquito samples, and three photos were taken from different angles: one directly above and two at a 45-degree angle to each other. As a result of these procedures, a total of 600 images were generated. 500 of these images were then preprocessed to generate the training dataset, and the remaining 100 images were used for validation. For preprocessing, the images were reduced to 1024 × 1024 pixels for faster training (without sacrificing accuracy). The images were enhanced by adding Gaussian blur and randomly flipping them from left to right. These methods are standard in image processing and better account for variations during runtime. Following this procedure, the training dataset was increased to 1500 images. It should be noted here that all mosquitoes used in this study were vectors of disease and ailments. Of these vectors, the Aedes aegypti mosquito is particularly dangerous because it transmits Zika, dengue, chikungunya, and yellow fever. This mosquito is now found worldwide.

[0108] A deep neural network framework based on Mask R-CNN. To achieve the goal of extracting anatomical components from mosquito images, a straightforward approach is to try a Gaussian mixture model to remove the background from the image. See References 1, 15. However, this only removes the background and cannot extract the anatomical components in the foreground separately. Other state-of-the-art methods exist in this field. One technique is U-Net, see Reference 16, which proposes semantic segmentation based on deep neural networks. However, this technique is not suitable for instance segmentation (i.e., segmentation and labeling pixels across multiple categories). Multi-task network cascade (see Reference 17) is an instance segmentation technique, but it is prone to information loss and is not suitable for complex images like mosquitoes with multiple anatomical components.

[0109] Fully Convolutional Instance-Aware Semantic Segmentation (see Reference 18) (FCIS) is another instance segmentation technique, but it is prone to systematic errors on overlapping instances and produces undesirable false edges. See DeepMask at Reference 19, developed by Facebook, which extracts a mask (i.e., pixels) and then uses a fast R-CNN technique (Reference 20) to classify the pixels within the mask. However, this technique is slow because it cannot perform segmentation and classification in parallel. Furthermore, it uses selective search to find regions of interest, which further increases the latency of training and inference.

[0110] On one hand, this disclosure utilizes Mask R-CNN (see reference 11), a neural network architecture for extracting masks (i.e., pixels) corresponding to objects of interest within an image. It eliminates selective search and also uses a Region Proposal Network (RPN) (reference 21) to learn the correct regions of interest. This approach is best suited for fast training and inference. Furthermore, it employs excellent alignment techniques on the feature maps, which helps prevent information loss. Figure 8 The basic architecture is shown in the figure. To adapt it to the problems addressed in this disclosure, the following series of steps are required.

[0111] 1. Annotation of Ground Realities. First, this study involves manually annotating the training and validation images using the VGG Image Annotator (VIA) tool, as described in Reference 22. To achieve this, this disclosure demonstrates manually (and carefully) placing boundary polygons around each anatomical component in our training and validation images. Pixels within the polygons and associated labels (i.e., chest, abdomen, wing, or leg) serve as ground realities. Figure 4 The image shown is a sample annotation image.

[0112] 2. Feature maps are generated using CNNs. The system and method disclosed in this paper then use semantically rich features from the training image dataset to identify (e.g.) Figure 8 Image 802 shows the complex anatomical components of a mosquito. To achieve this, a non-restrictive neural network architecture 800 is a combination of the popular Res-Net101 architecture 806 and a Feature Pyramid Network (FPN) as shown in Reference 12. In short, ResNet-101 (Reference 23) is a convolutional neural network (CNN) with residual connections and is specifically designed to remove gradient vanishing at later layers during training. It is relatively simple with 345 layers. Another study attempted to add a Feature Pyramid Network to ResNet, where the motivation was to leverage the inherent pyramid shape of CNNs and to create a subsequent Feature Pyramid Network that uses top-down paths and lateral connections to combine low-resolution strong semantic features with high-resolution weak semantic features. Reference 12.

[0113] The resulting architecture is well-suited for learning from images of different scales based on a single, minimal input image scale. Ensuring scale-invariant learning is crucial for this disclosure because mosquito images can be generated at different scales during runtime due to variations in camera hardware and human-induced changes. The output of the first convolutional neural network 806 is a set of corresponding feature maps 808, which separate anatomical pixels of corresponding anatomical body parts of the object insect in image 802.

[0114] 3. Place anchor points on the anatomical components in the image. This step utilizes the concept of a Region Proposal Network (RPN) 816 as described in Reference 21, and the results of the previous two steps, to design a simpler CNN that will learn feature maps corresponding to ground-tested anatomical components in the training images. A final goal is to place anchor points (in a non-limiting example, these are bounding boxes 812) that enclose the detected anatomical components of interest in the image.

[0115] 4. Classification and pixel-level extraction. Finally, this disclosure aligns the feature maps of the anchor points (i.e., regions of interest) learned from the above steps to a fixed-size feature map. Alignment step 824 provides the fixed-size feature map as input to the three branches of architecture 800:

[0116] (a) Used as Figure 3 The anatomical components shown are marked with anchor points;

[0117] (b) Extract only the representation within the anchor point, such as Figures 5A to 5D The pixels of the anatomical components shown; and

[0118] (c) such as Figure 8 The anchor point is tightened as shown at reference 812 in the diagram to improve accuracy.

[0119] All three steps are performed in parallel.

[0120] 5. Loss function. Regarding the problems considered in this disclosure, a non-limiting scenario indicates that there are three specific sub-problems: labeling anchor points as chest, abdomen, wings, or legs; occluding the corresponding anatomical pixels within each anchor point; and tightening the anchor points using a regressor. The embodiments of this disclosure incorporate loss functions for these three sub-problems. Loss functions are a key component used during the training and validation of deep neural networks to improve learning accuracy and avoid overfitting.

[0121] 6 Indicate (or classify) the loss.To classify anchor points, a non-limiting embodiment of this disclosure utilizes the classification cross-entropy loss function, and it performs well. For a single anchor point j, the loss is given by the statement "p is the model-estimated probability of the anchor point's ground-based class".

[0122] 7. Concealing losses. Masking is a challenging task in image processing due to the complexity of detecting only pixels corresponding to anatomical components within anchor points during neural network learning. The unrestricted experiments used in this study employed a simple binary cross-entropy loss function. This loss function showed good accuracy in pixels corresponding to the chest, wings, and abdomen. However, many pixels corresponding to the legs were misclassified as background. This is attributed to… Figure 2 The class imbalance is highlighted in the image, where we see a significantly larger number of background pixels compared to the number of foreground pixels (colored in blue) placed around the anchor points (located around the legs). This imbalance leads to poor learning of the legs because the binary class entropy loss function is biased towards background pixels (which are more easily classified).

[0123] Another study utilized a recently developed loss function called focus loss (discussed at reference 24), which reduces the impact of well-classified samples on the loss, instead focusing more on harder samples. Therefore, this loss function prevents more prevalent background pixels from overwhelming less prevalent foreground pixels during learning, thus overcoming the class imbalance problem. The focus loss for pixel i is expressed as:

[0124] Here, p is the model-estimated probability of the ground truth class, and gamma-γ is an adjustable parameter, optionally set to 2 in an example model. From these definitions, it's easy to see that when a pixel is misclassified and p → 0, the adjustment factor (1 − p)γ tends to 1, and the loss (log(p)) remains unaffected. However, when a pixel is correctly classified and p → 1, the loss is down-weighted. In this way, the priority during training is placed more on the difficult negative classifications, thus producing excellent classification performance even with an imbalanced dataset. Utilizing focus loss yields excellent classification results for all anatomical components.

[0125] 8. Regressor loss. To tighten the anchor points and thus improve masking accuracy, the loss function used in a non-restrictive example is based on the sum of the Smooth L1 functions calculated at the anchor points, the ground reality, and the predicted anchor points.

[0126] In one example, the Let(x, y) algorithm represents the predicted top-left corner coordinates of an anchor point. Let xa and x* represent the top-left corner coordinates of the anchor point generated by the RPN, and the manually generated ground reality, respectively. The annotations for the y-coordinate, width w, and height h of the anchor point are the same. The procedure may include first defining several terms, followed by presenting a loss function Lreg used in a non-limiting example architecture.

[0127] , ,

[0128] t x t y t w t h

[0129] smooth L1 = 0.5 x 2 If |x| < 1 and (3)

[0130] |x| - 0.5, otherwise

[0131] Lreg (t i , ) = ∑ i x,y,w,h smooth(L1) ( - t i ).

[0132] Hyperparameters. For convenience, Table 5 lists the values ​​of key hyperparameters in the final architecture.

[0133] Table 5. Values ​​of key hyperparameters in the architecture

[0134]

[0135] Therefore, this disclosure proposes a system for designing state-of-the-art artificial intelligence (AI) techniques, specifically based on mask-region-based convolutional neural networks, to extract anatomical components of mosquitoes from digital images and permanently archive them based on genus, species, and other taxonomic criteria. This disclosure demonstrates that the system and method of this disclosure have currently generated nearly 30,000 digital images of mosquitoes (captured via smartphones), labeled according to genus and species type. Once the anatomical structures of interest are extracted, this disclosure utilizes AI techniques to design models that can identify the genus and species type of mosquitoes. If the system is well-received by citizens and experts, and if users mobilize entomologists, there is an opportunity to scale up efforts globally to include more mosquito species and to improve our models over time.

[0136] One non-limiting claim of this disclosure is the ability to bypass human classification (currently through microscopic observation) and instead use digital cameras and the proposed technology for automatic classification of genera and species types. A second value claim is that the disclosed system, with its large-scale, labeled, citizen- and expert-generated images, can begin digitizing the anatomy of mosquitoes globally. This database could bring incalculable value to mosquito, and especially vector surveillance, training, and global information sharing.

[0137] Example - Extracting Anatomical Structures

[0138] The public approach for one example program is based on the concept of Mask R-CNN

[32] , which is achieved by adding to existing methods for discriminating bounding boxes (see below). Figure 4 The branches are parallel to predict the object mask to segment the pixels containing the various anatomical components of interest (i.e., pixel-by-pixel segmentation).

[0139] In this approach, several key steps need to be performed (each of which is challenging). First, one must train a model using a pre-trained convolutional neural network to generate proposals related to regions where objects may exist within an image. As discussed above, a non-restricted example of a convolutional neural network used in this paper is ResNet101, which is used as the backbone convolutional model. In a non-restricted example, the model is initialized using pre-trained weights from the MS COCO

[34] dataset. ResNet

[33] has a very deep network and introduces residual connections to obtain input from the previous layer to the next. Residual connections help to address the vanishing gradient problem. The MS COCO dataset is a large-scale object detection dataset. It contains 1.5 million object instances and 80 object categories. The next step is to design an object detector network that performs three tasks: classifying boxes according to their corresponding anatomical structures; shrinking the boxes; and generating masks for each anatomical component (i.e., pixel-wise segmentation). In constructing the architecture of the object detector network, this disclosure uses a pixel-wise sigmoid as an example, along with a binary cross-entropy loss function (to identify k anatomical components) and trains them rigorously.

[0140] It should be noted that generating the training dataset here is not easy, as a mask needs to be created for each anatomical component in our dataset. To begin training, the user first annotates 571 mosquito images using the VGG image annotator tool, which is itself a very tedious task. Figure 3 (Annotated images are shown in the figure). Of the 571 images, 404 are training images and 167 are validation images. These methods then iterate the model to optimize the weights / hyperparameters. This disclosure shows that the methods have optimized hyperparameters such as the base feature extractor model, learning rate, momentum, optimizer, number of steps per round, number of validation steps, and number of rounds, details of which are shown in Table 6 below.

[0141] Table 6

[0142]

[0143] result:

[0144] The metric used to measure the accuracy of the mask R-CNN algorithm is mAP (average precision across all classes). It is calculated by averaging the average precision across all classes at all IoU thresholds, and is 0.833. The IoU metric measures the intersection of the ratios of pixels belonging to the ground truth within the bounding box and the union of the ratios of predicted pixels within the box to the ground truth. In our design, the IoU threshold is set to 0.75.

[0145] Figure 6A , Figure 6B , Figure 6C and Figure 7A , Figure 7B , Figure 7C Example designs for smartphone applications 600 and 700 on Android and iOS are shown, which allow users to capture images 620 and 720 of mosquitoes, or select an image from the phone's local storage. The application categorizes the mosquitoes and also extracts anatomical structures of interest—thorax, wings, abdomen, and legs—to display in corresponding anatomical images 620, 720, 630, 730, 640, and 740. All generated images are archived in the cloud.

[0146] Example - Classification of genus and species types based on extracted anatomical structures

[0147] This disclosure explains the design of a neural network-based architecture for identifying mosquito species from the entire subject of an image. The network architecture is presented below. The results achieve nearly 80% accuracy in classifying various mosquito species (currently nine species) and nearly 99% accuracy in identifying the highly lethal mosquito, Aedes aegypti (which transmits Zika, dengue, chikungunya, and yellow fever), which is prevalent worldwide.

[0148] Table 3 Category Structure

[0149]

[0150] In another example, users are extending the above architecture to classify data based on anatomical structures as well. This is part of ongoing work, but the network architecture for each anatomical structure—chest, abdomen, wings, and legs—will be very similar to the architecture above.

[0151] Embodiments of this disclosure include non-limiting combinations of the work described above. Thus, in one example, a system 800 for identifying the genus and species of insects includes imaging devices 600, 700 configured to generate images of the insects. The imaging device may be a smartphone or other mobile computer device with camera functionality. A computer processor 1202 is connected to a memory 1204 that stores computer-implemented commands in software and receives images (i.e., the computer processor, memory, and imaging device can communicate data via a network or local connection). The software implements a computerized method for the corresponding image, which begins by applying a first convolutional neural network 806 to the corresponding image 802 to generate a feature map 808 relating to, for example, anatomical pixels 510, 520, 530, 540 in the corresponding image, which corresponds to body parts of the insect. By weighting filters in the first convolutional neural network, the system is configured to identify anatomical pixels as foreground pixels within the corresponding image and to identify remaining pixels as background pixels for removal during segmentation operations.

[0152] Next, the system uses a processor or other computer to apply anchor points 812, 817 to feature map 808, wherein the anchor points identify portions of corresponding layers of image data in the feature map containing corresponding anatomical pixels of corresponding body parts. In other words, the feature map can be a multi-dimensional image data layer, and the system can operate on a single image data layer or multiple sets of image data layers generated from the first convolutional neural network 806. The anchor points can take the form of any polygon, which defines the boundary of a desired set of anatomical pixels within an image, feature map, or image data layer. In a non-limiting example, the anchor points are generated by a computer processor and displayed on a graphics display unit as bounding boxes within or superimposed on the image. The software also includes an alignment function 814 to align the layer with anchor points 812, 817 with the original feature map.

[0153] The system uses these anchor points to generate a mask 818A, which segments corresponding anatomical pixels in a corresponding image data layer. Generating the mask may include applying a second convolutional neural network 818B to the corresponding layer, wherein the second convolutional neural network segments the anatomical pixels according to the corresponding body parts.

[0154] The mask allows the extraction of fully connected layers 822 from the corresponding layers of the first convolutional neural network 806, and the system is further configured to apply fully connected layers to a regressor network 826 and a classification network 824, wherein generating the mask for segmentation, applying the fully connected layers to the regressor network, and applying the fully connected layers to the classification network are parallel operations performed by software. In some non-limiting embodiments, generating the mask includes applying a second convolutional neural network to the corresponding layers, wherein the second convolutional neural network segments anatomical pixels according to the corresponding body parts.

[0155] In some non-limiting implementations, parallel operations occur simultaneously.

[0156] The regressor network 826 is a computer-implemented software program that calculates the error value regarding the iterative position of anchor points in the corresponding layer. The system uses the error value in a feedback loop 850 to tighten anchor points 812, 817 around anatomical pixels corresponding to the corresponding body parts. The regressor network and the associated computer-implemented software calculate the error value regarding the iterative position of anchor points in the corresponding layer, where the error value is derived from a binary cross-entropy loss function or a focus loss function.

[0157] Anchor points 812 and 817 can be bounding boxes or any other shape, originating from the region proposal network 816 that receives feature map 808 as the corresponding image data layer, and the feedback loop 850 transmits error values ​​from the regressor network 826 to the region proposal network 817 to tighten the boxes onto the appropriate pixels corresponding to the respective body parts. A region proposal network is an image processing software implementation that uses data from the feature map to predict possible image and image layer portions containing anatomical pixels corresponding to insect body parts.

[0158] The classification network 824 is a computer-implemented software tool used to generate classification output images 828, and in some embodiments, these classification output images include updated versions of the original images, wherein the updated versions have boundary polygons 812, 817, labeled with anatomical component names 1125, 1135, 1145, 1155, and even color coding as shown in Table 4 that may aid in genus and species identification. Example embodiments described herein show that the system can identify insects such as mosquitoes and anatomical component names, including wings, legs, thorax, and abdomen corresponding to the respective body parts. In some non-limiting embodiments, the classification network utilizes a pixel-wise sigmoid network. In non-limiting uses, the system populates a database storing test outputs of the classification network, wherein the output includes image versions labeled with anatomical component names, and wherein the database stores corresponding genus and species information and corresponding data associated with the respective genus and species.

[0159] The system implementation can be carried out using at least one computer that performs a computerized method for extracting information relating to anatomical components of a living organism from images. Images may include digital images of insects or other animals or even inanimate objects, wherein the digital images include views of the corresponding insect, animal, or inanimate object from directly above the sample and from an angle lateral to the background holding the corresponding sample. The method begins by identifying ground-based anatomical components with a set of accuracy by training a mask-based convolutional neural network using a set of training images (segmented using a computerized algorithm). Training the convolutional neural network used in this disclosure typically involves classifying the corresponding anatomical components in the training images and comparing the training images with ground-based images. By tightening the bounding boxes surrounding the anatomical components in the digital images, the method learns how to maximize the efficiency and accuracy of ultimately generating masks for extracting information from a second set of images (such as previously created feature maps). To form ground-based images, the computerized algorithm may utilize an image annotator tool configured for manual manipulation. Training iteratively updates hyperparameters for the anatomical pixels in the training dataset. This method has a reliable track record in tracking, identifying, and archiving genus-species identification data for multiple species across multiple genera of insects.

[0160] In the example implementation, at least some parts of the activity may be implemented in software provided on the network device. In some embodiments, one or more of these features may be implemented in computer hardware 1200, provided externally to these elements, or incorporated in any suitable manner to achieve the intended functionality. Various network elements may include: software (or reciprocating software) that coordinates image development across domains such as time, amplitude, and depth; and various classification measures that detect movement across image data frames and further detect specific objects in the field of view to achieve the operations outlined herein. In still other embodiments, these elements may include any suitable algorithm, hardware, software, component, module, interface, or object that facilitates its operation.

[0161] Additionally, the computer systems (and / or their associated structures) described and illustrated herein may also include suitable interfaces for receiving, transmitting, and / or otherwise conveying data or information in a networked environment. Furthermore, some of the processors 1202 and memory elements 1204 associated with various nodes may be removed or otherwise combined, such that a single processor and a single memory element handle certain activities. In a general sense, the arrangement shown in the figures may be more logically consistent in its representation, while the physical architecture may include various permutations, combinations, and / or mixtures of these elements. It must be noted that countless possible design configurations can be used to achieve the operational objectives outlined herein. Therefore, the associated infrastructure has countless alternative arrangements, design choices, device possibilities, hardware configurations, software implementations, equipment options, etc.

[0162] In some example implementations, one or more memory elements (e.g., memory) may store data for the operations described herein. This includes the memory's ability to store instructions (e.g., software, logic, code, etc.) in a non-transitory medium, causing the instructions to be executed to perform the activities described herein. A processor may execute any type of computer-readable instructions associated with data to perform the operations detailed herein. In one example, a processor (e.g., a processor) may transform an element or item (e.g., data) from one state or thing to another. In another example, the activities outlined herein may be implemented with fixed logic or programmable logic (e.g., software / computer instructions executed by a processor), and the elements identified herein may be some type of programmable processor, programmable digital logic (e.g., field-programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), ASIC including digital logic, software, code, electronic instructions, flash memory, optical disc, CD-ROM, DVD ROM, magnetic card or optical card, other types of machine-readable media suitable for storing electronic instructions, or any suitable combination thereof.

[0163] These devices can also store information in any suitable type of non-transitory storage medium 1212 (e.g., random access memory (RAM), read-only memory (ROM), field-programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable ROM (EEPROM), etc.), software, hardware, or any other suitable component, device, element, or object, when appropriate and based on specific needs. Any memory item discussed herein should be construed as included within the broad term 'memory element'. Similarly, any potential processing elements, modules, and machines described in this specification should be construed as included within the broad term 'processor'. Figure 12See the illustrative example shown, which illustrates a computing environment for an input device 1208A (such as the imaging device described above) and an output device 1208B (such as the smartphones of Figures 6 and 7). This computing environment is suitable for various network and cloud connections, as shown at reference 1206.

[0164] It must also be noted that, unless the context clearly indicates otherwise, the singular forms “a / an” and “described” as used in this specification and the appended claims include a plurality of references. The terms “comprising,” “containing,” or “including” mean that at least the specified compounds, elements, particles, or method steps are present in the composition, article, or method, but do not exclude the presence of other compounds, materials, particles, or method steps, even if such other compounds, materials, particles, or method steps have the same function as those named.

[0165] A range may be expressed herein as ranging from “about” or “approximately” one specific value to “about” or “approximately” another specific value. When expressing such a range, exemplary embodiments include ranging from one specific value to another. As used herein, “about” or “approximately” generally means within 20%, preferably within 10%, and more preferably within 5% of a given value or range, and may also include exact values ​​or ranges. The numerical quantities given herein may be approximate, meaning that the terms “about” or “approximately” can be inferred unless explicitly stated otherwise.

[0166] In describing example embodiments, terminology is used for clarity. Each term is intended to have the broadest meaning as understood by one of ordinary skill in the art, and includes all technical equivalents that operate in a similar manner to achieve a similar purpose. It should also be understood that references to one or more steps of a method do not exclude the existence of additional method steps or intermediate method steps between those explicitly identified. The steps of a method may be performed in a different order than those described herein without departing from the scope of this disclosure. Similarly, it should be understood that references to one or more components in an apparatus or system do not exclude the existence of additional components or intermediate components between those explicitly identified.

[0167] References

[0168] 1. Minakshi, M., Bharti, P. and Chellappan, S. Leveraging smartphone cameras and image processing techniques to classify mosquito species. In Proceedings of the 15th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, 77–86 (ACM, 2018).

[0169] 2. De Los Reyes, A. M. M., Reyes, A. C. A., Torres, J. L., Padilla, D. A. and Villaverde, J. Detection of aedes aegypti mosquito by digital image processing techniques and support vector machine. In 2016 IEEE Region 10 Conference (TENCON), 2342–2345 (IEEE, 2016).

[0170] 3. Fuchida, M., Pathmakumar, T., Mohan, R. E., Tan, N. and Nakamura, A. Vision-based perception and classification of mosquitoes using support vector machine. Appl. Sci. 7, 51 (2017).

[0171] 4. Favret, C. and Sieracki, J. M. Machine vision automated species identification scaled towards production levels. Syst. Entomol. 41, 133–143 (2016).

[0172] 5. Minakshi, M. et al., Automating the surveillance of mosquito vectors from trapped specimens using computer vision techniques. published in ACM COMPASS (ACM, 2020).

[0173] 6. Park, J., Kim, D. I., Choi, B., Kang, W. and Kwon, H. W. Classification and morphological analysis of vector mosquitoes using deep convolutional neural networks. Sci. Rep. 10, 1–12 (2020).

[0174] 7. Chen, Y., Why, A., Batista, G., Mafra-Neto, A. and Keogh, E. Flying insect detection and classification with inexpensive sensors. J. Vis. Exp. (JoVE) e52111 (2014).

[0175] 8. Mukundarajan, H., Hol, F. J., Castillo, E. A., Newby, C. and Prakash, M. Using mobile phones as acoustic sensors for the surveillance of spatio-temporal mosquito ecology (2016).

[0176] 9. Vasconcelos, D., Nunes, N., Ribeiro, M., Prandi, C. and Rogers, A. Locomobis: a low-cost acoustic-based sensing system to monitor and classify mosquitoes. In 2019 16th IEEE Annual Consumer Communications & Networking Conference (CCNC), 1–6 (IEEE, 2019).

[0177] 10. Ravi, P., Syam, U. and Kapre, N. Preventive detection of mosquito populations using embedded machine learning on low power iot platforms. In Proceedings of the 7th Annual Symposium on Computing for Development, 1–10 (2016).

[0178] 11. He, K., Gkioxari, G., Dollár, P. and Girshick, R. Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision 2961–2969 (2017).

[0179] 12. Lin, T.-Y. et al., Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2117–2125 (2017).

[0180] 13. Smithsonian homepage. https: / / www.si.edu / . The importance of learning identification of larvae and adult mosquitoes. https: / / juniperpublishers.com / jojnhc / pdf / JOJNHC.MS.ID.555636.pdf.

[0181] 14. Stauffer, C. and Grimson, W. E. L. Adaptive background mixture models for real-time tracking. In Computer Vision and Pattern Recognition, 1999. IEEE Computer Society Conference on., Volume 2, 246–252 (IEEE, 1999).

[0182] 15. Ronneberger, O., Fischer, P. and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 234–241 (Springer, 2015).

[0183] 16. Dai, J., He, K. and Sun, J. Instance-aware semantic segmentation via multi-task network cascades. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3150–3158 (2016).

[0184] 17. Li, Y., Qi, H., Dai, J., Ji, X. and Wei, Y. Fully convolutional instance-aware semantic segmentation. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2359–2367 (2017).

[0185] 18. Pinheiro, P. O., Collobert, R. and Dollár, P. Learning to segment object candidates. in Advances in Neural Information Processing Systems 1990–1998 (2015).

[0186] 19. Girshick, R. Fast R-CNN. in Proceedings of the IEEE International Conference on Computer Vision 1440–1448 (2015).

[0187] 20. Ren, S., He, K., Girshick, R. and Sun, J. Faster R-CNN: towards real-time object detection with region proposal networks. in Advances in Neural Information Processing Systems 91–99 (2015).

[0188] 21. Dutta, A. and Zisserman, A. The VGG image annotator (via). arXiv preprint arXiv:1904.10699 (2019).

[0189] 22. He, K., Zhang, X., Ren, S. and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778 (2016).

[0190] 23. Lin, T.-Y., Goyal, P., Girshick, R., He, K. and Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, 2980–2988 (2017).

[0191] 24. Mask RCNN code. https: / / github.com / matterport / Mask_RCNN.

[0192] 25. IFAS. Florida Medical Entomology Laboratory. https: / / fmel.ifas.ufl.edu / mosquito-guide / species-identification-table / species-identification-table-adult / .

[0193] 26. Glyshaw, P. and Wason, E. Anopheles quadrimaculatus. https: / / animaldiversity.org / accounts / Anopheles_quadrimaculatus / edu / mosquito-guide / species-identification-table / species-identification-table-adult / .

[0194] 27. Dharmasiri, A. G. et al., First record of anopheles stephensi in Sri Lanka: a potential challenge for prevention of malaria reintroduction. Malaria J. 16, 326 (2017).

[0195] 28. IFAS. Florida medical entomology laboratory. https: / / fmel.ifas.ufl.edu / publication / buzz-words / buzz-words-archive / is-it-culex-tarsalis-or-culex-coronator /

[0196] 29. Floore, T. A., Harrison, B. A. and Eldridge, B. F. The anopheles (anopheles) crucians subgroup in the united states (diptera: Culicidae) (Tech. Rep, Walter Reed Army Inst Of Research Washington Dc Department Of Entomology, 1976)

[0197] 30. Smithsonian Institution. https: / / collections.si.edu / search / results.htm?fq=tax_kingdom%3A%22animalia%22&fq=online_media_type%3A%22Images%22&fq=datasource%3A%22NMNH+-+Entomology+Dept.%22&q=NMNH-USNMENT01001576&gfq=CSILP_6.

[0198] 31. Smithsonian ttps: / / collections.si.edu / search / results.htm?fq=tax_kingdom%3A%22 Animedia_type%3A%22Images%22&fq=data_source%3A%22NMNH+Entomology +Dept. %22&q = NMNH- ENT01006317&gfq=CSILP_6.

[0199] 32. He, Kaiming, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. "Maskr-cnn." From Proceedings of the IEEE international conference on computervision, pp. 2961-2969, 2017.

[0200] 33. He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." From Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.

[0201] 34. Lin, Tsung-Yi, Michael Maire, Serge Belongie, James Hays, PietroPerona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. "Microsoft coco: Common objects in context." From the European conference on computer vision, pp. 740–755, Springer, Cham, 2014.

[0202] 35. Dr. Stewart, Matthew, *Simple Introduction to Convolutional Neural Networks*, from Towards Data Science (link provided). https: / / towardsdatascience.com / simple- introduction-to-convolutional-neural-networks-cdf8d3077bac.

Claims

1. An apparatus for capturing insects and collecting identification data about said insects, said apparatus comprising: A membrane body connected to a positioning disk, the membrane body defining an inlet opening for guiding an insect attractant into the membrane body; A microphone located in the device, the microphone being configured to receive an audio signal; At least one imaging device having a field of view covering the membrane body; A computer connected to the at least one imaging device, wherein the computer includes a processor connected to a computerized memory storing software thereon, the software including computer-implemented instructions that, when executed by the processor, cause the processor to: Receive audio signals from the microphone; Calculate the frequency signal corresponding to the wingbeat frequency of the insect; In response to determining that the wingbeat frequency is within a predetermined frequency range, the at least one imaging device is triggered to obtain multiple images of the insect; Extract at least one anatomical component of the insect from the plurality of images; and The data about the insect is stored in the computerized memory; as well as At least a portion of the stored data is transferred to the cloud server.

2. The device of claim 1, further comprising a membrane housing surrounding the membrane body to trap the insect within the membrane housing.

3. The device of claim 2, further comprising a closed chamber connected to the membrane housing via an outlet opening and configured to contain the insect after the data has been stored.

4. The device of claim 2, wherein the membrane body comprises a skin surface configured to support an insect thereon.

5. The device of claim 4, wherein the skin surface is a lambskin protective film.

6. The device of claim 1, wherein the positioning disk is connected to a carbon dioxide source that enters the membrane body through the inlet opening.

7. The device of claim 1, further comprising a heater connected to the positioning disk, the heater being configured to heat the insect attractant entering the membrane body.

8. The device of claim 1, wherein the positioning disc defines a recess for placing additional insect attractant.

9. The device of claim 1, wherein the at least one imaging device is a passive infrared sensor (PIR).

10. The device of claim 1, wherein the at least one imaging device is a camera.

11. The device of claim 1, wherein the device is configured as a mosquito trap.

12. The device of claim 11, wherein the at least one imaging device is a camera array having a corresponding field of view covering the membrane body.

13. The apparatus of claim 11, further comprising an ultraviolet light source oriented toward the membrane body to illuminate at least one fluorescent marker on the mosquito.

14. The device of claim 2, further comprising a plurality of enclosed chambers connected to the membrane housing via respective outlet openings and configured to contain the mosquitoes in the respective enclosed chambers according to mosquito classification criteria.

15. The apparatus of claim 11, further comprising an ambient environment sensor that collects environmental data relating to the environment of the trap and transmits the environmental data to the computer.

16. The device of claim 1, wherein the software uses at least one neural network algorithm to classify the insects, the at least one neural network algorithm being applied to image data from the imaging device.

17. The device of claim 2, further comprising a funnel-shaped object connected to the membrane housing to guide the insect into the device.

Images