Information processing device, information processing method, and program

Abstract

To provide an information processor, an information processing method, and a program that can generate detection data that detects a state of a plant body more easily.SOLUTION: An information processor comprising an image acquisition part that acquires a two-dimensional image including a plant body taken from above by a flying body, an image division part that divides a region of the image into a plurality of divided images of a predetermined size, a position acquisition part that acquires a geographical position of each of the divided images based on geographic information representing a center location of the image included in the image, a plant body detection part that detects a state of a predetermined type of plant body in each of the divided images, and a detection data generation part that generates detection data that associates the geographical position of the acquired divided images with the detected state of the predetermined type of plant body.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In a test field or a production site for cultivating plants (crops), measuring the density of plants (weeds), that is, detecting weeds, is useful for field management such as pesticide spraying and grassland renewal. Here, the plants for which the density is measured (detected) may include plants that are not cultivated, such as weeds, that is, plants such as cultivated crops. In field management, for example, it is important to visualize where weeds are growing densely by representing the measured (detected) weed density on a map. In this regard, various methods for weed detection using aerial images and machine learning have been proposed conventionally.

[0003] For example, Non-Patent Document 1 describes a method for detecting weeds from aerial images taken by a small unmanned aerial vehicle (UAV). In the weed detection method described in Non-Patent Document 1, three-dimensional data called an orthomosaic image that integrally represents the entire field is generated from a series of aerial images, and the generated orthomosaic image is analyzed by machine learning to detect weeds on the orthomosaic image. Thereby, in Non-Patent Document 1, the position of the detected weeds can be specified by displaying the position of the weeds on the orthomosaic image.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

[0005] However, generating orthomosaic images requires applying a technique such as Structure from Motion (SfM) to aerial images. SfM is a method for constructing a three-dimensional model from multi-view images, and although its use is being considered in the biomass field, it requires a lot of processing time. In other words, SfM is a technique with high processing costs.

[0006] Furthermore, the series of aerial images used to generate an orthomosaic image by SfM require a certain amount of overlap between each image. For example, the surveying manual published by the Geospatial Information Authority of Japan recommends an overlap of 80% or more in the flight direction of the unmanned aerial vehicle (UAV) and 60% or more between flight paths (courses). Therefore, generating an orthomosaic image requires taking many aerial images, meaning a large number of images must be taken, which increases the time required for taking the aerial images.

[0007] Furthermore, orthomosaic images can become distorted if external factors, such as wind blowing during aerial photography, cause the leaves of the weeds being photographed to sway. The likelihood of external factors causing distortion in orthomosaic images increases with the length of time required for aerial photography. Consequently, when detecting weeds using machine learning, for example, a machine learning model trained on the shape of weed leaves, the accuracy of weed detection will decrease in distorted orthomosaic images. For this reason, it can be difficult to detect weeds using orthomosaic images with machine learning.

[0008] This invention was made based on the above-mentioned recognition of problems, and aims to provide an information processing device, an information processing method, and a program that can generate detection data that more easily detects the state of a plant. [Means for solving the problem]

[0009] The information processing apparatus, information processing method, and program according to this invention employ the following configuration. An information processing device according to one aspect of the present invention is an information processing device comprising: an image acquisition unit that acquires a two-dimensional image including a plant body, taken from above by a flying object; an image division unit that divides the region of the image into a plurality of divided images of a predetermined size; a position acquisition unit that acquires the geographical position of each of the divided images based on geographical information representing the central position of the image contained in the image; a plant body detection unit that detects the state of a predetermined type of plant body depicted in each of the divided images; and a detection data generation unit that generates detection data relating the geographical position of the acquired divided images and the detected state of the predetermined type of plant body.

[0010] The position acquisition unit uses the center position of the image represented by the geographic information as a reference and acquires the geographic position of the segmented image, which is determined based on the direction of movement of the flying object when the image was taken, the height of the flying object from the ground when the image was taken, and the shooting range of the image.

[0011] The plant detection unit obtains and detects the state of a predetermined type of plant by inputting the segmented images to a plant training model that has been trained to output the state of a predetermined type of plant when the segmented images are input.

[0012] The state of the predetermined type of plant body is the coverage of the plant body within the segmented image.

[0013] The detection data generation unit generates detection data for each of the divided images, representing the degree of coverage of the predetermined type of plant.

[0014] The detection data generation unit generates a detection image representing the detection data in image format, and overlays the generated detection image onto a map image for display on a display device.

[0015] Another aspect of the present invention relates to an information processing method in which a computer acquires a two-dimensional image including a plant, taken from above by a flying object, divides the region of the image into a plurality of segmented images of a predetermined size, acquires the geographical location of each segmented image based on geographical information representing the central position of the image contained in the image, detects the state of a predetermined type of plant depicted in each segmented image, and generates detection data that associates the acquired geographical location of the segmented image with the detected state of the predetermined type of plant.

[0016] Another aspect of the present invention involves a program that causes a computer to acquire a two-dimensional image including a plant, taken from above by a flying object; to divide the region of the image into a plurality of segmented images of a predetermined size; to acquire the geographical location of each segmented image based on geographical information representing the central position of the image contained in the image; to detect the state of a predetermined type of plant depicted in each segmented image; and to generate detection data that associates the acquired geographical location of the segmented image with the detected state of the predetermined type of plant. [Effects of the Invention]

[0017] According to the present invention, a plant body can be detected more easily. [Brief Description of the Drawings]

[0018] [Figure 1] The figure shows an example of the configuration of an information processing apparatus according to an embodiment and an example of the usage environment of the information processing apparatus. [Figure 2] The figure schematically shows an example of a method for generating a plant body learned model. [Figure 3] The figure shows an example of detection data representing a detection result detected by a plant body detection unit. [Figure 4] The figure shows an example of a screen on which detection data generated by a detection data generation unit is displayed. [Figure 5] [[ID=二十六]]The figure is a flowchart showing an example of a process for detecting the state of a plant body in an information processing apparatus. [Figure 6] The figure schematically shows an example of a process for detecting the state of a plant body in an information processing apparatus. [Figure 7] The figure shows a comparative example of a detection result of the state of a plant body detected by an information processing apparatus. [Figure 8] The figure shows an example of detection data representing detection results in which a plant body detection unit provided in an information processing apparatus detects the states of different plant bodies. [Embodiments for Carrying Out the Invention]

[0019] Hereinafter, embodiments of an information processing apparatus, an information processing method, and a program of the present invention will be described with reference to the drawings.

[0020] [Configuration of Information Processing Apparatus] Figure 1 shows an example of the configuration of an information processing device according to an embodiment, and an example of the usage environment of the information processing device. The information processing device 100 detects the state of a predetermined type of plant body captured in a two-dimensional aerial image including plant bodies, based on a two-dimensional aerial image taken from above (from above), and generates detection data representing the detected state of the plant body. The aerial image is, for example, taken from above by a flying object such as a drone or a small unmanned aerial vehicle (UAV), including the area where the plant body whose state is to be detected is growing (hereinafter referred to as the "detection target area"). The plant body includes various types of plant bodies, such as crops and pasture grasses cultivated in fields and production sites, and weeds living in or around fields and production sites. The state of the plant body is, for example, the ratio of the size of the area occupied by a predetermined type of plant body captured in the aerial image, so-called plant body cover. The state of the plant body may be, for example, the growth state of a predetermined type of plant body captured in the aerial image. The growth status includes various conditions that are checked when cultivating plants, such as the degree of growth of the plants being grown (especially crops and pasture grasses) and whether or not the plants are affected by disease. The detection data is two-dimensional data, also known as raster data, in which information representing the status of the detected plants is placed within an area the same size as the aerial image.

[0021] Figure 1 shows an example where the flying object FO is a drone. In Figure 1, the detection area is the area centered on field F, and the flying object FO is shown photographing the detection area, including plants, from above field F. In the following explanation, the plant whose state is to be detected is assumed to be a weed, and the state of the plant is assumed to be the weed cover.

[0022] The flying object FO is controlled by, for example, a terminal device T1. Alternatively, the flying object FO may fly autonomously according to a program stored in its built-in memory. The flying object FO is equipped with an imaging device C, such as a digital camera. The imaging device C takes aerial images of the detection target area, including plants (in this case, weeds), from above at regular intervals or every time the flying object FO flies a certain distance. The imaging device C may take aerial images of the detection target area, including plants (weeds), from above in response to an operation from the terminal device T1. The imaging device C transmits the captured aerial images to the terminal device T1. Furthermore, the flying object FO is also equipped with a positioning device (not shown) that determines the position of the flying object FO itself based on signals received from satellites constituting a GNSS (Global Navigation Satellite System), such as GPS (Global Positioning System). The imaging device C associates the captured aerial image, information related to this aerial image (hereinafter referred to as "image information"), and information representing the position of the flying object FO (hereinafter referred to as "GPS information"), and transmits it to the terminal device T1. The image information includes at least information representing the focal length when the aerial image was taken. The focal length information included in the image information also represents the field of view for imaging device C. An example of image information is metadata recorded in Exif (Exchangeable image file format). The GPS information includes, for example, information such as the latitude, longitude, and altitude of the flying object FO, and information representing the direction of movement of the flying object FO, that is, the direction in which the flying object FO is flying. The GPS information also represents the geographical location such as the latitude, longitude, and altitude of the center of the captured aerial image, and the direction in which the aerial image was taken. If the flying object FO is equipped with an altimeter that measures altitude based on atmospheric pressure, for example, the imaging device C may transmit to the terminal device T1, in association with the aerial image, the altitude information of the flying object FO measured by the altimeter, in place of or in addition to the altitude information included in the GPS information. GPS information is an example of "geographic information" in the claims, and focal length information included in the image information is an example of "shooting range" in the claims.

[0023] Terminal device T1 controls, for example, the flight of the flying object FO and the capture of aerial images by the imaging device C. Terminal device T1 is a computer device such as a personal computer, smartphone, or tablet terminal. Applications for controlling the flight of the flying object FO and the imaging device C are executed on terminal device T1. In response to operations by user P1 using the information processing device 100, the application causes terminal device T1 to transmit information for controlling the flight of the flying object FO and the imaging device C. The application displays images including operation buttons for controlling the flight of the flying object FO and the imaging device C, and images showing the current shooting range transmitted by the imaging device C, on the display device provided by terminal device T1. When user P1 operates terminal device T1 to capture the detection target area with the imaging device C, they may move the flying object FO and take multiple shots so that the entire range of the detection target area is captured. The application may cause terminal device T1 to transmit information to the flying object FO and the imaging device C in order to control the flying object FO to fly within the detection target area along a pre-set flight path and to take images with the imaging device C at predetermined intervals. The application may also cause terminal device T1 to transmit the aerial images transmitted by the imaging device C of the flying object FO to the information processing device 100.

[0024] Terminal device T1 transmits aerial images to information processing device 100, for example, via a network NW. The network NW includes, for example, the Internet, WAN (Wide Area Network), LAN (Local Area Network), provider equipment, and wireless base stations. Terminal device T1 may, for example, temporarily store the aerial images transmitted by imaging device C in a portable memory, and then transfer the aerial images to information processing device 100 by attaching the portable memory to the information processing device 100.

[0025] In the following description, it is assumed that the flying object FO flies in a straight line at a predetermined altitude in a predetermined direction (azimuth). That is, the flying object FO flies at a constant yaw angle and flies in a straight line at the same altitude without changing direction except when it reaches the edge of the detection target area. The imaging device C takes an aerial image directly below the flying object FO. At this time, the imaging device C does not capture the entire range of the detection target area in a single aerial image, but rather captures the entire range of the detection target area in a series of aerial images taken multiple times. Furthermore, even if the direction of flight changes because, for example, the flying object FO reaches the edge of the detection target area, the upper side of the captured aerial image will remain in a constant direction (azimuth). That is, the upper side of the series of aerial images captured by the imaging device C will be fixed in terms of east, west, north, or south. Therefore, if we consider the initial direction (azimuth) of the flying object FO to be forward in order to photograph the detection target area with the imaging device C, when the flying object FO changes direction (turns around) upon reaching the edge of the detection target area, it may, for example, fly backward. For example, if the imaging device C has a structure that allows it to rotate the shooting direction, if the flying object FO does not fly backward but instead turns 180°, the imaging device C itself may rotate 180° so that the shooting direction does not change. For example, if the imaging device C has a function to rotate the captured aerial image, it may rotate the captured aerial image by 180° before transmitting it to the terminal device T1. For example, the imaging device C may add information indicating that the direction of the captured aerial image is 180° different to the image information before transmitting it to the terminal device T1.

[0026] The information processing device 100 includes, for example, a communication unit 110, an image acquisition unit 120, a storage unit 130, an image division unit 140, a position acquisition unit 150, a plant detection unit 160, and a detection data generation unit 170. The information processing device 100 may also communicate with a terminal device T2, which is separate from terminal device T1, via a network NW.

[0027] Terminal device T2 is a terminal device operated by, for example, a user P2 using the information processing device 100, for the purpose of checking the detection results of the information processing device 100, which is connected via a network NW. Like terminal device T1, terminal device T2 is a computer device such as a personal computer, smartphone, or tablet terminal. Terminal device T2 runs applications for controlling detection in the information processing device 100 and checking the detection results. The applications running on terminal device T2 control the setting of detection conditions when the information processing device 100 performs detection and the display of the detection results of the information processing device 100, in response to operations by user P2. The applications running on terminal device T2 display images including operation buttons for controlling the setting of detection conditions and images showing the detection results of the information processing device 100 on the display device provided by terminal device T2. The detection conditions include various conditions for detecting plants, such as specifying the type of plant to be detected and setting the detection range (i.e., the detection target area). A display device provided in either or both of terminal device T1 and terminal device T2, or in either or both of terminal device T1 and terminal device T2, is an example of a "display device" as defined in the claims.

[0028] Of the components of the information processing device 100 described above, all except the communication unit 110 and the storage unit 130 are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of the functions of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), or by the cooperation of software and hardware. Some or all of the functions of these components may be realized by a dedicated LSI. The program may be stored in advance in a storage device (a storage device with a non-transient storage medium) such as an HDD (Hard Disk Drive) or flash memory provided by the information processing device 100, or it may be stored in a removable storage medium (a non-transient storage medium) such as a DVD or CD-ROM, and installed in the storage device of the information processing device 100 when the storage medium is mounted in a drive device provided by the information processing device 100. The information processing device 100 may be implemented in a computer device or storage device, such as a personal computer. The information processing device 100 may also be implemented in a server device or storage device incorporated into a cloud computing system. In this case, the functions of the information processing device 100 may be implemented by multiple server devices and storage devices in the cloud computing system.

[0029] The communication unit 110 is a communication interface such as a network card for connecting to the network NW. The communication unit 110 communicates with the terminal device T1 via the network NW.

[0030] The image acquisition unit 120 acquires aerial images taken by the imaging device C mounted on the flying object FO from a terminal device T1 that communicates with the communication unit 110. The image acquisition unit 120 acquires at least one (1 image) of aerial images taken by the imaging device C via the communication unit 110 and the terminal device T1. The image acquisition unit 120 may acquire each aerial image taken by the imaging device C sequentially, or it may acquire all aerial images at once. The image acquisition unit 120 stores the acquired aerial images in the storage unit 130. At this time, the image acquisition unit 120 assigns identification information (hereinafter referred to as "aerial image ID") to each acquired aerial image and stores it in the storage unit 130. The aerial image ID is exclusive identification information for each aerial image. The aerial image ID is, for example, a number that has been updated (incremented) in the order in which the image acquisition unit 120 acquires the aerial images and stores them in the storage unit 130. If the aerial image includes information about the date and time of capture as image information, the image acquisition unit 120 may use the date and time of capture information as the aerial image ID, either in place of or in addition to the incremented number. The storage of the acquired aerial image in the storage unit 130 by the image acquisition unit 120 may be performed by DMA (Direct Memory Access).

[0031] The storage unit 130 stores data and information used by the components of the information processing device 100 when they perform processing. The storage unit 130 is, for example, a storage device such as an HDD or flash memory.

[0032] The image splitting unit 140 reads at least one (one image) of aerial photography acquired by the image acquisition unit 120 and stored in the storage unit 130 from the storage unit 130, and divides the aerial photography image into multiple split images by cutting out multiple regions of a predetermined size from the read aerial photography image. The size of each split image divided by the image splitting unit 140 is predetermined based on the unit (hereinafter referred to as "processing unit") used by the plant detection unit 160, which will be described later, to perform weed detection processing, and the resolution and number of pixels in the left-right and up-down directions of the split image are predetermined. The size of the processing unit in the plant detection unit 160, which will be described later, varies depending on the altitude of the flying object FO from which the aerial photography image was taken. For this reason, the image splitting unit 140 divides the region of the aerial photography image into split images of the same size (resolution and number of pixels) as the processing unit in the plant detection unit 160, which will be described later, based on the altitude of the flying object FO represented by the GPS information associated with the aerial photography image. For example, if the processing unit in the plant detection unit 160, described later, has 128 pixels in both the horizontal and vertical directions, the image division unit 140 divides the aerial image into multiple divided images, each containing a 128x128 pixel region. The image division unit 140 moves the region to be cut out as a divided image from the aerial image (hereinafter referred to as the "divided image region") sequentially from the upper left to the right (horizontally) of the aerial image. When the divided image region reaches the upper right of the aerial image, it returns to a point below the first divided image region on the left side of the aerial image where it does not overlap, and moves it to the right again, thereby cutting out each divided image region from the entire range of the aerial image. In other words, the image division unit 140 divides the entire range of the aerial image into each divided image by cutting out the divided image region sequentially from the upper left to the lower right of the aerial image, in what is known as raster order. The image division unit 140 may divide the entire range of the aerial photograph into separate divided images by sequentially moving the divided image region so that it expands outwards from the center of the aerial photograph. The range in which the image division unit 140 divides the aerial photograph into divided images is not limited to the entire range of the aerial photograph.For example, if the detection range (detection target area) for weed detection is set as a detection condition by an application running on terminal device T2, the image division unit 140 may move the divided image area to include the set range and divide it into separate divided images.

[0033] The image splitting unit 140 stores each of the divided images in the storage unit 130. At this time, the image splitting unit 140 assigns identification information (hereinafter referred to as "divided image ID") to each divided image to identify that divided image, and further associates the image information, GPS information, and aerial image ID that were associated with (assigned to) the original aerial image from which the divided image was split (hereinafter also referred to as "original aerial image") with information indicating the position in the original aerial image from which the divided image was split (hereinafter referred to as "splitting position information"), and stores this information in the storage unit 130. The divided image ID is exclusive identification information for each divided image. The divided image ID is, for example, a number that combines the aerial image ID and a number corresponding to the position from which the image was split from the original aerial image. The splitting position information is, for example, the coordinates that represent the center position of the divided image when the center position represented by the GPS information associated with the original aerial image from which the divided image was split is used as the reference coordinate position. The reading of the aerial image (original aerial image) from the storage unit 130 in the image division unit 140, and the storage of the divided images in the storage unit 130, may be performed by DMA.

[0034] The position acquisition unit 150 reads each divided image stored in the storage unit 130 after being divided by the image division unit 140, and acquires the geographical location of each read divided image. The geographical location of a divided image can be determined, for example, based on the center position of the original aerial photograph of the divided image, the direction of movement of the flying object FO when the original aerial photograph was taken, the altitude of the flying object FO when the original aerial photograph was taken, and the shooting range of the original aerial photograph. The center position of the original aerial photograph, the direction of movement of the flying object FO, and the altitude are indicated in the GPS information associated with the divided image. The shooting range of the original aerial photograph is indicated in the image information associated with the divided image. Therefore, the position acquisition unit 150 can determine the geographical location of a divided image based on the respective information associated with the divided image. For example, if the imaging device C takes a photograph such that the front of the flying object FO is on the upper side of the aerial photograph as it flies, the position acquisition unit 150 can determine which direction the upper horizontal side in the original aerial photograph is based on the direction of movement of the flying object FO. The position acquisition unit 150 can determine the horizontal and vertical ranges [m] of the area captured in the original aerial photograph, based on the shooting range of the original aerial photograph indicated in the image information and the altitude indicated in the GPS information. In other words, the position acquisition unit 150 can determine the horizontal and vertical distances of the area captured in the original aerial photograph. The position acquisition unit 150 then uses the center position of the original aerial photograph, as represented by the GPS information, as the reference coordinate position, and from the difference (i.e., the difference in coordinates) between the reference coordinate position and the coordinates of the center position of the divided image, as represented by the divided position information, it can determine the latitude and longitude of the center position of the divided image. From the difference between the shooting range of the original aerial photograph (which may be the resolution or number of pixels) and the size of the divided image (which may be the resolution or number of pixels), it can determine the horizontal and vertical distances of the area captured in the divided image. The position acquisition unit 150 can also determine the horizontal and vertical distances of a single pixel constituting the divided image, based on the horizontal and vertical distances of the area captured in the divided image. The position acquisition unit 150 acquires each of the obtained results as the geographical location of the divided image.

[0035] The position acquisition unit 150 stores information representing the geographical location of the acquired segmented image (hereinafter referred to as "location information") in the storage unit 130. At this time, the position acquisition unit 150 associates the segmented image ID with the location information and stores it in the storage unit 130. The position acquisition unit 150 may further associate the location information with the segmented image from which the geographical location has been acquired and store it in the storage unit 130 as a new segmented image. In other words, the position acquisition unit 150 may further associate the location information with the segmented image that has been read from the storage unit 130 and associated with the image information, GPS information, aerial image ID, segmented image ID, and segmented location information, and store it in the storage unit 130. The reading of segmented images from the storage unit 130 by the position acquisition unit 150, and the storage of location information, or new segmented images with further associated location information, in the storage unit 130 may be performed by DMA.

[0036] The plant detection unit 160 reads out the segmented images (which may be segmented images stored in the storage unit 130 by the position acquisition unit 150 as new segmented images) that have been segmented by the image segmentation unit 140 and stored in the storage unit 130, and detects the state of a predetermined type of plant (in this case, weed cover) depicted in the read segmented images. The detection of the state of a plant in the plant detection unit 160 is performed using a pre-trained model (hereinafter referred to as the "plant trained model") in which various states of a plant have been learned in advance for each type of plant. The plant trained model is pre-trained to output the state of the plant to be detected when an image of the plant whose state to be detected (in this case, a segmented image) is input. The plant trained model is generated, for example, by learning the relationship between the characteristics of a plant, such as the shape, size, and color of its leaves and flowers, and the growth state of that plant, as the state of the plant, for example, using functions of AI (Artificial Intelligence).

[0037] Here, we will describe an example of a pre-trained plant model used by the plant detection unit 160 to detect weed cover. Figure 2 is a schematic diagram showing an example of a method for generating a pre-trained plant model.

[0038] A pre-trained plant model TM is a model trained using techniques such as CNN (Convolutional Neural Network) or DNN (Deep Neural Network). More specifically, a pre-trained plant model TM is a model trained to recognize (detect) the type of plant depicted in a two-dimensional image when it is input, and to output as detection results a numerical value representing the percentage (coverage) of the area occupied by the detected plant within the input image, and an image representing this value in terms of color and color gradation (brightness, intensity, etc.) (hereinafter referred to as the "coverage image"). A CNN is a neural network in which several layers such as convolutional layers and pooling layers are connected. A DNN is a neural network in which layers of any form are connected in multiple layers. A pre-trained plant model TM is generated by machine learning using a plant recognition machine learning model LM, for example, with a computing device not shown.

[0039] The plant recognition machine learning model LM is a model with provisionally set parameters, for example, taking the form of a CNN or DNN. The plant recognition machine learning model LM may also be a model generated as learning in a domain different from that of detecting weed cover (for example, a model generated by learning to distinguish between plant species). In this case, the plant-trained model TM is generated by transfer learning in a computing device not shown.

[0040] The unillustrated computing device is, for example, a device operated by a user P1 or user P2 who uses the information processing device 100, a provider who provides the functions of the information processing device 100, or an administrator who manages the functions of the information processing device 100. The unillustrated computing device is, for example, a computer device such as a personal computer or a server device. The unillustrated computing device may also be, for example, a terminal device T1 or a terminal device T2. When generating a plant-trained model TM by machine learning, the unillustrated computing device receives plant-training data DI as input data to the input side of the plant-recognition machine learning model LM, and plant-coverage ground truth data DO as training data to the output side of the plant-recognition machine learning model LM.

[0041] The plant training data (DI) is image data of multiple two-dimensional images containing at least the target plant, extracted to the size of a processing unit from a training two-dimensional image taken separately from aerial images. The size of the processing unit is large enough to recognize at least the shape of the plant's leaves. The positions of the plant training data (DI) extracted from the training two-dimensional image (TIM) are random. Figure 2 shows how multiple segmented images (SIM) of processing unit size, containing regions showing weeds (in this case, dock), are input to the plant recognition machine learning model (LM) as plant training data (DI).

[0042] The plant cover ground truth data DO represents the detection results in the plant training data DI. Figure 2 shows how the weed regions WA contained in each segmented image SIM, which are cut out from the training two-dimensional image TIM, are extracted, and how the extracted region WA represents the proportion (coverage) of the plant within each segmented image SIM cut out from the training two-dimensional image TIM. This plant cover data, represented by the plant cover ground truth data DO, is input into the plant recognition machine learning model LM. The extraction of region WA within the segmented image SIM can be performed, for example, by user P1 or user P2 using the information processing device 100, a provider of the functions of the information processing device 100, or an administrator managing the functions of the information processing device 100, or it can be performed using a pre-trained model that has been trained to distinguish between plant types. In other words, the extraction of region WA within the segmented image SIM can be performed manually or automatically. The plant cover in the segmented image SIM can be determined, for example, by the number of pixels in region WA relative to the total number of pixels in the segmented image SIM. For example, in the example shown in Figure 2, the plant cover in each segmented image SIM can be determined by dividing the number of pixels in the region WA within the corresponding segmented image WASIM by the total number of pixels in the segmented image WASIM. The plant cover in the segmented image SIM may be determined manually or automatically. The plant cover ground truth data DO may include, in addition to the plant cover data, data representing the weed region WA extracted from the segmented image SIM cut out from the training two-dimensional image TIM (e.g., segmented image WASIM).

[0043] A computing device (not shown) adjusts the parameters of the plant recognition machine learning model LM so that when the plant learning data DI is input to the plant recognition machine learning model LM, the output of the plant recognition machine learning model LM approaches the detection result (plant cover data) indicated by the correct plant cover data DO. One method for adjusting the parameters is, for example, the backpropagation method. The plant recognition machine learning model LM, whose parameters have been adjusted by the computing device (not shown), becomes the plant-trained model TM. The plant-trained model TM is an example of the "plant-trained model" in the claims.

[0044] As described above, the plant training data DI and plant cover ground truth data DO, which are input to the plant recognition machine learning model LM to generate the plant training model TM, are image data of a size that corresponds to a processing unit, and their size (number of pixels) is such that they can at least recognize the shape of the plant's leaves. The size of the processing unit is also the size of the divided images that the image division unit 140 divides from the aerial image. However, the aerial images taken by the information processing device 100 to actually detect the state of weeds will have different field of view depending on the altitude of the flying object FO, that is, the distance between the weeds on the ground and the imaging device C, even if the same weeds were photographed at the same time. For this reason, if aerial images of the same weeds taken from different altitudes are divided into divided images of the same processing unit size, the detection results output by the plant training model TM will be different. For example, when the altitude of the flying object FO is low, the same weeds will be photographed larger, resulting in a high weed cover, while when the altitude of the flying object FO is high, the same weeds will be photographed smaller, resulting in a low weed cover. Therefore, in order to obtain the same detection results from aerial images of the same weed taken from different altitudes, the plant recognition machine learning model LM is generated not only for each type of plant but also for each altitude of the flying object FO. For example, for the same type of plant, multiple plant recognition machine learning models LM are generated, each receiving input images with different processing unit sizes, i.e., segmented images of different sizes. For example, for an aerial image taken from an altitude of 8m from the flying object FO, a plant recognition machine learning model LM is generated with a processing unit size of 128 pixels × 128 pixels in both the horizontal and vertical directions. For example, for an aerial image taken from an altitude of 4m from the flying object FO, a plant recognition machine learning model LM is generated with a processing unit size of 256 pixels × 256 pixels, and for an aerial image taken from an altitude of 16m from the flying object FO, a plant recognition machine learning model LM is generated with a processing unit size of 64 pixels × 64 pixels.Here, if the altitude of the flying object FO is within a predetermined range (for example, within a range of 1 m in front or behind), the plant detection unit 160 can detect the state of the plant using the plant recognition machine learning model LM corresponding to the closest altitude. Therefore, it is not necessary to generate an unnecessarily large number of plant recognition machine learning models LM, such as generating a different plant recognition machine learning model LM every 50 cm of the altitude of the flying object FO.

[0045] Returning to Figure 1, the plant detection unit 160 obtains a detection result by inputting the read segmented images to the plant detection model. The plant detection unit 160 obtains one detection result for each segmented image. The plant detection unit 160 stores the obtained detection results in the storage unit 130. At this time, the plant detection unit 160 stores in the storage unit 130 information indicating which segmented image the detection result was obtained for. More specifically, the plant detection unit 160 stores in the storage unit 130 the segmented image ID that was assigned to the segmented image for which the detection result was obtained, in association with the detection result. The plant detection unit 160 may also store in the storage unit 130 a detection result that is further associated with the position information that was associated with the segmented image for which the detection result was obtained. The plant detection unit 160 may also store in the storage unit 130 a new segmented image by associating the detection result with the segmented image for which the detection result was obtained. In other words, the plant detection unit 160 may store the detection results in the storage unit 130, along with the segmented images associated with the image information, GPS information, aerial image ID, segmented image ID, and segmented position information (which may also include position information) read from the storage unit 130. The reading of segmented images from the storage unit 130 by the plant detection unit 160, and the storage of detection results, or segmented images associated with detection results, in the storage unit 130 may be performed by DMA.

[0046] The detection data generation unit 170 reads the detection results detected by the plant detection unit 160 and stored in the storage unit 130, and generates detection data representing the detection results for each segmented image based on the read detection results. The detection data is a compilation of the detection results detected for each segmented image, grouped into units of aerial images. In other words, the detection data is data (raster data) that combines the detection results into the same two-dimensional region as the aerial image. The detection data generation unit 170 generates detection data that combines at least the detection results and position information for each segmented image into units of aerial images. The detection data may also be in the form of image data in which the coverage image of the detection result is placed at the position of each segmented image included in the aerial image. If the detection data is in the form of image data, the detection data generation unit 170 places the position information and the coverage image as pixel data at the position corresponding to each segmented image within the same image region as the aerial image. In the following description, detection data in the form of image data will be referred to as the "detection image". The detection data generation unit 170 may, instead of, or in addition to, placing position information as pixel data at the positions corresponding to each segmented image, add geographical location information (GPS information) representing the latitude, longitude, altitude, etc., of the center position of the aerial image as information for the center position of the detection image. The detection image is image data in TIFF (Tagged Image File Format) format, also known as GeoTIFF format, which contains geographical location information, i.e., geo-referenced information. The format of the detection data is not limited to the format of the image data, and may be, for example, data (e.g., text data) that associates the numerical value of the weed cover represented by the detection result with the geographical location (latitude and longitude of the center position of the segmented image) represented by the position information corresponding to each segmented image included in the aerial image.

[0047] Here, we will describe an example of a detection image generated by the detection data generation unit 170 when the plant detection unit 160 detects the weed cover. In the following description, it is assumed that the detection result obtained by the plant detection unit 160 is a cover image that represents the weed cover in the divided images using shades of color. Figure 3 is an example of detection data (detection image) representing the detection result detected by the plant detection unit 160. Figure 3(a) shows an example of an aerial image (original aerial image) AIM of the detection target area where the weeds whose cover is detected by the plant detection unit 160 are growing. Figure 3(b) shows an example of when the detection data generation unit 170 generates a detection image DIM with the same image area as the aerial image AIM, based on the detection results (coverage images) detected by the plant detection unit 160 for each divided image divided from the aerial image AIM. In the detection image DIM shown in Figure 3(b), the coverage image is placed as pixel data at the position corresponding to each segmented image within the detection image DIM, based on the positional information associated with the segmented image from which the coverage image was obtained. In the detection image DIM shown in Figure 3(b), each of the rectangular regions shown in the same shade of color is equivalent to the segmented image region of each segmented image separated from the aerial image AIM.

[0048] Returning to Figure 1, the detection data generation unit 170 stores the generated detection data in the storage unit 130. At this time, the detection data generation unit 170 stores in the storage unit 130 information indicating which aerial image the detection data corresponds to, in association with the detection data. More specifically, the detection data generation unit 170 stores in the storage unit 130 the aerial image ID that was attached to the aerial image corresponding to the generated detection data, in association with the detection data. The detection data generation unit 170 may also store in the storage unit 130 the GPS information that was attached to the aerial image corresponding to the generated detection data, in further association with the detection data. The reading of detection results from the storage unit 130 and the storage of detection data in the storage unit 130 by the detection data generation unit 170 may be performed by DMA.

[0049] Later, for example, an application running on terminal device T2 reads the detection data stored in the storage unit 130 by the detection data generation unit 170 from the storage unit 130 via the network NW, and presents the read detection data to user P2 by displaying it on a display device provided by terminal device T2. The method of presenting the detection data to user P2 on terminal device T2 may be, for example, by image display software capable of displaying image data in GeoTIFF format.

[0050] When the detection data generation unit 170 generates a detection image, it may display the generated detection image on, for example, a display device (not shown) provided by the information processing device 100. In this case, the detection data generation unit 170 may read an aerial image corresponding to the detection image to be displayed from the storage unit 130 and superimpose the detection image onto the read aerial image to represent the location where weed cover was detected on the aerial image. Furthermore, the detection data generation unit 170 may separately obtain a map image corresponding to the location where the aerial image was taken from, for example, the Geospatial Information Authority of Japan or other organizations (including companies), and superimpose the detection data onto the obtained map image to represent the location where weed cover was detected on the map image. The display device (not shown) provided by the information processing device 100 is also an example of a "display device" in the claims.

[0051] Here, we will explain an example of how detection images generated by the detection data generation unit 170 are superimposed on a map image. Figure 4 is a diagram showing an example of a screen displaying detection data (detection images) generated by the detection data generation unit 170. Figure 4 shows an example of how detection images are presented to user P2 by displaying them on a display device provided by terminal device T2. The presentation of detection images on the screen shown in Figure 4 is performed, for example, by an application running on terminal device T2 (hereinafter simply referred to as "application"). In Figure 4, terminal device T2 superimposes multiple detection images DIM corresponding to each of the multiple (series) aerial images it has taken onto a map image MIM acquired separately, and then superimposes circular points representing the center position of each detection image DIM, presenting them to user P2. Figure 4 shows a case where the ranges captured by adjacent aerial images overlap in the multiple (series) aerial images taken. In this case, the detection image DIMs for the overlapping ranges of the aerial images will also overlap. In this case, the application may display the detection image DIM corresponding to the aerial image taken later in the overlapping area, or it may display the detection result for the overlapping area between each detection image DIM, that is, the average pixel value in the detection image DIM.

[0052] Furthermore, Figure 4 shows the operation areas for the detection target selection panel TS, for which user P2 selects (sets) the plant to be detected, and the detection result selection panel RS, for which user P2 selects the detection result to be displayed. When user P2 selects a plant to be detected for coverage using the detection target selection panel TS shown in Figure 4, this information is sent to the information processing device 100 by the application. As a result, the information processing device 100 detects the coverage of the plant selected by user P2 and stores the detection result as new detection data (detection image DIM) in the storage unit 130. The application then reads the new detection data (detection image DIM) generated by the information processing device 100 from the storage unit 130 and overlays it on the map image MIM, or replaces the current detection image DIM overlaid on the map image MIM with the new detection image DIM. When user P2 selects a detection result to display using the detection result selection panel RS shown in Figure 4, the application, based on the respective detection results (i.e., pixel values) placed in the detection image DIM, either superimposes the detection image DIM with the detection result (pixel value) that matches user P2's selection onto the map image MIM, or replaces the current detection image DIM superimposed on the map image MIM.

[0053] In Figure 4, the detection image DIM is superimposed on the map image MIM and displayed, along with the detection target selection panel TS and the detection result selection panel RS. However, this is merely one example. The application may, for example, initially superimpose an aerial image onto the map image MIM and display a detection condition setting panel for the user P2 to set detection conditions such as the detection target area. In this case, when the user P2 sets or changes the detection conditions, the application sends that information to the information processing device 100. As a result, the information processing device 100 detects the state of the plant using the detection conditions set or changed by the user P2 and stores the detection result as new detection data (detection image DIM) in the storage unit 130. The application then reads the detection data (detection image DIM) generated by the information processing device 100 from the storage unit 130 and presents it to the user P2 by superimposing it onto the map image MIM as the detection result detected under the set detection conditions.

[0054] [Plant detection processing in information processing equipment] Next, we will describe an example of the overall flow of the process for detecting the state of a plant (in this case, weed cover) in the information processing device 100. Figure 5 is a flowchart of an example of the process for detecting the state of a plant in the information processing device 100. Figure 6 is a schematic diagram showing an example of the process for detecting the state of a plant in the information processing device 100. In the following explanation, we will refer to Figure 6 as appropriate to explain the process flow in the information processing device 100 shown in Figure 5. In the following explanation, we will assume that the flight of the flying object FO and the capture of aerial images by the imaging device C are controlled by the terminal device T1, and that the detection result in the information processing device 100 is confirmed by displaying the detected image on the display device provided by the terminal device T2. Furthermore, we will assume that the information processing device 100 already has a plant training model TM prepared that corresponds to the detection of the plant (detection of weed cover), that is, a plant model that is adapted to the type of plant and the altitude of the flying object FO.

[0055] In response to control by terminal device T1, the flying object FO flies in a straight line at the same altitude in a certain direction, and the imaging device C takes multiple (sequential) aerial images. Here, it is assumed that the imaging device C takes each (sequential) aerial image AIM in the order of aerial image AIM-1, aerial image AIM-2, and aerial image AIM-3, as shown in Figure 6(a). In each aerial image AIM shown in Figure 6(a), a circle is shown at the center position represented by the associated GPS information, indicating that the flying object FO is flying in a straight line (at the same altitude) in a certain direction. In response to control by terminal device T1, the imaging device C transmits each captured aerial image AIM to terminal device T1. Terminal device T1 transmits each aerial image AIM transmitted by the imaging device C to the information processing device 100. In the following explanation, we will focus on the case where the information processing device 100 detects the degree of weed cover captured in aerial image AIM-2, which is one of the aerial images AIM-1 to AIM-3 transmitted by the terminal device T1.

[0056] The image acquisition unit 120 of the information processing device 100 acquires the aerial image AIM-2 transmitted by the terminal device T1 (step S100). The image acquisition unit 120 assigns an aerial image ID to the acquired aerial image AIM-2 and stores it in the storage unit 130. The image acquisition unit 120 may also notify the image splitting unit 140 that it has acquired the aerial image AIM-2 and stored it in the storage unit 130.

[0057] The image splitting unit 140 reads the aerial image AIM-2 stored in the storage unit 130 and splits it into split image SIMs (step S102). Figure 6(a) shows an example of a split image SIM (split image region) split by the image splitting unit 140 for each aerial image AIM. As shown in Figure 6(a), the image splitting unit 140 splits each split image SIM from each aerial image AIM in the same way. The image splitting unit 140 assigns a split image ID to each split image SIM, associates the image information of the original aerial image, GPS information, and aerial image ID with the split position information, and stores it in the storage unit 130. The image splitting unit 140 may notify the position acquisition unit 150 that each split image SIM has been stored in the storage unit 130.

[0058] The position acquisition unit 150 sequentially reads each segmented image SIM (in this case, each segmented image SIM separated from the aerial image AIM-2) stored in the storage unit 130 and acquires the geographical location of each segmented image SIM (step S104). The position acquisition unit 150 associates the segmented image ID of each segmented image SIM with the location information representing the geographical location acquired for the segmented image SIM of this segmented image ID and stores it in the storage unit 130. The position acquisition unit 150 may also notify the plant detection unit 160 that it has acquired the geographical location of each segmented image SIM and stored it in the storage unit 130.

[0059] The plant detection unit 160 sequentially reads each segmented image SIM (in this case, each segmented image SIM divided from the aerial image AIM-2) stored in the storage unit 130, and detects the target plant whose cover to be detected based on the shape of the leaves of each plant shown in the segmented image SIM (step S106). Then, if the plant detection unit 160 detects the target plant, it detects the cover of this plant (step S108). The plant detection unit 160 associates the segmented image ID of the segmented image SIM with the detection result for each segmented image SIM and stores it in the storage unit 130. The plant detection unit 160 may also notify the detection data generation unit 170 that the detection result for each segmented image SIM has been stored in the storage unit 130.

[0060] The detection data generation unit 170 sequentially reads each detection result stored in the storage unit 130 (in this case, the detection results of each segmented image SIM separated from the aerial image AIM-2) and generates a detection image DIM as detection data (step S110). Figure 6(b) shows examples of detection image DIMs for DIM-1 corresponding to aerial image AIM-1, DIM-2 corresponding to aerial image AIM-2, and DIM-3 corresponding to aerial image AIM-3. In each detection image DIM shown in Figure 6(b), a coverage image is placed at the position corresponding to each segmented image SIM. In each detection image DIM shown in Figure 6(b), a circle is shown at the center position represented by the GPS information associated with the corresponding aerial image AIM. As shown in Figure 6(b), each detection image DIM is also a linear arrangement of the flying object FO flying in a certain direction. In each detection image DIM shown in Figure 6(b), a circle is indicated at the center position represented by the GPS information associated with the aerial image ID of the corresponding aerial image AIM, indicating that each detection image DIM is arranged in a linear fashion. The detection data generation unit 170 associates the aerial image ID of the aerial image AIM corresponding to each generated detection image DIM and stores it in the storage unit 130. The detection data generation unit 170 may also notify the terminal device T2 that it has detected the weed cover in the aerial image AIM-2 and stored it in the storage unit 130.

[0061] Through this process, the information processing device 100 detects the weed cover for each aerial image AIM captured by the imaging device C mounted on the flying object FO, generates a detection image DIM as a result of the detection, and stores it in the storage unit 130. The detection of weed cover for each aerial image AIM can be performed by repeating steps S100 to S110 shown in Figure 5 the same number of times as the number of aerial image AIMs for which weed cover is detected. In this case, if the information processing device 100 has acquired all aerial image AIMs captured by the imaging device C simultaneously in step S100, it can repeat steps S102 to S110 shown in Figure 5 the same number of times as the number of acquired aerial image AIMs (aerial image AIMs for which weed cover is detected).

[0062] Subsequently, the terminal device T2 displays the detection images DIM stored in the storage unit 130 on the display device (step S200). At this time, the terminal device T2 displays each detection image DIM corresponding to the entire range of the detection target area. Figure 6(c) shows an example of a display screen in which each detection image DIM is displayed together to cover the entire range of the detection target area. This allows the user P2 to confirm the detection result of the weed cover in the detection target area by the information processing device 100, which is displayed on the display device of the terminal device T2.

[0063] Through this configuration and processing, the information processing device 100 acquires aerial images captured by the imaging device C mounted on the flying object FO. The information processing device 100 then divides the acquired aerial images into multiple segmented images, each representing a processing unit for detecting the state of a plant, and obtains the geographical location of each segmented image. Subsequently, the information processing device 100 uses a plant-trained model TM to detect the state of a plant in each segmented image and generates detection data that aggregates the detection results into units of aerial images. Here, the detection data generated by the information processing device 100 is data that can be provided to the user P2 by, for example, displaying it on a display device provided by the terminal device T2.

[0064] Furthermore, the information processing device 100 can detect the state of a plant from two-dimensional aerial images captured by the imaging device C mounted on the flying object FO, without using three-dimensional data such as orthomosaic images generated by the SfM method, which is considered a conventional method for detecting the state of a plant. In other words, the information processing device 100 does not require the use of SfM, a method with high processing costs to generate orthomosaic images, nor does it require many overlaps to apply SfM to each aerial image. Moreover, it can detect the state of a plant more simply and with less processing cost, without being affected by external factors that may cause distortion in the orthomosaic image. In addition, the information processing device 100 can obtain detection results that are sufficiently effective for the purpose of checking the state of a plant even if there are no overlapping areas between adjacent aerial images, or in other words, even if there are gaps between adjacent aerial images.

[0065] Here, we will compare the difference in detection results when there is overlap between aerial images and when there is no overlap. Figure 7 shows a comparison of the detection results of the state of plant bodies (in this case, weed cover) detected by the information processing device 100. Figures 7(a) and (b) show examples of screens in which multiple detection images DIM, representing the detection results corresponding to each of the multiple (series) aerial images that have been taken, are superimposed on a map image MIM. The screen shown in Figure 7(a) is an example where adjacent aerial images are laid out without gaps (the overlapping area is not sufficient to use the SfM method, but some areas of the aerial images overlap), and the screen shown in Figure 7(b) is an example where there are gaps between adjacent aerial images (there are areas where the aerial images do not overlap).

[0066] As can be seen by comparing the screen shown in Figure 7(a) and the screen shown in Figure 7(b), in both screens, areas with high and low weed cover can be similarly distinguished. More specifically, as shown in Figure 7(b), even when there is a certain gap G_A-B between detection image DIM-A and detection image DIM-B, areas with high and low weed cover can be distinguished (confirmed) in the same way as when detection image DIM-A and detection image DIM-B overlap as shown in Figure 7(a).

[0067] Therefore, the information processing device 100 can tolerate a certain amount of space between each aerial photograph. As a result, the information processing device 100 can intentionally reduce the number of aerial photographs taken to detect the state of plants within the detection target area. In this case, the information processing device 100 can further reduce the processing cost for detecting the state of plants.

[0068] As described above, the information processing device 100 of the embodiment acquires a two-dimensional aerial image captured by the imaging device C mounted on the flying object FO. The information processing device 100 of the embodiment then divides the acquired aerial image into multiple segmented images, which are processing units for detecting the state of the plant, and acquires the geographical location of each segmented image. Subsequently, the information processing device 100 of the embodiment detects the state of the plant for each segmented image using a plant-trained model TM. This allows the information processing device 100 of the embodiment to detect the state of the plant more easily and with less processing cost. The information processing device 100 of the embodiment then generates detection data that allows the detection results to be provided to the user P2 by, for example, displaying them on a display device provided by the terminal device T2, by combining the detection results into units of aerial images. In other words, the information processing device 100 of the embodiment can detect the state of the plant while keeping the two-dimensional aerial image captured by the imaging device C. This allows the information processing device 100 of the embodiment to generate detection data that detects the state of the plant more easily and with less processing cost.

[0069] In this embodiment, the case was described in which the information processing device 100 detects a predetermined weed cover as the state of a plant, that is, the state of one type of plant. However, as described above, the plant learning model TM used by the information processing device 100 to detect the state of a plant is prepared for various states of a plant and for each type of plant. Therefore, the information processing device 100 (more specifically, the plant detection unit 160) can also detect different states of plants or different types of plants from the same aerial image.

[0070] Here, we will describe an example of how the information processing device 100 can detect the state of different types of plants from the same aerial image. In the following description, the plant that the plant detection unit 160 detects is pasture grass, and the state of the plant is the cover of the pasture grass. The plant detection unit 160 will detect the cover of two types of pasture grass, for example, pasture grass with narrow leaves (narrow-leaved), such as grasses of the Poaceae family, and pasture grass with broad leaves (broad-leaved), such as pasture grasses of the Fabaceae family. In the following description, the detection result obtained by the plant detection unit 160 will be a cover image in which the cover of the pasture grass in the divided image is represented by the intensity of the colors.

[0071] Figure 8 shows an example of detection data (detection image) representing the detection result when the plant detection unit 160 of the information processing device 100 detects the state of different plants. Figure 8(a) shows an example of an aerial image AIM2 of the detection target area of ​​a mixed-cropped field where two types of pasture grass are cultivated for which the plant detection unit 160 detects the cover. The aerial image AIM2 shows both narrow-leaved pasture grass and broad-leaved pasture grass. Figure 8(b-1) shows an example in which the detection data generation unit 170 generates a detection image DIM2-1 having the same image area as the aerial image AIM2, based on the detection result (coverage image) of narrow-leaved pasture grass detected by the plant detection unit 160 for each divided image divided from the aerial image AIM2. Figure 8(b-2) shows an example in which the detection data generation unit 170 generates detection images DIM2-2, which have the same image area as the aerial image AIM2, based on the detection results (coverage images) of broadleaf pasture grass detected by the plant detection unit 160 for each segmented image divided from the aerial image AIM2. In the detection image DIMs shown in Figure 8(b-1) and (b-2), the corresponding coverage images are placed as pixel data at the positions corresponding to each segmented image within the detection image DIM, based on the positional information associated with the segmented image from which the coverage image was obtained. In the detection image DIMs shown in Figure 8(b-1) and (b-2), each of the rectangular areas shown in the same shade of color is equivalent to the segmented image area of ​​each segmented image divided from the aerial image AIM2.

[0072] In this way, the information processing device 100 can detect different states of plants or the states of different types of plants from the same aerial image. In this case, the operation and processing of the information processing device 100 should be equivalent to the operation and processing of the information processing device 100 in the embodiment described above. This is true not only when the plant state detected by the information processing device 100 is cover, but also when detecting the growth state of plants, such as when detecting whether or not a plant is affected by disease, that is, when detecting the state of a plant to diagnose disease symptoms, or when detecting the state of a plant to check the growth rate of a cultivated plant.

[0073] The detection data detected by the information processing device 100 can then be displayed as detection image DIMs on a display device, for example, on a terminal device T2. In this case, multiple detection data can be provided to the user P2 simultaneously by superimposing multiple detection image DIMs and displaying them on the same screen of the display device. As a result, the user P2 can check multiple different states of plant bodies, or the states of multiple different types of plant bodies, detected by the information processing device 100, from the screen displayed on the display device of the terminal device T2.

[0074] In the embodiment, the flying object FO was described as flying in a straight line at a predetermined altitude in a predetermined direction, and the imaging device C was described as capturing the upper part of the aerial image in a constant direction (azimuth) even if the direction of flight of the flying object FO changed. However, if the direction or altitude of flight of the flying object FO changes unintentionally due to external factors such as wind blowing during the capture of the aerial image, it is conceivable that the field of view and shooting direction of the aerial image captured by the imaging device C may be shifted. In other words, it is conceivable that inconveniences not caused by the processing performed to detect the state of the plant body may occur. In this case, if the field of view and shooting direction of the aerial image are set to the intended values ​​before the information processing device 100 starts detecting the state of the plant body, the state of the plant body can be detected in the same way as in the embodiment. For example, if the flight path and altitude correction functions of the flying object FO itself allow it to return to the intended field of view and shooting direction in a short time, the aerial image capture can continue. If a large gap appears between adjacent aerial images, the system may capture the gap again after the initial series of aerial images have been captured. For example, the imaging device C may capture an aerial image with a wider field of view, and under normal circumstances, a predetermined range based on the center position of the aerial image is cropped. If a shift occurs in the field of view or shooting direction, the position and range from which the image is cropped can be changed to eliminate the influence of external factors within the expected range (to achieve the intended field of view and shooting direction). This image cropping process may be performed, for example, by the electronic zoom function of the imaging device C, or by terminal devices T1 and T2, or even by the information processing device 100.

[0075] In the embodiment described above, the terminal device T1 controls the flight of the flying object FO and the capture of aerial images by the imaging device C, and the image acquisition unit 120 of the information processing device 100 acquires the aerial images captured by the imaging device C via the communication unit 110 and the terminal device T1. However, the information processing device 100 may also be configured to include a control unit that directly controls the flight of the flying object FO and the capture of aerial images by the imaging device C. In other words, the information processing device 100 may be configured to operate in cooperation with the flying object FO and the terminal device T1. In this case, the operation and processing of the information processing device should be equivalent to the operation and processing of the information processing device 100 and terminal device T1 in the embodiment described above.

[0076] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0077] 100... Information Processing Device 110... Communications Department 120...Image acquisition unit 130...Storage section 140...Image division section 150...Position acquisition unit 160... Plant detection unit 170...Detection data generation unit F...field FO...Flying object C-Imaging Device T1, T2... Terminal devices NW... Network TM... Plant body trained model AIM,AIM-1,AIM-2,AIM-3,AIM2...Aerial image SIM... Split image DIM, DIM-1, DIM-2, DIM-3, DIM-A, DIM-B, DIM2-1, DIM2-2... Detection Images MIM...Map image

Claims

1. An image acquisition unit that acquires a two-dimensional image including plant bodies, taken from above by a flying object, An image division unit that divides the region of the aforementioned image into multiple divided images of a predetermined size, A position acquisition unit that acquires the geographical location of each of the divided images based on geographical information representing the central position of the image contained in the aforementioned image, A plant detection unit that detects the state of a predetermined type of plant captured in each of the aforementioned segmented images, A detection data generation unit generates detection data that associates the geographical location of the acquired segmented image with the state of the predetermined type of plant detected, Equipped with, The state of the predetermined type of plant body is the coverage for each type of plant body, The detection data generation unit, The detection data is associated with the predetermined type of plant body, and a detection image is generated in which the coverage of the plant body is represented by the intensity of the color, and the detection data is represented in image format. The generated detection image is superimposed onto the map image and displayed on a display device. Information processing device.

2. The position acquisition unit acquires the geographical position of the segmented image, which is determined based on the center position of the image represented by the geographic information, the direction of movement of the flying object when the image was taken, the height of the flying object from the ground when the image was taken, and the shooting range of the image, using the center position of the image represented by the geographic information as a reference. The information processing apparatus according to claim 1.

3. The plant detection unit, by inputting the segmented images into a plant training model that has been trained to output the coverage of each predetermined type of plant, obtains and detects the state of the predetermined type of plant. The information processing apparatus according to claim 1 or claim 2.

4. The coverage for each type of plant is the coverage of that plant within the divided image. An information processing apparatus according to any one of claims 1 to 3.

5. The detection data generation unit generates the detection data for each of the divided images, representing the magnitude of coverage for each type of plant. The information processing apparatus according to claim 4.

6. Computers A two-dimensional image, including plants, is obtained from above by a flying object. The region of the aforementioned image is divided into multiple segmented images of a predetermined size, Based on the geographic information representing the central position of the image contained in the aforementioned image, the geographical location of each of the aforementioned divided images is obtained. The state of a predetermined type of plant captured in each of the aforementioned segmented images is detected, Detection data is generated by associating the geographical location of the acquired segmented image with the state of the predetermined type of plant detected. The state of the predetermined type of plant body is the coverage for each type of plant body, The coverage of the plant body is represented by the intensity of color for each predetermined type associated with the detection data, A detection image is generated that represents the detection data in image format. The generated detection image is superimposed onto the map image and displayed on a display device. Information processing methods.

7. On the computer, A two-dimensional image, including plant matter, is obtained from above by a flying object. The region of the aforementioned image is divided into multiple segmented images of a predetermined size. Based on the geographic information representing the central position of the image contained in the aforementioned image, the geographical location of each of the aforementioned divided images is obtained. The state of a predetermined type of plant captured in each of the aforementioned segmented images is detected. The system generates detection data that associates the geographical location of the acquired segmented image with the state of the predetermined type of plant detected. The state of the predetermined type of plant body is the coverage for each type of plant body, The degree of coverage of the plant body is represented by the intensity of color for each predetermined type associated with the detection data. A detection image is generated that represents the detection data in image format. The generated detection image is superimposed onto the map image and displayed on a display device. program.

Images