Method, program and information processing device
The use of multispectral imaging and processing systems allows agricultural businesses to accurately assess and manage crop growth conditions, addressing the challenge of visual inspection limitations.
Patent Information
- Application Number
- JP2022088324
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-05-31
- Publication Date
- 2024-11-20
- Estimated Expiration
- 2042-05-31
AI Technical Summary
Agricultural businesses face challenges in accurately assessing crop growth conditions beyond visual inspection, necessitating improved methods for managing growing conditions during harvesting and shipping.
A method involving the use of multispectral cameras mounted on drones and satellites to capture spectral images, which are processed by an information processing device to identify areas with specific crop states, enabling precise management of growing conditions.
Enables agricultural businesses to manage crop growth conditions more effectively, allowing for timely interventions and improved yield management.
Smart Images

Figure 0007587282000001 
Figure 0007587282000002 
Figure 0007587282000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a method, a program, and an information processing device. [Background technology]
[0002] There is known a technology for capturing images of land on which agricultural crops (hereinafter generally referred to as "crops" in this specification) are grown using a spectral camera to obtain a spectral image of the land, for the purpose of understanding the growth of the crops, pests, and soil conditions (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2017 / 179378 Summary of the Invention [Problem to be solved by the invention]
[0004] For agricultural businesses, it is assumed that growers know where crops are planted when they are cultivated and harvested. However, when harvesting and shipping crops, it may not be possible to determine whether the crops are growing well just by looking at the density of vegetation.
[0005] Therefore, there is a need for technology that enables businesses that grow and harvest crops to more appropriately manage the growing conditions.
[0006] Therefore, the present disclosure has been made to solve the above-mentioned problems, and its purpose is to provide a method, program, and information processing device that enable businesses engaged in harvesting operations to more appropriately manage the growing conditions. [Means for solving the problem]
[0007] According to one embodiment, a method for operating a computer is provided, which causes a processor of the computer to execute a first step of storing information for detecting that a crop being observed is in a specific state in a memory for comparison with a spectral image captured by a multispectral camera, a second step of generating a spectral image in the observed area by capturing an image with the multispectral camera at wavelengths for detecting that the crop is in the specific state, a third step of identifying an area in the observed area where a crop in the specific state is located based on the captured spectral image and the information for detecting that the crop is in the specific state, and a fourth step of outputting information on the identified area. Effect of the Invention
[0008] According to the present disclosure, it becomes possible for business operators who carry out harvesting businesses to more appropriately manage the growing conditions. [Brief description of the drawings]
[0009] [Figure 1] 1 is a diagram showing an overview of a system according to an embodiment. [Diagram 2] 1 is a block diagram showing a configuration of a system according to an embodiment. [Diagram 3] FIG. 1 is a front view showing a drone according to an embodiment. [Figure 4] FIG. 1 is a plan view showing a drone according to an embodiment. [Diagram 5] FIG. 2 is a block diagram showing the hardware configuration of the drone according to the embodiment. [Figure 6] FIG. 2 is a block diagram showing the functional configuration of a drone according to an embodiment. [Figure 7] FIG. 1 is a conceptual diagram showing an image capturing state of land by a drone according to an embodiment. [Figure 8] FIG. 2 is a block diagram showing a hardware configuration of a satellite according to an embodiment. [Figure 9] FIG. 2 is a block diagram showing the functional configuration of a satellite according to the embodiment. [Figure 10] FIG. 2 is a conceptual diagram showing an image capturing state of land by a satellite according to the embodiment. [Figure 11] 1 is a block diagram showing a hardware configuration of an information processing device according to an embodiment. [Figure 12] FIG. 2 is a diagram illustrating a functional configuration of an information processing device according to an embodiment. [Figure 13] 4 is a diagram showing a data structure of a first imaging condition DB stored in a satellite according to the embodiment. FIG. [Figure 14] FIG. 4 is a diagram showing a data structure of an image DB stored in a satellite according to the embodiment. [Figure 15] FIG. 13 is a diagram showing a data structure of a second imaging end time DB stored in the drone according to the embodiment. [Figure 16] A diagram showing the data structure of a flight plan DB stored in the drone of the embodiment. [Figure 17] 4 is a diagram showing a data structure of a wide area image DB stored in the information processing device according to the embodiment. FIG. [Figure 18] 4 is a flowchart for explaining an example of an operation of a satellite according to an embodiment. [Figure 19] 10 is a flowchart illustrating an example of an operation of the information processing device according to the embodiment. [Figure 20] 11 is a flowchart illustrating an example of an operation of a drone according to an embodiment. [Figure 21] 10 is a flowchart for explaining another example of the operation of the information processing device according to the embodiment. [Figure 22] 11A to 11C are diagrams illustrating an example of an operation of generating a wide-area image in the information processing device according to the embodiment. [Diagram 23] FIG. 1 is a conceptual diagram showing a procedure for determining whether a crop is in a particular state using a spectral image. [Figure 24] FIG. 11 is a diagram illustrating an example of a screen displayed on the information processing device according to the embodiment. [Diagram 25] FIG. 11 is a diagram showing another example of a screen displayed on the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In all the drawings explaining the embodiment, the same reference numerals are given to common components, and repeated explanations are omitted. Note that the following embodiment does not unduly limit the contents of the present disclosure described in the claims. In addition, not all of the components shown in the embodiment are essential components of the present disclosure. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.
[0011] In the following description, a "processor" refers to one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). The at least one processor may be a single-core or multi-core.
[0012] Furthermore, the at least one processor may be a processor in the broad sense, such as a hardware circuit (for example, a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) that performs part or all of the processing.
[0013] In the following explanation, information that gives an output for an input may be described using an expression such as "xxx table", but this information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, the "xxx table" may be called "xxx information".
[0014] Furthermore, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.
[0015] In addition, in the following explanation, the processing may be described with the "program" as the subject, but since the program is executed by a processor to perform a specified processing step by appropriately using a memory unit and / or an interface unit, etc., the subject of the processing may be the processor (or a device such as a controller having the processor).
[0016] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.
[0017] In the following description, identification numbers are used as identification information for various objects, but other types of identification information (for example, identifiers including alphabetic characters or symbols) may be used instead.
[0018] In addition, in the following description, when describing elements of the same type without distinguishing between them, reference signs (or common signs among the reference signs) may be used, and when describing elements of the same type with distinction between them, the identification numbers (or reference signs) of the elements may be used.
[0019] In the following description, the control lines and information lines are those that are considered necessary for the description, and not all control lines and information lines in the product are necessarily shown. All components may be connected to each other.
[0020] In this specification, the term "spectral image" refers to an image consisting of only reflected light of a specific wavelength band (including a specific wavelength) from an object. As an example, as shown in the embodiment described later, a spectral image is generated based on an image sensor that focuses only reflected light of a specific wavelength band by passing the reflected light from an object (crop) through a filter (preferably with a variable transmission wavelength band) that passes only a specific wavelength band, and an image signal output from the image sensor. Alternatively, the spectral image can be generated by focusing the reflected light from an object on an image sensor without passing through a filter, and converting (filtering) the image signal output from the image sensor into an image signal of only a specific wavelength band by signal processing. Note that an image obtained by collecting a plurality of spectral images consisting of only reflected light of a specific wavelength band from the same object may also be referred to as a spectral image.
[0021] In addition, in this specification, the term "aircraft" is used to refer to a moving body that flies in the air and can be operated without a crew member (e.g., an aircraft, drone, satellite, or airship that can be remotely controlled by radio or autonomously).
[0022] Furthermore, in this specification, a crop being in a "specific state" includes the following. - Whether or not the crop is diseased, and what diseases it has or is likely to have -Crop growth status (including whether or not the crop is ready for harvest) - If the target area is forest, the state of vegetation and distribution of vegetation
[0023] And crops include not only agricultural crops but also vegetation. Therefore, when a crop is in a particular state, it also includes the state of the vegetation on which the crop is growing (for example, if the crop is apples, the state of the leaves on the tree on which the apples are growing).
[0024] <Embodiment> <Overview of the embodiment> In the system according to the embodiment, the ground surface including the area where crops are grown is imaged by a first multispectral camera, and an operator checks the spectral image obtained by the image capture, and associates the difference in the appearance of the spectral image in a specific wavelength band in a certain area (this corresponds to the intensity of reflected light in a specific wavelength band and reflected light in other wavelength bands for the same object) with a specific state of the crops in that area. This association is stored as a library. Next, the area to be observed is similarly imaged by a second multispectral camera to obtain a spectral image. The spectral image of the area to be observed is then analyzed according to the above-mentioned library, and an area in this area where the crops are in a specific state is identified.
[0025] The first and second multispectral cameras are preferably mounted on a mobile object, and the mobile object is moved over the ground surface and the area to be observed to acquire a spectral image. As an example of the mobile object, a drone or a satellite, which is an air vehicle, is preferably mentioned, but there is no particular limitation as long as the mobile object can acquire a spectral image by moving the first and second multispectral cameras. Furthermore, the mobile object does not have to be an air vehicle, and the mobile object on which the first and second spectral cameras are mounted may move over the ground surface. For example, the spectral image may be acquired by a means such as an operator holding the first and second spectral cameras moving on foot over the ground surface, or a car equipped with the first and second spectral cameras running over the ground surface. As an example, a multispectral camera that realizes the spectroscopic terminal device disclosed in Japanese Patent No. 6342594 by a smartphone is used.
[0026] The moving body on which the first multispectral camera is mounted and the moving body on which the second multispectral camera is mounted do not have to be the same. In the embodiment described later, the first multispectral camera is mounted on a satellite and the second multispectral camera is mounted on a drone, but the first multispectral camera may be mounted on a drone and the second multispectral camera may be mounted on a satellite. Furthermore, both the first and second multispectral cameras may be mounted on a satellite, or both the first and second multispectral cameras may be mounted on a drone. Furthermore, as described above, at least one of the first and second multispectral cameras may be mounted on a moving body moving on the ground surface.
[0027] Both the first multispectral camera and the second multispectral camera receive reflected light from the earth's surface in a plurality of wavelength bands and generate a spectral image based on the reflected light. Preferably, the first multispectral camera has a member capable of switching the wavelength to be imaged, such as a liquid crystal wavelength tunable filter, and can generate spectral images for a greater number of wavelengths than the second multispectral camera. On the other hand, the second multispectral camera can generate spectral images for a specific wavelength. In this case, it is preferable that the specific wavelength cannot be switched while the second multispectral camera is imaging the observation target area. In other words, the specific wavelength is a predetermined wavelength. Naturally, there is no particular restriction on the magnitude relationship between the number of wavelength bands of the spectral images that can be imaged by the first multispectral camera and the number of wavelength bands of the spectral images that can be imaged by the second multispectral camera.
[0028] In this specification, the term "library" refers to a highly versatile data set and / or program. Preferably, the library is realized as training data in artificial intelligence technology. More preferably, in the system of this embodiment, a learning model is generated using the training data, and an area where the crop is in a specific state is identified based on this training model. Therefore, in the following description, the term "library" is mainly used to refer to training data, but may be referred to as "library" without making any particular distinction between training data and training models. In the following description, the training model also includes a model that has already been trained, i.e., a trained model.
[0029] Although details will be described later, when creating a library, it is necessary to take into consideration the solar altitude at the time the spectral images were captured. This is because the wavelength distribution of the light reflected from the crop varies depending on the angle at which the sun hits the crop (especially the crop leaves), so it is necessary to consider the effect of the solar altitude (sun angle) on the library. However, when creating a library based on the spectral images captured by the first multispectral camera, there is a lack of variation in the spectral images captured by the first multispectral camera (especially the variation in the spectral images based on different solar angles) at the beginning of the operation of the system, and it may be difficult to create a library that takes the solar angle into consideration.
[0030] Therefore, the spectral image captured by the first multispectral camera is taken at a certain solar angle (for example, a spectral image at noon altitude), and a library is created based on the spectral image at the certain solar angle. Then, when acquiring a spectral image with the second multispectral camera, the spectral image is acquired at the same solar angle as the solar angle when the spectral image with the first multispectral camera was acquired, and the influence of the solar angle can be eliminated as much as possible to identify the area where the crop is in a specific state.
[0031] After this, the variation of the spectral images acquired by the first and second multispectral cameras is increased, i.e., spectral images at various solar angles are acquired, and the effect of the solar angle on the library can be formulated.
[0032] When creating a library, and further when identifying an area where crops are in a specific state, there is a need to create a library for a wide range of observation target areas and to perform area identification work. Meanwhile, the imaging range of the spectral images captured by the first and second multispectral cameras is predetermined depending on the angle of view of each multispectral camera and the altitude from the ground surface of the mobile body carrying the multispectral camera. In this case, it is preferable to generate a wide range of spectral images by connecting multiple spectral images. Methods for connecting spectral images are well known, and one example is a method of generating ortho-images from spectral images and connecting the ortho-images. The method for connecting spectral images may be applied to spectral images acquired by either the first multispectral camera or the second multispectral camera.
[0033] An overview of a system to which a method according to an embodiment is applied will be described with reference to Fig. 1. In the system according to the embodiment, an example in which a first multispectral camera is mounted on a satellite and a second multispectral camera is mounted on a drone will be mainly described, but the moving body on which the first and second multispectral cameras are mounted and the method of moving the first and second multispectral cameras are not limited to the disclosure of the embodiment. Furthermore, the configurations of the first and second multispectral cameras (particularly the number of wavelength bands of spectral images that can be acquired) are also not limited to the disclosure of the embodiment.
[0034] In the system 1 according to the embodiment, spectral images for creating a library are generated by a first multispectral camera mounted on a satellite 2, which is an example of an aircraft, and the area to be observed is imaged by a second multispectral camera mounted on a drone 3, which is an example of an aircraft, and the obtained spectral images are used to identify areas where crops are in a specific state.
[0035] However, the aircraft for capturing the spectral images for creating the library and the aircraft for capturing the spectral images for identifying the region may be the same, and there is no restriction on the type. However, capturing the spectral images for creating the library by the satellite 2 has the advantage of being able to efficiently obtain spectral images from a wider range of the earth's surface. On the other hand, capturing the spectral images for identifying the region by the drone 3 has the advantage of being able to easily obtain spectral images by flying the drone 3 only in the region where the state needs to be identified, and furthermore, it has the advantage of being able to reduce the cost of capturing the same region multiple times.
[0036] In addition, although details will be described later, the satellite 2 is equipped with a first multispectral camera using a liquid crystal tunable filter (LCTF), while the drone 3 is equipped with a second multispectral camera (preferably a multispectral camera equipped with multiple cameras capable of capturing only reflected light in a specific wavelength band) capable of capturing spectral images based on a combination of multiple wavelength bands that can clearly identify that the crop is in a specific state in library creation. Of course, there is no limitation on the configuration of the multispectral camera as long as it is capable of capturing images based on reflected light in multiple wavelength bands. In addition, the drone 3 may be equipped with a first multispectral camera using LTCF, and the satellite 2 may be equipped with a second multispectral camera capable of capturing spectral images based on a combination of multiple wavelength bands.
[0037] The first multispectral camera mounted on the satellite 2 can capture reflected light in a specific wavelength band among reflected light in a wide wavelength band from the visible range to the near infrared range by the action of a liquid crystal tunable filter. In capturing reflected light from the Earth's surface, the system 1 of this embodiment scans the wide wavelength band to capture a spectral image based on reflected light in multiple wavelength bands for the same area.
[0038] The spectral image captured by the first multispectral camera mounted on the satellite 2 is transmitted to the information processing device 4. The operator of the information processing device 4 compares a plurality of (preferably three or more wavelength bands) of spectral images captured by the first multispectral camera mounted on the satellite 2 of the same region, identifies a shading pattern indicating that the crop is in a specific state from the shading pattern in each spectral image, and associates this shading pattern with information indicating the specific state. Therefore, it is assumed that the operator of the information processing device 4 understands the growth state of the crops on the ground surface in the above-mentioned same region, and has previously understood what specific state the crops in which region are in based on this growth state. The above-mentioned association is stored in the memory of the information processing device 4 as a library. Preferably, this library is realized as teaching data in artificial intelligence technology. More preferably, when creating the library, the angle of the sun with respect to the ground surface is acquired or calculated from the time when the ground surface is imaged by the satellite 2, and the library is created by associating this angle of the sun. However, as already described, the dependency on the sun angle is not essential, including at the beginning of operation of the system 1.
[0039] Next, the drone 3 is flown based on a predetermined flight plan in the observation target area where it is to be determined whether the crops are in a specific state, and an image is taken by the second multispectral camera mounted on the drone 3. As described above, the imaging range of the second multispectral camera mounted on the drone 3 is relatively narrow, so it is preferable to fly a plurality of drones 3 at the same time and capture images in parallel with the second multispectral camera.
[0040] The flight plan is preferably created in advance by an operator of the information processing device 4. In this case, it is preferable for the operator to grasp the topography of the area to be observed in advance and create a flight plan in which the drone 3 is flown with a focus on areas where crops are expected to be in a specific state. In this case, a mode in which the information processing device 4 recommends areas where crops are expected to be in a specific state is also possible. Examples of areas where crops are expected to be in a specific state include areas along rivers when crops are affected by humidity, areas where fog is expected to occur for long periods of time, and areas within a specific altitude range when crops are expected to be in a specific state within this altitude range.
[0041] In system 1 of the present embodiment, multiple wavelength bands that can clearly determine whether or not a specific state is present are selected in advance, and multiple cameras (preferably three or more, more preferably four) that can capture the reflected light of each of these wavelength bands are prepared, and a multispectral camera is formed from these multiple cameras.
[0042] The spectral image obtained by capturing an image of the observation target area by such a second multispectral camera is input to the information processing device 4, and compared with the library to identify the area where the crop is in a specific state. In FIG. 1, the area indicated by hatching is identified as the area where the crop is in a specific state. In this case, it is preferable that the information processing device 4 connects the individual spectral images captured by the second multispectral camera mounted on the drone 3 to generate one spectral image, and uses this spectral image to identify the area where the crop is in a specific state. This is because the drone 3 flies at a lower altitude than the satellite 2, and when the observation target area is wide, it is difficult to identify the entire observation target area with one spectral image. The meaning of connecting the spectral images is similarly applied to the spectral images obtained from the first multispectral camera mounted on the satellite 2, so as already explained, it is preferable to appropriately connect the spectral images obtained from the first multispectral camera. Preferably, when identifying the area where the crop is in a specific state, the angle of the sun in the area at the time of capturing the spectral image is acquired and calculated, and the area where the crop is in a specific state is identified taking into account this angle of the sun.
[0043] <Basic configuration of System 1> The basic configuration of a system 1 to which a method according to an embodiment is applied will be described with reference to FIG.
[0044] The system 1 of this embodiment has an information processing device 10, a drone 20, and a satellite 30, which are connected via a network N. The hardware configuration of the information processing device 10 is shown in Fig. 11, the hardware configuration of the drone 20 in Fig. 5, and the hardware configuration of the satellite 30 in Fig. 8. The drone 20 and the satellite 30 are equipped with information processing devices.
[0045] The information processing device is configured by a computer having an arithmetic unit and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later.
[0046] The network N is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a predetermined access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes one directly connected by a USB (Universal Serial Bus) cable, etc.
[0047] The information processing device 10 receives a spectral image captured by the satellite 30 and creates a library based on this spectral image. Then, the information processing device 10 receives a spectral image captured by the drone 20 and identifies an area where crops are in a particular state in the observation target area based on this spectral image and the library.
[0048] The configuration and operation of each device will be described below.
[0049] <Drone 20 hardware configuration> The drone 20 of this embodiment is configured to be capable of stationary flight, and is equipped with a spectral camera control system 23 having a spectral camera control device 21 and a spectral camera 22 controlled by the spectral camera control device 21, as shown in Figures 3 and 4. Each component will be described in detail below.
[0050] The drone 20 is an aircraft having a function of flying stationary in the air, i.e., a so-called hovering function, and in this embodiment, it is a multicopter type drone having multiple rotors. The drone 20 in this embodiment has a function of flying autonomously along a pre-specified flight path and a function of flying by remote control from a communication device or the like. Furthermore, although not shown in Figs. 3 and 4, the drone 20 has a GPS (Global Positioning System) receiver for detecting the position (latitude and longitude) and altitude of the drone during flight, and an attitude sensor for detecting the attitude of the drone during flight.
[0051] As shown in FIG. 5 , the drone 20 is equipped with a spectral camera control system 23 having a spectral camera control device 21 and a spectral camera 22, which is a second multispectral camera controlled by the spectral camera control device 21.
[0052] As shown in Figure 5, the spectral camera control system 23 mainly includes a spectral camera 22, an attitude position detector 24 that detects attitude and position information of the spectral camera 22, a spectral camera control device 21 that controls the spectral camera 22, and a battery 25 that supplies power to each of these devices.
[0053] As shown in FIG. 5, the spectral camera 22 is mounted on the drone 20 and faces vertically downward so that the ground surface is the subject of imaging while the drone 20 is in stationary flight.
[0054] The spectral camera 22 mounted on the drone 20 of this embodiment has an image sensor capable of detecting reflected light of wavelength bands corresponding to each of a combination of a plurality of wavelength bands that can detect that the crop is in a specific state as a result of library generation described later. In detail, the spectral camera 22 has an image sensor and a plurality of filters that can pass only light of a plurality of wavelength bands, and when the spectral camera 22 captures an image of the earth's surface, the plurality of filters are appropriately switched (replaced) to allow only reflected light of each wavelength band to reach the image sensor, and the filters are continuously switched to obtain an imaging result based on reflected light of a wavelength band corresponding to each of the combinations of a plurality of wavelength bands.
[0055] In this case, before the drone 20 captures an image of the observation target area, it is determined in advance which particular state the crop is to be detected in, and the spectral camera 22 consisting of a combination of filters determined based on the determination result is mounted on the drone 20. However, the spectral camera 22 may be configured by mounting a plurality of cameras capable of capturing only light of a single wavelength band. In this way, the spectral camera 22 generates a spectral image 2104 based on reflected light of a predetermined specific wavelength band, and is configured not to switch wavelength bands while capturing an image of the observation target area by the drone 20. Note that the above description does not intend to exclude the spectral camera 22 being configured to be able to arbitrarily switch between a plurality of wavelength bands, similar to the spectral camera 32 mounted on the satellite 30.
[0056] The image sensor captures a spectral image by a snapshot method. In this embodiment, the image sensor 222 is a two-dimensional image sensor that can capture images within the field of view at the same time, such as a CMOS image sensor or a CCD image sensor. The image sensor 222 captures images based on an imaging command signal transmitted from the spectral camera control device 21.
[0057] The attitude position detector 24 is a device that detects the attitude and position state of the spectral camera 22. In this embodiment, the attitude position detector 24 has a GPS receiver 240 that detects position information and altitude information of the spectral camera 22, and an attitude sensor 241 that detects attitude information of the spectral camera 22.
[0058] The GPS receiver 240 acquires current position information and altitude information by capturing the positions of multiple artificial satellites. In this embodiment, the GPS receiver 240 acquires longitude information and latitude information as position information, and altitude information as altitude information. Note that the position information and altitude information are not limited to those acquired by the GPS receiver 240, and may be acquired by other methods. For example, a reference point may be set, and distance and altitude information from the reference point may be acquired by a distance measuring device using reflection of laser light or sound.
[0059] The attitude sensor 241 detects attitude information such as the tilt angle, angular velocity, and acceleration of the spectroscopic camera 22. Although not shown, the attitude sensor 241 in this embodiment is composed of a gyro sensor that utilizes gyro characteristics and an acceleration sensor, and is adapted to obtain the tilt angle, angular velocity, and acceleration in three axial directions as attitude information.
[0060] In this embodiment, the position information and the attitude information are acquired from the attitude position detector 24 provided as the spectral camera control system 23, but the present invention is not limited to this configuration. For example, the position information and the attitude information may be acquired from a GPS receiver and an attitude sensor that the drone 20 already has.
[0061] <Functional configuration of the spectrum camera control device 21> 6 shows the functional configuration realized by the spectral camera control device 21 of the drone 20. The spectral camera control device 21 includes a storage unit 210, a control unit 211, and a communication unit 212. The communication unit 212 is configured by a communication IF (not shown) of the spectral camera control device 21, the storage unit 210 is configured by a main storage device and an auxiliary storage device (not shown) of the spectral camera control device 21, and the control unit 211 is mainly configured by a processor (not shown) of the spectral camera control device 21.
[0062] The communication unit 212 communicates with the information processing device 10 and the like via a network N (not shown).
[0063] <Configuration of the storage unit 210 of the spectroscopic camera control device 21> The storage unit 210 of the spectroscopic camera control device 21 includes a flight plan DB (DataBase) 2101 , a second imaging condition DB 2102 , an image DB 2103 , and a spectroscopic image 2104 .
[0064] Of these flight plan DB 2101, etc., all except for the spectral image 2104 are databases. The database referred to here refers to a relational database, which is used to manage data sets called tables in a tabular format structurally defined by rows and columns by associating them with each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated.
[0065] Usually, a column that serves as a primary key for uniquely identifying a record is set in each table, but setting a primary key to a column is not essential. The control unit 211 can cause the processor to add, delete, or update records in a specific table stored in the storage unit 210 according to various programs.
[0066] The flight plan DB2101 is a database in which a flight plan for flying the drone 20 along a predetermined route is stored when the drone 20 is flown in the air, the second imaging condition DB2102 is a database related to conditions when an observation target area is imaged by the spectral camera 22 mounted on the drone 20, and the image DB2103 is a database for managing a spectral image 2104 obtained as a result of imaging by the spectral camera 22 according to the second imaging condition DB2102, and the spectral image 2104 is a spectral image obtained as a result of imaging by the spectral camera 22 according to the second imaging condition DB2102. Details of the flight plan DB2101, the second imaging condition DB2102, and the image DB2103 will be described later. <Configuration of the control unit 211 of the spectroscopic camera control device 21>
[0067] The control unit 211 of the spectroscopic camera control device 21 includes a reception control unit 2110, a transmission control unit 2111, a flight control unit 2112, an attitude position information acquisition unit 2113, and an imaging control unit 2114. The control unit 211 executes an application program 2100 stored in the storage unit 210, thereby realizing functional units such as the reception control unit 2110.
[0068] The reception control unit 2110 controls the process in which the spectroscopic camera control device 21 receives signals from an external device in accordance with a communication protocol.
[0069] The transmission control unit 2111 controls the process in which the spectroscopic camera control device 21 transmits a signal to an external device in accordance with a communication protocol.
[0070] The flight control unit 2112 controls the flight of the drone 20 toward the target point specified by the flight plan DB 2101 based on the geographical information of the target point specified by the flight plan DB 2101 and the attitude position information acquired by the attitude position information acquisition unit 2113.
[0071] The attitude position information acquisition unit 2113 acquires the position information, altitude information, and attitude information of the drone 20 detected by the attitude position detector 24, and provides the acquired results to the flight control unit 2112 and the imaging control unit 2114.
[0072] The imaging control unit 2114 refers to the second imaging condition DB2102, images the area to be observed using the spectral camera 22 at the imaging position specified by the second imaging condition DB2102, stores the captured spectral image 2104 in the memory unit 210, and also stores the conditions under which the spectral image 2104 was captured in the image DB2103.
[0073] 7 conceptually illustrates a state in which the drone 20 of this embodiment captures an image of an observation target area to obtain a spectral image. The area captured by the spectral camera 22 at one time is, for example, a rectangular (square) area with one side of about 150 m, and the altitude of the drone 20 at this time is about 150 m.
[0074] <Hardware configuration of Satellite 30> The satellite 30 of this embodiment revolves around the earth at a substantially constant speed on an orbit above the earth. As shown in FIG. 8, the satellite 30 is equipped with a spectral camera control system 33 having a spectral camera control device 31 and a spectral camera 32, which is a first multispectral camera controlled by the spectral camera control device 31. The configurations of the spectral camera control device 31, the spectral camera 32, and the spectral camera control system 33 are similar to those of the spectral camera control device 21, the spectral camera 22, and the spectral camera control system 23 of the drone 20. Therefore, the description of the substantially identical configurations will be omitted, and the description will be focused on the main differences.
[0075] The spectral camera control system 33 mainly comprises a spectral camera 32 equipped with a liquid crystal tunable filter 34, an attitude position detector 35 that detects attitude and position information of the spectral camera 32, a liquid crystal tunable filter control circuit 36 that controls the liquid crystal tunable filter 34 of the spectral camera 32, a spectral camera control device 31 that controls the spectral camera 32, and a battery 37 that supplies power to each of these devices.
[0076] The spectral camera 32 is used to capture spectral images using a snapshot method, and mainly comprises a lens group 320, a depolarizing plate 321 for converting polarized light into unpolarized light, a liquid crystal wavelength-tunable filter 34 that can arbitrarily select the transmitted wavelength, and an image sensor 322 for capturing a two-dimensional spectral image.
[0077] The lens group 320 utilizes light refraction to transmit light from the image capture target to the liquid crystal tunable filter 34 and collect the transmitted light on the image sensor 322. In this embodiment, the lens group 320 is composed of a light entrance lens 320a that collects the light from the image capture target and causes it to enter the liquid crystal tunable filter 34, and a condenser lens 320b that collects only the light of the transmitted wavelength after passing through the liquid crystal tunable filter 34 on the image sensor 322. The type and number of lenses are not particularly limited, and may be appropriately selected depending on the performance of the spectral camera 32, etc.
[0078] The depolarization plate 321 is for depolarizing the light to make it unpolarized. In this embodiment, the depolarization plate 321 is provided on the light entrance side of the liquid crystal tunable filter 34, and is adapted to depolarize the light before it passes through the liquid crystal tunable filter 34, thereby reducing the polarization characteristics.
[0079] The liquid crystal tunable filter 34 is an optical filter that can select any transmission wavelength within a predetermined wavelength range. Although not shown, the liquid crystal tunable filter 34 is configured by alternately stacking multiple plate-shaped liquid crystal elements and plate-shaped polarizing elements. The orientation state of each liquid crystal element is independently controlled by an applied voltage supplied from the liquid crystal tunable filter control circuit 36. Therefore, the liquid crystal tunable filter 34 is capable of transmitting light of any wavelength by combining the orientation state of the liquid crystal elements with the polarizing elements.
[0080] In this embodiment, the width of the transmission wavelength of the liquid crystal tunable filter 34 is approximately 20 nm or less, the transmission center wavelength can be set in 1 nm increments, and the wavelength switching time is approximately 10 ms to several hundreds of ms.
[0081] The image sensor 322 captures a spectral image by a snapshot method. In this embodiment, the image sensor 322 is a two-dimensional image sensor that can capture images within a field of view at the same time, such as a CMOS image sensor or a CCD image sensor. The image sensor 322 captures images based on an imaging command signal transmitted from the spectral camera control device 31.
[0082] The liquid crystal tunable filter control circuit 36 controls the liquid crystal tunable filter 34. In this embodiment, when the liquid crystal tunable filter control circuit 36 receives a wavelength specifying signal transmitted from the spectral camera control device 31, it supplies an applied voltage corresponding to the wavelength specifying signal to the liquid crystal element of the liquid crystal tunable filter 34. The wavelength specifying signal includes information on a transmission wavelength to be transmitted by the liquid crystal tunable filter 34, and the liquid crystal tunable filter control circuit 36 determines which liquid crystal element the applied voltage should be supplied to based on the information on the transmission wavelength, and supplies the applied voltage to the specified liquid crystal element.
[0083] In this embodiment, the liquid crystal wavelength tunable filter control circuit 36 is configured independently from other components such as the spectral camera control device 31, but is not limited to this and may be provided in, for example, the spectral camera control device 31 or the spectral camera 32.
[0084] <Functional configuration of the spectrum camera control device 31> 9 shows the functional configuration realized by the spectroscopic camera control device 31 of the satellite 30. The spectroscopic camera control device 31 includes a memory unit 310, a control unit 311, and a communication unit 312. The communication unit 312 is configured by a communication IF (not shown) of the spectroscopic camera control device 31, the memory unit 310 is configured by a main memory device and an auxiliary memory device (not shown) of the spectroscopic camera control device 31, and the control unit 311 is mainly configured by a processor (not shown) of the spectroscopic camera control device 31.
[0085] The communication unit 312 communicates with the information processing device 10 and the like via a network N (not shown).
[0086] <Configuration of the storage unit 310 of the spectroscopic camera control device 31> The storage unit 310 of the spectroscopic camera control device 31 includes a first imaging condition DB (DataBase) 3101 , an image DB 3102 , and a spectroscopic image 3103 .
[0087] Of the first imaging condition DB 3101 etc., all except for the spectral image 3103 are databases. The database referred to here refers to a relational database, which is used to manage data sets called tables in a tabular format structurally defined by rows and columns in association with each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated.
[0088] Usually, a column that serves as a primary key for uniquely identifying a record is set in each table, but setting a primary key to a column is not essential. The control unit 311 can cause the processor to add, delete, or update records in a specific table stored in the storage unit 310 according to various programs.
[0089] The first imaging condition DB3101 is a database related to conditions when the earth's surface is imaged by the spectral camera 32 mounted on the satellite 30, the image DB3102 is a database for managing spectral images obtained as a result of imaging by the spectral camera 32 according to the first imaging condition DB3101, and the spectral images 3103 are spectral images obtained as a result of imaging by the spectral camera 32 according to the first imaging condition DB3101. Details of the first imaging condition DB3101 and the image DB3102 will be described later. <Configuration of the control unit 311 of the spectroscopic camera control device 31>
[0090] The control unit 311 of the spectroscopic camera control device 31 includes a reception control unit 3110, a transmission control unit 3111, an imaging determination unit 3112, an attitude position information acquisition unit 3113, an imaging control unit 3114, and a wavelength setting unit 3115. The control unit 311 executes an application program 3100 stored in the storage unit 310 to realize functional units such as the reception control unit 3110.
[0091] The reception control unit 3110 controls the process in which the spectroscopic camera control device 31 receives a signal from an external device in accordance with a communication protocol.
[0092] The transmission control unit 3111 controls the process in which the spectroscopic camera control device 31 transmits a signal to an external device in accordance with a communication protocol.
[0093] The imaging determination unit 3112 determines whether or not to image the earth's surface using the spectroscopic camera 32 mounted on the satellite 30 based on the imaging start time and imaging end time stored in the first imaging condition DB 3101, and sends the determination result to the imaging control unit 3114.
[0094] The attitude position information acquisition unit 3113 acquires the position information, altitude information, and attitude information of the satellite 30 detected by the attitude position detector 35 , and provides the acquired results to the imaging control unit 3114 .
[0095] The imaging control unit 3114 receives the determination result from the imaging determination unit 3112, and if the determination result is to perform imaging, images the ground surface using the spectral camera 32, stores the captured spectral image 3103 in the memory unit 310, and also stores the conditions under which the spectral image 3103 was captured in the image DB 3102.
[0096] 10 conceptually illustrates how the satellite 30 of this embodiment captures an image of an observation target area to obtain a spectral image. The area captured by the spectral camera 32 at one time is, for example, a rectangular (square) area with one side measuring about 2 to 10 km, and the altitude of the satellite 30 at this time is about 500 km.
[0097] <Hardware configuration of information processing device 10> 11 is a block diagram showing a basic hardware configuration of an information processing device 10. The information processing device 10 has at least a processor 101, a main memory device 102, an auxiliary memory device 103, a communication IF (Interface) 104, an input IF 105, and an output IF 106. These are electrically connected to each other by a communication bus 107. In addition, an input device 110 is connected to the information processing device 10 via the input IF 105, and an output device 111 is connected to the information processing device 10 via the output IF 106.
[0098] The processor 101 is hardware for executing an instruction set written in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.
[0099] The main storage device 102 is for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).
[0100] The auxiliary storage device 103 is a storage device for saving data and programs, such as a flash memory, a solid state drive (SSD), a hard disc drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.
[0101] The communication IF 104 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard.
[0102] The input IF 105 functions as an interface with an input device 110 for receiving an input operation from an operator of the information processing device 10. The output IF 106 functions as an interface with an output device 111 for presenting information to the operator. The input device 110 is an input device (e.g., a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.) for receiving an input operation from the operator. The output device 111 is an output device (a display, a speaker, etc.) for presenting information to the operator.
[0103] The information processing device 10 can be virtually realized by distributing all or part of each hardware configuration among multiple computers and connecting them via a network. In this way, the information processing device 10 is a concept that includes not only a computer housed in a single housing or case, but also a virtualized computer system.
[0104] <Functional configuration of information processing device 10> 12 shows a functional configuration realized by the hardware configuration of the information processing device 10. The information processing device 10 includes a storage unit 120, a control unit 130, and a communication unit 140. The communication unit 140 is configured by a communication IF 104, the storage unit 120 is configured by a main storage unit 102 and an auxiliary storage unit 103 of the information processing device 10, and the control unit 130 is mainly configured by a processor 101 of the information processing device 10.
[0105] The communication unit 140 communicates with drones 20 and the like via the network N.
[0106] <Configuration of storage unit 120 of information processing device 10> The memory unit 120 of the information processing device 10 has a flight plan DB (DataBase) 122, an image DB 123, a wide area image DB 124, teacher data 125, a learning model 126, a spectral image 127, and a wide area spectral image 128.
[0107] Of these flight plan DB 122 and the like, the flight plan DB 122, image DB 123, and wide-area image DB 124 are databases. The database referred to here refers to a relational database, which is used to manage data sets called tables in a tabular format structurally defined by rows and columns in association with each other. In a database, a table is called a table, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables can be set and associated.
[0108] Usually, a column that serves as a primary key for uniquely identifying a record is set in each table, but setting a primary key to a column is not essential. The control unit 130 can cause the processor 101 to add, delete, or update records in a specific table stored in the storage unit 120 according to various programs.
[0109] The flight plan DB122 is similar to the flight plan DB2101 stored in the storage unit 210 of the drone 20. The image DB123 is similar to the image DB2103 stored in the storage unit 210 of the drone 20 and the image DB3102 stored in the storage unit 310 of the satellite 30. In the information processing device 10 of this embodiment, the image DB123, which is a combination of the image DB2103 and the image DB3102, is stored in the storage unit 120. The wide-area image DB124 is a database for managing the wide-area spectral image 128 generated by the image synthesis unit 136 of the control unit 130 based on the spectral image 127. <Configuration of the control unit 130 of the information processing device 10>
[0110] The control unit 130 of the information processing device 10 includes a reception control unit 131, a transmission control unit 132, a learning model generation unit 133, a flight plan creation unit 134, an aircraft control unit 135, an image synthesis unit 136, an area identification unit 137, and a type identification unit 138. The control unit 130 executes an application program 121 stored in the storage unit 120, thereby realizing these functional units such as the reception control unit 131.
[0111] The reception control unit 131 controls the process in which the information processing device 10 receives a signal from an external device in accordance with a communication protocol.
[0112] The transmission control unit 132 controls the process in which the information processing device 10 transmits a signal to an external device in accordance with a communication protocol.
[0113] The learning model generation unit 133 generates teacher data 125 based on the spectral image 127 captured by the spectral camera 32 mounted on the satellite 30 and the image DB 123 transmitted from the satellite 30, as well as the spectral camera 22 mounted on the drone 20 and the image DB 123 transmitted from the drone 20, and generates a learning model 126 based on this teacher data 125.
[0114] The following describes in detail the operation of the learning model generation unit 133 when using the spectral image 127 captured by the spectral camera 32 mounted on the satellite 30. The operation is similar when using the spectral image 127 captured by the spectral camera 22 mounted on the drone 20.
[0115] The spectral camera 32 mounted on the satellite 30 captures an image of the same earth surface area for each wavelength band obtained by subdividing a wide range of wavelength bands, and obtains a large number of spectral images 3103 (spectral images 127) of the same area with different wavelength bands. The learning model generation unit 133 presents the multiple spectral images 127 of the same area with different wavelength bands to the operator of the information processing device 10, and allows the operator to specify the spectral image 127 of the wavelength band that can clearly distinguish the area in which the crop is in a specific state in this area. Next, the learning model generation unit 133 allows the operator to specify the specific state of the crop that can be specified by the spectral image 127 of the specified wavelength band, and associates the spectral image 127 of the specified wavelength band with the specific state of the specified crop. Then, the learning model generation unit 133 stores the specified wavelength band, the spectral image 127 of the specified wavelength band, the specific state of the specified crop, and the associations between them in the storage unit 120 as the teacher data 125.
[0116] For example, as shown in FIG. 23, for an original image (in this embodiment, multiple spectral images 127 can be obtained for the same area, but only one image is shown for the sake of simplicity) captured on the same ground surface, multiple spectral images 127 of multiple wavelength bands (three types of wavelengths α, β, and γ are shown in FIG. 23, but the number of wavelength bands is not limited) can be obtained. Among these spectral images 127, a shading distribution different from that of the other wavelengths α and γ may occur for a certain wavelength (wavelength β in FIG. 23). Since the spectral image 127 is for a specific wavelength band, it is assumed that the image will be displayed in a black and white shading pattern (i.e., grayscale display) when displayed on a display or the like. The operator of the information processing device 10 knows in advance what specific state the crops are in for the same ground surface area (for example, by a field survey, etc.), and inputs the association between the shading distribution and the specific state of the crops by looking at a specific shading distribution (shown by hatching in FIG. 23) for wavelength β that is not seen for the other wavelengths α and γ.
[0117] At this time, the learning model generation unit 133 refers to the image DB 123 and calculates the angle between the earth's surface and the sun when the spectral image of the specified wavelength band was captured. This is because the spectrum of reflected light from the crop differs depending on the angle of the sun. The learning model generation unit 133 also stores the angle of the sun in the storage unit 120 as teacher data 125. Alternatively, the learning model generation unit 133 calculates a correction formula (or correction value) for correcting the spectral image to a spectral image of a spectrum captured at a solar angle at a specified time based on the calculated solar angle, generates teacher data 125 using the spectral image corrected by this correction formula or the like, and stores it in the storage unit 120. At this time, the correction formula itself may be stored in the learning model 126 described later, or the correction formula may be stored separately in the storage unit 120. However, correction based on the solar angle is not essential.
[0118] Next, the learning model generation unit 133 generates a learning model 126 in artificial intelligence technology based on the teacher data 125 generated by the above-mentioned procedure. Since known methods for generating the learning model 126 and types of the learning model 126 can be suitably applied, further explanation will be omitted here.
[0119] The learning model generation unit 133 then repeats the above-mentioned procedure for generating the teacher data 125 to update the teacher data 125, and re-learns the learning model 126 after updating the teacher data 125 or at predetermined time intervals. This improves the accuracy with which the type identification unit 138, which will be described later, identifies that the crop is in a specific state.
[0120] The above-mentioned updating of the teacher data 125 by the learning model generating unit 133 and the re-learning of the learning model 126 are preferable operations, particularly when the spectral image 127 at a limited solar angle is acquired at the beginning of the operation of the system 1, and the teacher data 125 and the learning model 126 are generated based on this spectral image 127. In other words, the teacher data 125 and the learning model 126 may be generated based on the spectral image 127 at a limited solar angle at the beginning, and then the spectral images obtained from the spectral camera 32 mounted on the satellite 30 and / or the spectral images obtained from the spectral camera 22 mounted on the drone 20 may be acquired at various solar angles, the teacher data 125 including the influence of the solar angle may be generated, and the learning model 126 may be re-learned based on this teacher data 125. This allows the start of operation of the system 1 to be accelerated, and the accuracy of the identification result by the type identification unit 138 to be progressively improved. In addition, the identification result by the type identification unit 138 described later may be compared with the actual identified state of the crop, and the actual identified state of the crop may be fed back to create / modify the teacher data 125 and re-learn the learning model 126. This also makes it possible to progressively improve the accuracy of the identification result by the type identification unit 138.
[0121] At this time, it is preferable that the learning model generating unit 133 acquires topographical data of the ground surface captured by the spectral camera 32 of the satellite 30 and / or the spectral camera 22 of the drone 20, identifies an area where the crop is estimated to be in a specific state based on the topographical data, extracts a spectral image 127 obtained by capturing the area by the spectral cameras 22, 32, and presents it to the operator. Whether the crop is in a specific state or not often depends on the topography. As an example, it is known that if the humidity of the place where the crop is grown is high, the crop is more likely to be susceptible to disease. Therefore, the learning model generating unit 133 acquires topographical data of the ground surface, estimates the environment (sunlight, humidity, wind direction, etc.) of the place where the crop is grown based on the topographical data, and presents the spectral image 127 obtained by capturing the place where the environment is specific to the operator. The topographical data may be stored in advance in the storage unit 120 of the information processing device 10, or may be acquired from an external service.
[0122] In addition, without using the learning model generation unit 133, the teacher data 125, and the learning model 126, it is also possible to determine whether or not the crop is in a specific state based only on the identified wavelength band, the spectral image 127 of the identified wavelength band, the identified specific state of the crop, the angle of the sun when the spectral image 127 was captured, and the correlation between them.
[0123] The flight plan creation unit 134 creates a flight plan for flying the drone 20 over an area to be observed and capturing an image of the area to be observed by the spectral camera 22 mounted on the drone 20, and stores the flight plan in the storage unit 120 as a flight plan DB 122. Then, the flight plan creation unit 134 transmits the created flight plan DB 122 to the drone 20. Note that when the flight of the drone 20 is controlled (the drone 20 is piloted) by the aircraft control unit 135 described later, the flight plan creation unit 134 and the flight plan DB 122 may not be provided.
[0124] In this case, it is preferable that the flight plan creation unit 134 creates a flight plan in which multiple drones 20 are flown simultaneously over the area to be observed and the area to be observed is imaged in parallel using the spectral cameras 22 mounted on these multiple drones 20.
[0125] In addition, it is preferable that the flight plan creation unit 134 acquires topographical data of the area to be observed and creates a flight plan based on this topographical data. As described above, whether a crop is in a specific state often depends on the topography. As an example, it is known that if the humidity of the area where the crop is grown is high, the crop is more likely to become ill. Therefore, the flight plan creation unit 134 acquires topographical data of the area to be observed, estimates the environment (sunshine, humidity, wind direction, etc.) of the crop growing place based on this topographical data, flies the drone 20 focusing on the place where this environment is specific, and acquires a spectral image by the spectral camera 22. The topographical data may be stored in advance in the storage unit 120 of the information processing device 10, or may be acquired from an external service.
[0126] The flight plan creation by the flight plan creation unit 134 may be performed by the operator of the information processing device 10 individually specifying the flight target position of the drone 20, or the operator may identify only the area to be observed and the flight plan creation unit 134 may generate the flight target position based on the flight speed of the drone 20 and the speed at which the spectral image is acquired by the spectral camera 22.
[0127] The flying object control unit 135 controls the flight of the drone 20 based on the flight plan DB 122. However, if the drone 20 can fly autonomously based on the flight plan DB 2101 stored in its own memory unit 210, the flying object control unit 135 does not need to be provided. Alternatively, if the drone 20 is operated by the information processing device 10, the flying object control unit 135 operates the drone 20 based on a control signal input by an operator via a remote control, which is an example of an input device 110 provided in the information processing device 10.
[0128] The image synthesis unit 136 stitches together the spectral images 2104 and 127 captured by the spectral camera 22 of the drone 20 to generate a wide-spectrum image 128 that preferably captures the entire observation target area. This is because the imaging range of the spectral camera 22 of the drone 20 is often narrower than the observation target area in which it is necessary to determine whether the crop is in a specific state, so it is preferable to generate a wide-spectrum image 128 that covers a wide range in order to collectively identify the specific state of the crop. Then, the image synthesis unit 136 stores the generated wide-spectrum image 128 in the storage unit 120 and updates the wide-spectrum image DB 124. The image synthesis unit 136 also stitches together the spectral images 3103 and 127 captured by the spectral camera 32 of the satellite 30 to generate a wide-spectrum image 128 that preferably captures the ground surface over a wide range.
[0129] In this case, the image synthesis unit 136 preferably generates the broad spectrum image 128 based on a known method for generating an orthoimage. An orthoimage is an image obtained by converting an aerial photograph into an image that is displayed at the correct size and position without tilt as seen from directly above, like a map, by eliminating positional deviation of the image on the photograph (hereinafter referred to as "orthotransformation"). To perform orthotransformation on an aerial photograph, it is necessary to make the position on the aerial photograph correspond to the horizontal position on the ground. This orthotransformation is performed using a digital elevation model (elevation data) that represents the three-dimensional shape of the earth's surface. Since the method for generating an orthoimage is known, further explanation will be omitted here.
[0130] 22 is a diagram showing an example of image synthesis of the spectral images 2104 and 127 by the image synthesis unit 136. Spectral images 127 obtained by capturing an image of an area 2200 (a square in the figure, but not limited to this) by one drone 20 are arranged in the horizontal direction in the figure with a predetermined overlapping area, and these spectral images 127 are synthesized to generate a wide-band spectral image 128. This operation is also performed on a spectral image 127 obtained by capturing an image of an area 2201 by another drone 20, to obtain a wide-band spectral image 128 that encompasses the area to be observed.
[0131] The area identification unit 137 receives, from the operator of the information processing device 10, the identification of an area for the type identification unit 138 to identify whether or not the crop is in a specific state, by referring to the wide-band spectrum image 128 created by the image synthesis unit 136.
[0132] The type identification unit 138 identifies whether or not the crop is in a specific state for the area identified by the area identification unit 137, and if so, what state it is in, based on the broad spectrum image 128 and the learning model 126. At this time, the type identification unit 138 may accept an input of a specific state of the crop to be identified based on an instruction input from an operator of the information processing device 10, and identify whether or not the crop is in a specific specific state for the area identified by the area identification unit 137. At this time, the type identification unit 138 calculates the angle between the earth's surface and the sun when the spectral image 127, which is the basis of the broad spectrum image 128, was captured, and corrects the spectrum using the learning model 126 or a solar angle correction formula stored in the storage unit 120 so that the broad spectrum image 128 is captured at a predetermined time.
[0133] Next, the type identification unit 138 displays the identification result on a display, which is an example of the output device 111. There are no particular limitations on the display mode of the type identification unit 138, but one example is a mode in which map data of the observation target area is displayed by referring to a map database, and an area where crops are in a specific state is superimposed on this map data.
[0134] In addition, the type identification unit 138 may receive a time axis designation input from the operator of the information processing device 10, and display the spread of the area on the time axis where the crop is in a specific state. Furthermore, the type identification unit 138 may cause the display to display the fact that the crop is in a specific state, the crop, and an estimated result of the cost required to restore the specific state (for example, cutting down the crop in the specific state, spraying pesticides, etc.), and may calculate and display the estimated amount of damage on the display. <Data Structure> FIG. 13 is a diagram showing the data structure of the first imaging condition DB 3101 stored in the storage unit 310 of the satellite 30. As shown in FIG.
[0135] The first imaging condition DB 3101 is a table having columns of imaging start time, imaging end time, and wavelength condition, using an imaging condition ID as a key for identifying the imaging conditions of the spectroscopic camera 32 of the satellite 30.
[0136] "Imaging condition ID" is information for identifying the imaging conditions of the spectral camera 32 of the satellite 30. "Imaging start time" is information relating to the time when imaging by the spectral camera 32 of the satellite 30 starts. "Imaging end time" is information relating to the time when imaging by the spectral camera 32 of the satellite 30 ends. "Wavelength condition" is information relating to the wavelength condition when imaging by the spectral camera 32 of the satellite 30 is performed. The spectral camera 32 mounted on the satellite 30 is capable of imaging in a plurality of wavelength bands (and therefore capable of generating spectral images in a plurality of wavelength bands), and in the example shown in FIG. 13, information relating to the range of the wavelength band and the wavelength width for changing the wavelength band is stored as the wavelength condition.
[0137] Each column in the first imaging condition DB 3101 is generated by an information processing device including the information processing device 10, and is stored in the storage unit 310 by being transmitted to the satellite 30.
[0138] FIG. 14 is a diagram showing the data structures of image DBs 2103, 3102, and 123 stored in the memory unit 210 of the drone 20, the memory unit 310 of the satellite 30, and the memory unit 120 of the information processing device 10, respectively.
[0139] Image DB2103, 3102, 123 are tables having columns for image file name, capture time, location information, altitude information, municipal information, and wavelength, with an image ID as a key to identify spectral images captured by the spectral cameras 22, 32 of the drone 20 and satellite 30.
[0140] The "image ID" is information for identifying the spectral images captured by the spectral cameras 22 and 32 of the drone 20 and the satellite 30. The "image file name" is information indicating the file names of the spectral images 2104, 3103, and 127 identified by the image ID and stored in the storage unit 210 of the drone 20, the storage unit 310 of the satellite 30, and the storage unit 120 of the information processing device 10. The "imaging time" is information indicating the time when the spectral images 2104, 3103, and 127 identified by the image ID were captured. The "position information" is the position information of the drone 20 and the satellite 30 when the spectral images 2104, 3103, and 127 identified by the image ID were captured. The "altitude information" is the altitude information of the drone 20 and the satellite 30 when the spectral images 2104, 3103, and 127 identified by the image ID were captured. "Attitude information" is attitude information of the drone 20 and the satellite 30 when the spectral images 2104, 3103, and 127 identified by the image ID were captured. "Wavelength" is information indicating the wavelength bands set in the spectral cameras 22 and 32 of the drone 20 and the satellite 30 when the spectral images 2104, 3103, and 127 identified by the image ID were captured.
[0141] Each column in the image DB 2103, 3102, 123 is generated by the spectral camera control devices 21, 31 of the drone 20 and the satellite 30 when the spectral images 2104, 3103 are generated by the spectral cameras 22, 32 of the drone 20 and the satellite 30.
[0142] FIG. 15 is a diagram showing the data structure of the second imaging condition DB 2102 stored in the memory unit 210 of the drone 20.
[0143] The second imaging condition DB2102 is a table having columns of imaging position, imaging altitude, and imaging attitude, with an imaging condition ID as a key for identifying the imaging conditions of the spectroscopic camera 22 of the drone 20.
[0144] The “imaging condition ID” is information for identifying the imaging conditions of the spectral camera 22 of the drone 20. The “imaging position” is position information of the drone 20. The “imaging altitude” is altitude information of the drone 20. The “imaging attitude” is attitude information of the drone 20.
[0145] Each column in the second imaging condition DB 2102 is generated by an information processing device including the information processing device 10, and is stored in the storage unit 210 by transmitting it to the drone 20.
[0146] FIG. 16 is a diagram showing the data structures of the flight plan DBs 122 and 2101 stored in the memory unit 120 of the information processing device 10 and the memory unit 210 of the drone 20.
[0147] The flight plan DB122, 2101 is a table having columns of elapsed time, flight position, and flight attitude, with a flight plan ID as a key for identifying the flight plan of the drone 20.
[0148] The "flight plan ID" is information for identifying the flight plan of the drone 20, more specifically, the destination of the drone 20. The "elapsed time" is information indicating the elapsed time from the start of the flight of the drone 20 by which the drone 20 should reach the destination of the drone 20 identified by the flight plan ID. The "flight position" is target position information that the drone 20 should reach at the destination of the drone 20 identified by the flight plan ID. The "flight altitude" is target altitude information that the drone 20 should reach at the destination of the drone 20 identified by the flight plan ID. The "flight attitude" is target attitude information that the drone 20 should reach at the destination of the drone 20 identified by the flight plan ID.
[0149] Each column in the flight plan DB 122, 2101 is generated by the flight plan creation unit 134 of the control unit 130 of the information processing device 10, and is stored in the memory unit 120 and sent to the drone 20. The spectroscopic camera control device 21 of the drone 20 stores the sent flight plan DB 122, 2101 in the memory unit.
[0150] FIG. 17 is a diagram showing the data structure of the wide area image DB 124 stored in the storage unit 120 of the information processing device 10. As shown in FIG.
[0151] The wide-area image DB 124 is a table having columns of wide-area image file name, imaging time, regional information, solar angle, position information, altitude information, attitude information, wavelength, and image ID, with a wide-area image ID as a key for identifying the wide-area spectrum image 128 stored in the memory unit 120 of the information processing device 10.
[0152] The “wide area image ID” is information for identifying the wide area spectrum image 128 stored in the storage unit 120 of the information processing device 10. The “wide area image file name” is information indicating the file name of the wide area spectrum image 128 identified by the wide area image ID and stored in the storage unit 120 of the information processing device 10.
[0153] The "imaging time" is information indicating the time when the wide range spectrum image 128 specified by the wide range image ID was captured. Here, the wide range spectrum image 128 is formed by combining a plurality of spectral images 127, and the imaging times of the individual spectral images 127 are different in the strict sense. Therefore, when the image synthesis unit 136 synthesizes a plurality of spectral images 127 to generate the wide range spectrum image 128, it uses the imaging time of any one of the plurality of spectral images 127 that are the source of synthesis as the imaging time of the wide range spectrum image 128. This operation is also performed for the "sun angle," "position information," "altitude information," and "attitude information."
[0154] The "region information" is information indicating the region where the wide-spectrum image 128, which is specified by the wide-area image ID and stored in the storage unit 120 of the information processing device 10, was captured. The "sun angle" is information indicating the angle of the sun when the wide-spectrum image 128 specified by the wide-area image ID was captured. The value of the "sun angle" field may be blank, for example, at the beginning of operation of the system 1. The "position information" is position information of the drone 20 and the satellite 30 when the spectral images 2104, 3103, and 127 specified by the image ID were captured. The "altitude information" is altitude information of the drone 20 when the wide-spectrum image 128 specified by the wide-area image ID was captured. The "attitude information" is attitude information of the drone 20 when the wide-spectrum image 128 specified by the wide-area image ID was captured. The "wavelength" is information indicating the wavelength band in which the wide-spectrum image 128 specified by the wide-area image ID was captured. The “image ID” is information for identifying the multiple spectral images 127 that are the source of synthesis of the wide band spectrum image 128 identified by the wide band image ID, and is the same as the image ID of the image DB 2103 .
[0155] Each column in the wide area image DB 124 is generated by the image synthesis unit 136 of the control unit 130 of the information processing device 10 and stored in the storage unit 120.
[0156] <System 1 Operation> The processing of the system 1 will be described below with reference to the flowcharts of FIGS.
[0157] FIG. 18 is a flowchart showing a spectral image capturing process performed by the spectral camera control device 31 of the satellite 30.
[0158] First, the spectroscopic camera control device 31 refers to the first imaging condition DB 3101 and waits for the imaging start time described in the first imaging condition DB 3101 to arrive (S1800). Then, when the imaging start time arrives (YES in S1800), the spectroscopic camera control device 31 refers to the first imaging condition DB 3101 and sets the wavelength band for imaging by the spectroscopic camera 32 (S1801), causes the spectroscopic camera 32 to image the ground surface in the set wavelength band, obtains the spectroscopic image 3103 and stores it in the storage unit 310, and also stores the imaging conditions in the image DB 3102 of the storage unit 310 (S1802).
[0159] Next, the spectroscopic camera control device 31 refers to the first imaging condition DB 3101 and determines whether the imaging operation in S1802 has been completed for all wavelength bands (S1803). If it is determined that the imaging operation in all wavelength bands has been completed (YES in S1803), the process proceeds to S1804, and if it is determined that there is a wavelength band for which imaging operation has not yet been performed (NO in S1803), the process returns to S1801, sets the next wavelength band, and performs the subsequent processes.
[0160] In S1804, the spectroscopic camera control device 31 determines whether the imaging end time has arrived by referring to the first imaging condition DB 3101. If it is determined that the imaging end time has arrived (YES in S1804), the process proceeds to S1805, and if it is determined that the imaging end time has not yet arrived (NO in S1804), the process returns to S1800 and waits for the next imaging start time.
[0161] In S1805, the spectroscopic camera control device 31 determines whether or not the imaging operation has been completed for all imaging conditions stored in the first imaging condition DB 3101. If it is determined that the imaging operation has been completed for all imaging conditions (YES in S1805), the spectroscopic image 3103 stored in the storage unit 310 and the image DB 3102 are transmitted to the information processing device 10 (S1806), and if it is determined that there is an imaging condition for which the imaging operation has not yet been completed (NO in S1805), the process returns to S1800 and waits for the next imaging start time.
[0162] Next, FIG. 19 is a flowchart showing a learning model generation process by the learning model generation unit 133 of the control unit 130 of the information processing device 10.
[0163] First, the learning model generation unit 133 refers to the image DB 123 of the storage unit 120, and specifies / selects the location (area, region) where the spectral image 127, which is the source for generating the learning model, was captured (S1900). Next, the learning model generation unit 133 specifies / selects the wavelength band where the spectral image 127 was captured, in order to identify the spectral image 127, which is the source for generating the learning model, from among the spectral images 127 stored in the storage unit 120 (S1901).
[0164] Next, the learning model generation unit 133 extracts from the memory unit 120 a spectral image 127 that matches the conditions specified / selected in S1900 and S1901, and displays the extracted spectral image 127 on a display, which is an example of the output device 111 (S1902).
[0165] Furthermore, the learning model generation unit 133 refers to the image DB 123, acquires the time when the spectral image 127 extracted in S1902 was captured from the item "imaging time", and calculates the angle (sun angle) between the earth's surface and the sun when the spectral image 127 was captured from the imaging time. Next, the learning model generation unit 133 calculates a correction formula (or a correction value) for the spectrum when the spectral image 127 was captured at a specific time (for example, 12:00 noon), and corrects the spectrum of the spectral image 127 using this correction formula (S1903). This correction formula etc. are stored as the learning model 126.
[0166] Thereafter, the learning model generation unit 133 receives a selection input of a region to be learned from the spectral image 127 displayed on the display from the operator of the information processing device 10 using a pointing device such as a mouse, which is an example of the input device 110 (S1904).Then, the learning model generation unit 133 receives a designation input of a specific state of the crop for the region designated in S1903 from the operator of the information processing device 10 using a pointing device such as a mouse, which is an example of the input device 110 (S1905).
[0167] The learning model generation unit 133 receives input designations from S1900 to S1905 from the operator of the information processing device 10, and waits for an instruction to the effect that various inputs for generating the learning model have been completed (S1906). Upon receiving the instruction (YES in S1906), the learning model generation unit 133 generates teacher data 125 based on the input designations from S1900 to S1905 and image data of the target spectral image 127 (S1907). The learning model generation unit 133 generates a learning model 126 based on the teacher data 125 generated in S1907 (S1908). The learning model generation unit 133 stores the teacher data 125 and the learning model 126 generated in S1907 and S1908 in the storage unit 120.
[0168] Next, FIG. 20 is a flowchart showing a spectral image capturing process by the spectral camera control device 21 of the drone 20.
[0169] First, the spectroscopic camera control device 21 flies the drone 20 in accordance with the flight plan DB 2101 stored in the memory unit 210 so that the drone 20 sequentially reaches a target position (S2000).
[0170] Next, the spectroscopic camera control device 21 refers to the second imaging condition DB2102 stored in the storage unit 210, and waits for the drone 20 to reach the imaging position described in the second imaging condition DB2102 (S2001). Then, when the drone 20 reaches the imaging position (YES in S2001), the spectroscopic camera control device 21 causes the spectroscopic camera 22 to capture an image of the observation target area, stores the captured spectroscopic image 2104 in the storage unit 210, and stores the imaging conditions in the image DB2103 of the storage unit 210 (S2002).
[0171] Then, the spectroscopic camera control device 21 judges whether the imaging operation has been completed for all imaging positions stored in the second imaging condition DB 2102 (S2003). If it is judged that the imaging operation has been completed for all imaging conditions (YES in S2003), the spectroscopic image 2104 stored in the storage unit 210 and the image DB 2103 are transmitted to the information processing device 10 (S2004). If it is judged that there is an imaging position for which the imaging operation has not yet been completed (NO in S2003), the process returns to S2001 and waits for the next imaging position to be reached.
[0172] After this, the spectral camera control device 21 ends the flight of the drone 20 (S2005).
[0173] FIG. 21 is a flowchart showing a process performed by the area specifying unit 137 and the type specifying unit 138 of the control unit 130 of the information processing device 10 to specify an area where the crop is in a specific state.
[0174] First, the area specifying unit 137 refers to the image DB 123 of the storage unit 120, and specifies or selects the position (area, region) where the spectroscopic image 127, which is used to specify an area where the crop is in a specific state, was captured (S2100).
[0175] Next, the type identification unit 138 extracts from the storage unit 120 the spectral image 127 that matches the condition specified / selected in S2100, and displays the extracted spectral image 127 on a display, which is an example of the output device 111 (S2101).
[0176] Next, the image synthesis unit 136 generates the wide band spectrum image 128 based on the spectral image 127 extracted in S2101 (S2102), and displays the generated wide band spectrum image 128 on a display or the like (S2103). Note that the operation of generating the wide band spectrum image 128 by the image synthesis unit 136 may be performed prior to the processing shown in FIG.
[0177] Next, the type identification unit 138 refers to the image DB 123, acquires the time when the spectral image 127, which is the source of the wide band spectrum image 128, was captured from the item "imaging time", and calculates the angle (sun angle) between the earth's surface and the sun when the spectral image 127 was captured from the imaging time. Then, based on the learning model 126 or a correction formula or the like stored in the storage unit 120, the type identification unit 138 corrects the spectrum of the spectral image 127 so that it becomes the spectrum when the spectral image 127 is captured at a specific time (e.g., noon), and corrects the spectrum of the wide band spectrum image 128 as a whole (S2104).
[0178] Then, the type identification unit 138 uses the learning model 126 stored in the memory unit 120 to perform an estimation process (inference operation) to determine whether or not crops are in a specific state in the observation area where the wide-band spectrum image 128 was captured, and if so, which area the crops are in (S2105), and displays the inference result on a display or the like (S2106).
[0179] <Screen example> FIG. 24 is a diagram showing an example of a screen displayed on a display, which is an example of the output device 111 of the information processing device 10, in the learning model generation process of the learning model generation unit 133 shown in the flowchart of FIG.
[0180] Buttons 2401 are displayed on the screen 2400 for specifying the spectral image 127 that is the source for generating a learning model, and the operator of the information processing device 10 presses the buttons to specify spectral images 2402-2404 using a pointing device or the like, which is an example of the input device 110. The specified spectral images 2402-2404 are displayed on the screen 2400 for each wavelength band.
[0181] The operator of the information processing device 10 browses these spectral images 2402-2404, and specifies the area 2405 in which the crop is in a specific state for the spectral images 2402-2404 (spectral image 2403 of wavelength β in the example shown in FIG. 24) that best indicates that the crop is in a specific state, based on information on the specific state of the crop in this region and the area in which the crop is in a specific state that has been acquired in advance, using a pointing device or the like that is an example of the input device 110. Next, the operator of the information processing device 10 uses a pull-down menu 2406 to specify the specific specific state of the crop in the area 2405. Then, after completing the specification of the area 2405 and the specification of the specific specific state of the crop using the pull-down menu 2406, the operator of the information processing device 10 clicks an OK button 2407 using a pointing device or the like to input an end of the specification operation.
[0182] Next, FIG. 25 is a diagram showing an example of a screen displayed on a display, which is an example of the output device 111 of the information processing device 10, in the process of identifying an area where a crop is in a specific state by the area identification unit 137 and the type identification unit 138, as shown in the flowchart of FIG. 21.
[0183] A spectral image 2501 is displayed on a screen 2500, and an area 2502 is provided in which information indicating the image capture date and time and the image capture area of the spectral image 2501 is displayed. Map data 2503 corresponding to the area in which the spectral image 2501 was captured is also displayed. When an operator of the information processing device 10 specifies a specific state of a crop to be identified using a pull-down menu 2504, an area 2505 that is estimated to be in the specified specific state is displayed on the spectral image 2501. This area 2505 is also the shading distribution of the spectral image 2501.
[0184] <Effects of the embodiment> As described above in detail, the system 1 of the present embodiment enables business operators who carry out harvesting businesses to more appropriately manage the growing conditions.
[0185] <Modification> In addition, the above-described embodiments are described in detail to clearly explain the present disclosure, and are not necessarily limited to those including all of the described configurations. In addition, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.
[0186] As an example, in the system 1 of the above-mentioned embodiment, the spectral camera 32 mounted on the satellite 30 captures the ground surface to obtain the spectral image 3103, and the spectral camera 22 mounted on the drone 20 captures the observation target area to obtain the spectral image 2104. However, the moving body on which the spectral camera for capturing the spectral image is mounted is not limited to the satellite 30 or the drone 20, and is not limited as long as it is a moving body that can capture the ground surface from the air. Furthermore, a method in which a moving body mounted with a spectral camera moves on the ground surface, for example, an operator holding a spectral camera moves on the ground surface on foot, or a car mounted with a spectral camera runs on the ground surface, may be used. As an example, a multispectral camera that realizes the spectroscopic terminal device disclosed in Japanese Patent No. 6342594 by a smartphone may be used.
[0187] In this way, there are various combinations and variations in the method of acquiring the spectral image 3103 for creating the library and the method of imaging the observation target area to acquire the spectral image 2104. Modified examples of these variations will be described in detail below.
[0188] The LTCF camera, which is the spectral camera 32 mounted on the satellite 30, can receive reflected light in many (for example, several hundred) wavelength bands with a single spectral camera 32 and acquire a spectral image 3103. Meanwhile, the spectral camera 22 mounted on the drone 20 acquires a spectral image 2104 from the reflected light in these multiple wavelength bands, limited to (multiple) wavelength bands that can detect that the crops in the observation target area are in a specific state, based on the library.
[0189] Such a difference in the configuration of the spectral cameras 22, 32 is also reflected in the weight and price of the spectral cameras 22, 32. That is, the spectral camera 32 is more expensive and heavier than the spectral camera 22. The difference in weight between the spectral cameras 22, 32 also affects the configuration of the aircraft on which the spectral cameras 22, 32 are mounted. That is, when the spectral camera 32 is mounted on the drone 20, the configuration of the drone 20 is inevitably enlarged, and further, the time that the drone can fly at one time may be limited to a short time.
[0190] On the other hand, if the satellite 30 is equipped with the spectral camera 32, the restriction due to the weight of the spectral camera 32 is somewhat alleviated, but the frequency at which the spectral image 3103 of the observation target area can be acquired is inevitably low. In other words, the satellite 30 may pass over the observation target area only once every few months, and therefore the frequency at which the spectral image 3103 can be acquired may also be only once every few months.
[0191] For this reason, when the system 1 is first put into operation, it is possible to limit the number of times that the spectral camera 32 captures the spectral image 3103 of the observation target area to a certain number, and also fix the time at which the spectral camera 32 captures the observation target area (for example, 12:00 noon), and create a library using the meridian altitude at that time for that area as the solar angle. In this case, the solar angle is not corrected when creating the library.
[0192] Next, if the time when the spectral camera 22 captures the image of the observation target area is set to the time when the spectral camera 32 captures the image of the observation target area (12:00 noon in the above example), it is possible to detect whether the crops in the observation target area are in a specific state without correcting the solar altitude. After that, the drone 20 or the like is repeatedly flown by changing the time when the spectral camera 22 captures the image of the observation target area to obtain the spectral images 2104 of the observation target area at multiple times, and the solar angle is corrected for the spectral images 2104 to detect whether the crops in the observation target area are in a specific state. As described above, the correction value of the solar angle can be stored in the teacher data 125 and the learning model 126 can be retrained.
[0193] The advantage of acquiring the spectral image 3103 for creating the library by the spectral camera 32, which is an LTCF camera, is that the above-mentioned spectral image 3103 of reflected light of many wavelength bands can be acquired by a few (even once) shots. By acquiring the spectral image 3103 of reflected light of many wavelength bands, it is possible to accurately select a wavelength band suitable for detecting whether or not the crops in the observation area are in a specific state. This allows the number of wavelength bands detected by the spectral camera 22 to be reduced without lowering the detection accuracy of whether or not the crops in the observation area are in a specific state. This also leads to a reduction in the weight of the spectral camera 22, which leads to the use of a highly versatile aircraft as an aircraft such as the drone 20 equipped with the spectral camera 22, and further leads to flying the drone 20 for a long time and with high frequency. If the drone 20 can be flown for a long time, the cost for acquiring the spectral image 2104 can be reduced, and if the drone 20 can be flown with high frequency, it also leads to the detection of whether or not the crops in the observation area are in a specific state as early as possible.
[0194] Furthermore, by aligning the acquisition time of the spectral image 2104 with the acquisition time of the spectral image 3103 without correcting the sun angle when creating the initial library, it is possible to save time when starting up the system 1 and reduce the time and date required to introduce the system 1. Furthermore, by acquiring the spectral images 2104 and 3103 at multiple times as the system 1 is operated, it is possible to improve the accuracy of detection as to whether or not the crop is in a specific state.
[0195] Furthermore, in order to identify the area where the crops are in a specific state, the system 1 of the above-described embodiment uses a spectral image, but in addition to the spectral image, the temperature distribution of the area may be used. For this purpose, when acquiring the spectral image that is the basis for creating the library, the temperature of the area may be acquired by a sensor or the like, and when acquiring the spectral image of the observation target area, the temperature of the area may also be acquired by a sensor or the like. In this case, it is preferable to provide a plurality of temperature measurement points so that the temperature distribution has a two-dimensional spread. In this case, the library also includes temperature distribution data of the area.
[0196] Furthermore, in the above-described embodiment of system 1, the learning model 126 was used to identify areas in the observed region where crops are in a particular state, but so-called image analysis technology may also be used to identify areas where crops are in a particular state.
[0197] Furthermore, in the system 1 according to the present disclosure, the following configuration is also possible.
[0198] When storing information in a memory for detecting that the crop being observed is in a specific state for comparison with a spectral image captured by the multispectral camera, an aircraft equipped with a first multispectral camera is flown to capture images of the earth's surface and generate a spectral image.
[0199] Next, an aircraft equipped with a second multispectral camera different from the first multispectral camera is flown in an observation target area according to an operation plan, and a spectral image is generated by having the second multispectral camera capture the crops at a wavelength for detecting that the crops are in a specific state. Preferably, the aircraft equipped with the first multispectral camera and the aircraft equipped with the second multispectral camera are different aircraft.
[0200] The first multispectral camera has a member capable of switching the wavelength to be imaged, such as a liquid crystal wavelength tunable filter 34, and can generate spectral images for a large number of wavelengths. On the other hand, the second multispectral camera can generate spectral images for a specific wavelength. The specific wavelength cannot be switched while the area to be observed by the second multispectral camera is being imaged. In other words, the specific wavelength is a predetermined wavelength.
[0201] The second multispectral camera may not have a specific member for capturing a spectral image, as compared with the first multispectral camera. For example, the second multispectral camera may generate a spectral image for a specific wavelength without having a member capable of switching wavelengths, unlike the first multispectral camera having a member capable of switching wavelengths. The second aircraft carrying the second multispectral camera may have a low load weight, unlike the first aircraft carrying the first multispectral camera. As a result, the first aircraft and the second aircraft may have different weights (the second aircraft has a smaller aircraft weight than the first aircraft), and the first aircraft and the second aircraft may have different airspaces (geographical ranges determined by latitude and longitude, and altitudes) based on the difference in weight. The second aircraft may be carrying a plurality of (for example, about four) second multispectral cameras, and each of the second multispectral cameras may generate a spectral image for a different wavelength.
[0202] The first multispectral camera 32 capable of acquiring spectral images for many wavelength bands, such as the multispectral camera 32 equipped with the liquid crystal tunable filter 34, tends to be expensive and heavy. On the other hand, the second multispectral camera 22 capable of generating spectral images only for a specific wavelength can be constructed to be inexpensive and lightweight.
[0203] Since the first multispectral camera 32 generates spectral images for creating a library, it is preferable to generate spectral images based on reflected light of a large number of wavelength bands. On the other hand, the second multispectral camera 22 is for identifying areas where crops are in a specific state in the observation area, and it is preferable to take frequent images using an aircraft such as a drone 20. Therefore, the second multispectral camera 22 has a great advantage in that it is lighter and less expensive than the first multispectral camera 32.
[0204] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.
[0205] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Java (registered trademark), and the like.
[0206] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.
[0207] <Additional Notes> The matters described in the above embodiments will be supplemented below.
[0208] (Appendix 1) A method for operating a computer (10), comprising: a first step (S1908) of causing a processor (101) of the computer (10) to store in a memory (120) information (126) for detecting that a crop being observed is in a specific state, for comparison with a spectral image (127) captured by a multispectral camera (22); a second step (S2002) of generating a spectral image (127) in an observation target area by capturing an image with the multispectral camera (22) at a wavelength for detecting that the crop is in a specific state; a third step (S2105) of identifying an area in the observation target area where a crop in the specific state is located, based on the spectral image (127) resulting from the capturing and the information (126) for detecting that the crop is in a specific state; and a fourth step (S2106) of outputting information on the identified area. A method for executing the above. (Appendix 2) The method described in Appendix 1, in a second step, an aircraft (20) equipped with a multispectral camera (22) is flown over the observation area according to an operation plan, and a spectral image (127) is generated by having the multispectral camera (22) capture images at wavelengths for detecting that the crop is in a specific state. (Appendix 3) The method according to claim 1, wherein in a first step (S1908), as information for detection, information indicating the area in which the crop to be observed is grown, a spectral image (127) obtained by photographing the area with a multispectral camera (22) and information on the wavelengths for the photographing, and information on the angle of the sun when photographing the crop in the area are associated with each other, and a trained model (126) generated by performing machine learning using as training data a database is stored in the memory (120); and in a third step (S2105), the area in which the crop in a specific state is located is identified based on the spectral image (127) resulting from the photographing and the trained model (126). (Appendix 4) The method according to Appendix 3, in which in the first step (S1908), a trained model (126) is generated by performing machine learning using a database associated with information on the temperature of the area when the area is imaged by a multispectral camera (22) as training data (125) as information for detection, and a trained model (126) is stored in the memory (120), and in the third step, an area in which crops in a specific state are located is identified based on the temperature of the observed area when the observed area is imaged by the multispectral camera (22). (Appendix 5) The method described in Appendix 3, in a first step (S1908), a database that is training data (125) is further associated with information on a specific type of state, and a trained model (126) generated by the database including the information on the specific type of state is stored in a memory (120), and in a third step (S2105), an area in which crops in a specific state are located and the type of specific state are identified based on the trained model (126). (Appendix 6) The method described in Appendix 2, in which in a second step (S2002), as an operation plan, a plurality of aircraft (20) are flown over different regions, and the spectral images (127) taken by each aircraft are overlaid based on the region in which they were taken to generate a spectral image (128). (Appendix 7) The method according to Appendix 2, in a second step (S2002), as an operation plan, a wavelength of the multispectral camera (22) is specified for each observation area according to the crop to be observed, and the wavelength is switched for each area according to the specified wavelength, thereby generating a spectral image (127) by the multispectral camera (22) of the aircraft. (Appendix 8) The method according to Appendix 2, in a second step (S2002), as an operation plan, if the topography around the crop growing area satisfies a predetermined condition, a spectral image (127) is generated by photographing the area that satisfies the predetermined condition with the multispectral camera (22) of the aircraft. (Appendix 9) The method according to any one of appendices 1 to 8, wherein in a third step (S2105), a geographical area where the crop is in a specific state is distinguished from a geographical area where the crop is not in a specific state, and further, the spectral image (127) is overlaid on a map image based on the point where the image was taken to display the area where the crop is in the specific state. (Appendix 10) The method according to any one of Appendices 1 to 9, wherein in a third step (S2105), a designation of the photographing time is accepted from a user, and a geographical area in which crops are in a particular state is displayed based on a spectral image (127) taken at the designated time. (Appendix 11) A method as described in any of Appendices 1 to 10, in which in a third step (S2105), based on the result of detecting that the crop is in a specific state, information about the crop that is in the specific state and information about the costs of dealing with the crop being in the specific state are displayed. (Appendix 12) The method according to any one of Appendices 1 to 11, in which in a first step (S1908), an aircraft (30) equipped with a multispectral camera (32) is flown to photograph the earth's surface, and based on the obtained spectral image (127), information indicating the area where the crop to be observed is grown is associated with a spectral image (127) obtained by photographing the area with the multispectral camera (32) and information on wavelengths for the photographing and stored in the memory (120); and further, a processor (101) of a computer (10) executes a fifth step (S1904) of accepting, from a user, a designation that the crop is in a specific state in association with the spectral image (127) by having a user view the stored spectral image (127), and a sixth step (S1906) of storing in the memory (120) a database (125) for detecting that the crop is in a specific state in accordance with the user's designation. (Appendix 13) 13. The method according to claim 12, wherein the spectral images for creating the database are images of the same region using a first wavelength group consisting of a plurality of wavelengths, and the spectral images in the second step are images of the same observation target region using a second wavelength group consisting of a plurality of wavelengths, and the number of wavelengths in the first wavelength group is greater than the number of wavelengths in the second wavelength group. (Appendix 14) The method according to claim 12 or 13, wherein in a first step (S1908), information indicating a region is associated with a spectral image (127) and information on wavelengths for capturing the image, and information on a type of a particular state is further stored in a memory (120). (Appendix 15) The method according to any one of appendices 12 to 14, further comprising identifying wavelengths for detecting, based on a spectral image (127), that the crop is in a particular state, based on a designation received from a user in a fifth step (S1904). (Appendix 16) The method according to any one of Addendums 12 to 15, wherein in the first step (S1908), information on the temperature of the area when the area is imaged by the multispectral camera (23) is further associated and stored in the memory. (Appendix 17) The method according to any one of Appendices 12 to 16, wherein, when the topography surrounding a location where a crop is grown satisfies a predetermined condition from among the information stored in the memory (120), a spectral image (127) of an area that satisfies the predetermined condition is extracted, and the extracted spectral image (127) is presented to a user in a selectable manner. (Appendix 18) The method according to any one of Appendices 12 to 17, in response to receiving an input of information about a region from a user, extracting a spectral image (127) of the region related to the input, and presenting the extracted spectral image (127) to the user in a specifiable manner. (Appendix 19) A program for causing a computer (10) to execute the method according to any one of appendices 1 to 18. (Appendix 20) 19. An information processing device for executing the method according to any one of claims 1 to 18. [Explanation of symbols]
[0209] 1...system 2, 30...satellite 3, 20...drone 4, 10...information processing device 101...processor 102...main memory device 103...auxiliary memory device 120, 210, 310...memory unit 121, 2100, 3100...application program 122...flight plan DB 123, 2103, 3102...image DB 124...wide-area image DB 125...teacher data 126...learning model 127, 2104, 3103...spectral image 128...wide-area spectral image 130, 211, 311...control unit 133...learning model generation unit 134...flight plan creation unit 135...aircraft control unit 136...image synthesis unit 137...area identification unit 138...type identification unit 21, 31...spectral camera control device 22, 32...spectral camera N...network
Claims
1. 1. A method for operating a computer, comprising: A first step of storing information for detecting a growth state of a crop to be observed in a memory in advance, the information being to be compared with a spectral image captured by a multispectral camera; a second step of generating the spectral image by capturing images of the crops in a region to be observed using the multispectral camera in a plurality of wavelength bands determined in advance for detecting the growth state of the crops; a third step of identifying the area in which the crop in the particular growth state is located in the observation target area based on the spectral image obtained by the photographing and information for detecting the growth state of the crop; and a fourth step of outputting information on the identified area. In the first step, an aircraft equipped with the multispectral camera is flown to photograph the ground surface, and based on the obtained spectral image, information indicating the region in which the crop to be observed is grown is associated with the spectral image obtained by photographing the region with the multispectral camera and information on the wavelength band for photographing the spectral image, and the associated information is stored in advance in the memory; In the second step, the aircraft is flown in the area to be observed according to an operation plan, and the spectral image is generated by photographing the crop with the multispectral camera using the wavelength band for detecting that the crop is in the specific growth state; The first step comprises: a receiving step of receiving, from the user, a designation that the crop is in the specific growth state in association with the spectral image by allowing the user to view the stored spectral image; and a storage step of storing in the memory a database for detecting that the crop is in the specific growth state in response to the user's designation received in the receiving step.
2. In the first step, the information for the detection includes: Information indicating the region in which the crop to be observed is grown; and The spectral image obtained by photographing the area with the multispectral camera and information on the wavelength band for photographing the spectral image; A trained model is generated by performing machine learning using a database in which information on the angle of the sun when photographing the crops in the region is associated as training data, and the trained model is stored in the memory; The method according to claim 1 , wherein in the third step, the area in which the crop in the particular growth state is located is identified based on the spectral image resulting from the photographing and the trained model.
3. In the first step, the information for detection further includes The learned model is generated by performing the machine learning using a database associated with information on the temperature of the area when the area is imaged by the multispectral camera as training data, and the trained model is stored in the memory; 3. The method according to claim 2, further comprising: identifying the area in which the crop is in the particular growth state based on a temperature of the observed area when the observed area is imaged by the multispectral camera in the third step.
4. In the first step, the database serving as the teacher data is further associated with information on the specific growth state type, and the trained model generated by the database including the information on the specific growth state type is stored in the memory; The method according to claim 2 , in the third step, the area in which the crop in the particular growth state is located and the type of the particular growth state are identified based on the trained model.
5. The method according to claim 1, wherein in the second step, as the operation plan, a plurality of the aircraft are flown over different areas, and the spectral images taken by each of the aircraft are overlaid based on the area in which they were taken to generate the spectral image.
6. 2. The method of claim 1, wherein in the second step, as the operation plan, the wavelength band of the multispectral camera is specified for each of the regions to be observed according to the crop to be observed, and the wavelength band is switched in each region in accordance with the specification of the wavelength band, and the spectral image is generated by the multispectral camera of the aircraft.
7. The method according to claim 1, wherein in the second step, when the operation plan includes a step of generating the spectral image by photographing the area where the terrain surrounding the growing location of the crop satisfies a specified condition using the multispectral camera of the aircraft, the area that satisfies the specified condition is given priority.
8. 2. The method of claim 1, wherein in the fourth step, a geographical area in which the crop is in the specific growth state is distinguished from a geographical area in which the crop is not in the specific growth state, and the spectral image is overlaid on a map image based on a point at which the spectral image was taken to display the area in which the crop is in the specific growth state.
9. 2. The method according to claim 1, wherein in the fourth step, a designation of a photographing time is accepted from a user, and a geographical area in which the crop is in the particular growth state is displayed based on the spectral image photographed at the designated time.
10. The method according to claim 1, wherein in the fourth step, based on the result of detecting that the crop is in the specific growth state, information about the crop in the specific growth state and information about the cost of dealing with the crop being in the specific growth state are displayed.
11. the spectral images for creating the database are images of the same region for a first wavelength group consisting of a plurality of wavelength bands; the spectral images in the second step are images of the area of the same observation target for a second wavelength group consisting of a plurality of wavelength bands; The method of claim 1 , wherein the number of wavelength bands in the first wavelength group is greater than the number of wavelength bands in the second wavelength group.
12. The method according to claim 1 , wherein in the first step, information indicating the region is associated with information on the spectral image and the wavelength band for capturing the spectral image, and information on the type of the specific growth condition is further stored in the memory.
13. The method according to claim 1 , further comprising identifying a wavelength band for detecting, from the spectral image, that the crop is in the specific growth state, based on the designation received from the user in the receiving step.
14. In the first step, The method according to claim 1 , further comprising: storing in the memory in association with information on temperature of the area when the area is imaged by the multispectral camera.
15. 2. The method according to claim 1, further comprising: extracting the spectral image of the area that satisfies a predetermined condition from the information stored in the memory, and presenting the extracted spectral image to the user in a selectable manner when the topography surrounding the location where the crop is grown satisfies the predetermined condition.
16. The method according to claim 1 , further comprising: extracting a spectral image of the area related to the input from the user; and presenting the extracted spectral image to the user in a selectable manner.
17. A program for causing a computer to execute the method according to claim 1.
18. An information processing apparatus for executing the method according to claim 1.
Citation Information
Patent Citations
Analytical method for growth condition of crop
JP2006250827A
Mapping field anomalies using digital imagery and machine learning models
JP2022510487A
Plant growth index measurement device and method, and plant growth index measurement system
WO2016181743A1
Spectral camera control device, spectral camera control program, spectral camera control system, aircraft equipped with said system, and spectral image capturing method
WO2017179378A1
Growth assessment system, growth assessment server and growth assessment method
WO2021157032A1
