Crop type classification in images

Abstract

A method includes: receiving input containing search parameters by a computing device, and presenting a crop indication image depicting a portion of a geographic region selected based on the search parameters, including at least one image from a multispectral and time-series image overlaid with an indication of a crop type classification determined for a specific location; wherein the crop type classification determined for a specific location in the at least one image is determined by: obtaining an image set including the multispectral and time-series images, predicting the crop type growing in the specific location; and determining a crop type classification for each specific location based on the crop type; wherein determining the crop type classification includes: selecting a crop type classification as the crop type in response to determining that the crop types predicted for the specific location include a majority of the predicted crop types; and dividing the specific location into multiple sub-specific locations in response to determining that the crop types not included in the majority of the predicted crop types; and classifying them as the corresponding crop types.

Description

[0001] This application is a divisional application of the invention patent application filed on January 15, 2019, with application number 201980009776.0 and invention title "Classification of Crop Types in Images".

[0002] Cross-references to related applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 620,939, filed January 23, 2018, the entire disclosure of which is incorporated herein by reference. Technical Field

[0004] This disclosure generally relates to image feature detection, and particularly, but not exclusively, to the use of machine learning in image feature detection. Background Technology

[0005] Currently, approximately 11% of the land surface is used for crop production. While agriculture is vital to human survival, environmental impact, national impact, business enterprises, and markets, there is no consistent, reliable, and / or precise knowledge about which crops can be grown within a geographic region, county, state, country, continent, planet, or any of these areas. For example, if more information were available about agricultural fields, seed and fertilizer companies could better determine the market availability of their products in different geographic regions; crop insurance companies could more accurately and cost-effectively assess premiums; banks could provide farm loans more precisely; and / or governments could better evaluate taxes, allocate subsidies, determine regional food capacity, plan infrastructure, and so on.

[0006] Regarding map data that may be associated with agricultural land, such data is often inconsistent, inaccurate, outdated, and / or incomplete for many practical uses. For example, a government entity may survey or sample all agricultural land and / or a small subset of farmers within a geographic area and infer a small dataset to approximate the location, size, shape, crop type, quantity, etc., of all agricultural land actually existing within the geographic area. Due to the labor-intensive nature of collecting such data, agricultural land data is often not updated frequently (or too infrequently for many commercial purposes).

[0007] Agricultural land use often varies by region or time. Farms in developing countries tend to be much smaller than those in developed countries. Crops may also differ seasonally or annually for the same plot of land. Agricultural land can be repurposed for non-agricultural uses (e.g., housing development). Therefore, it would be beneficial to identify agricultural land at a sufficiently granular level, inexpensively, accurately, and frequently. Summary of the Invention

[0008] According to embodiments of this disclosure, a method for crop type classification is provided, comprising: receiving input by a computing device including one or more search parameters, wherein the one or more search parameters include one or more of county, size, shape, and identifier; and presenting by the computing device a crop indication image depicting a portion of a geographic region, wherein the portion of the geographic region is selected based on the one or more search parameters, and wherein the crop indication image includes at least one image from a set of multispectral and time-series images overlaid with an indication of a crop type classification determined for a specific location depicted in at least one image; The crop type classification determined for a specific location depicted in the at least one image is determined by: obtaining multiple image sets associated with a geographic region and a time period, wherein each image set in the multiple image sets includes multispectral and time-series images depicting a corresponding specific portion of the geographic region during the time period; predicting one or more crop types growing in each specific location within the specific portion of the geographic region associated with the image sets in the multiple image sets; and determining a crop type classification for each specific location based on the predicted one or more crop types for the corresponding specific location; wherein determining the crop type classification for each specific location includes classifying crop types in the crop region based on the image by: selecting the majority of predicted crop types as the crop type classification in response to determining that the predicted crop types for the corresponding specific location include the majority of predicted crop types; and dividing the corresponding specific location into multiple sub-specific locations in response to determining that the predicted crop types for the corresponding specific location do not include the majority of predicted crop types; and classifying each corresponding sub-specific location as a corresponding crop type among the predicted crop types for the specific location.

[0009] Optionally, the method further includes providing a user interface through which a user can manually modify the crop type classification; receiving crop type classification modifications from the user; and updating a database storing the at least one image and crop type classification using the crop type classification modifications.

[0010] Optionally, predicting the one or more crop types to grow at each specific location includes: predicting the presence of crops at the specific location; determining crop boundary locations within a specific part of a geographic area based on the predicted presence of crops at the specific location; and predicting the one or more crop types to grow within each determined crop boundary location.

[0011] Optionally, the method further includes the computing device presenting an estimate of crop yield for each specific location based on a crop type classification determined for that specific location.

[0012] Optionally, the method further includes the computing device presenting guidance on crop management practices for each specific location based on a crop type classification determined for that specific location.

[0013] Optionally, determining the crop type classification for each specific location includes determining the crop type classification to a sub-meter ground resolution for each specific location.

[0014] Optionally, predicting the one or more crop types growing at each specific location includes applying the image set to one or more machine learning systems or convolutional neural networks (CNNs).

[0015] Optionally, the one or more machine learning systems or CNNs are configured to predict the one or more crop types to grow in each specific location after supervised training on real-world ground data.

[0016] Optionally, the ground-based real data includes one or more of the following: government crop data, publicly available crop data, images of crop areas identified at low ground resolution, images of crop types identified at low ground resolution, images of manually identified crop boundaries, images of manually identified crop boundaries and crop types, crop survey data, sampled crop data, and farmer reports.

[0017] Optionally, the first resolution of the first image of the image set is different from the second resolution of the second image of the image set, the first resolution is lower than the third resolution of the crop indicator image, and the fourth resolution of at least a portion of the ground truth data is lower than the third resolution of the crop indicator image.

[0018] Optionally, predicting the one or more crop types growing at each specific location includes: for each specific location, analyzing the changes over time of pixels in a time series image associated with the corresponding specific location, wherein a specific pattern of change of the pixels is associated with at least one crop type.

[0019] According to embodiments of this disclosure, a method for crop type classification is provided, comprising: obtaining a plurality of image sets associated with a geographic region and a time period, wherein each image set in the plurality of image sets includes multispectral and time-series images depicting a corresponding specific portion of the geographic region during the time period; predicting one or more crop types growing in each specific location within the specific portion of the geographic region associated with the image sets in the plurality of image sets; determining a crop type classification for each specific location based on the one or more crop types predicted for the corresponding specific location; and generating a crop indication image including at least one image from the multispectral and time-series images of the image sets overlaid with an indication of the crop type classification determined for the corresponding specific location; wherein predicting the one or more crop types growing in each specific location includes applying the image sets to one or more machine learning systems, wherein the one or more machine learning systems include convolutional neural networks (CNNs); and wherein the one or more machine learning systems are configured to predict the one or more crop types growing in each specific location after supervised training on ground-based data.

[0020] Optionally, predicting the one or more crop types to grow at each specific location includes: predicting the presence of crops at the specific location; determining crop boundary locations within a specific part of a geographic area based on the predicted presence of crops at the specific location; and predicting the one or more crop types to grow within each determined crop boundary location.

[0021] Optionally, determining the crop type classification for each specific location includes: for each specific location, selecting the majority of predicted crop types from the crop types predicted for that specific location, wherein the majority of predicted crop types are crop type classifications.

[0022] Optionally, determining the crop type classification for each specific location includes: for each specific location, if there is no major majority predicted crop type, dividing the corresponding specific location into multiple sub-specific locations, and classifying each of the multiple sub-specific locations as a corresponding crop type among the predicted crop types for the specific location.

[0023] Optionally, determining the crop type classification for each specific location includes determining the crop type classification to a sub-meter ground resolution for each specific location.

[0024] Optionally, the ground-based real data includes one or more of the following: government crop data, publicly available crop data, images of crop areas identified at low ground resolution, images of crop types identified at low ground resolution, images of manually identified crop boundaries, images of manually identified crop boundaries and crop types, crop survey data, sampled crop data, and farmer reports.

[0025] Optionally, the first resolution of the first image of the image set is different from the second resolution of the second image of the image set, the first resolution is lower than the third resolution of the crop indicator image, and the fourth resolution of at least a portion of the ground truth data is lower than the third resolution of the crop indicator image.

[0026] Optionally, predicting the one or more crop types growing at each specific location includes: for each specific location, analyzing the changes over time of pixels in a time series image associated with the corresponding specific location, wherein a specific pattern of change of the pixels is associated with at least one crop type.

[0027] Optionally, the method further includes estimating crop yield for each specific location based on crop type classification determined for that specific location.

[0028] Optionally, the method further includes determining crop management practices for each specific location based on a crop type classification determined for that specific location.

[0029] Optionally, the method further includes: displaying a crop indicator image on a user-accessible device; and modifying a specific indicator among indicators received from the user for a crop type classification determined for a specific location, wherein the modification includes a manual reclassification of the crop type for the specific location associated with the specific indicator.

[0030] According to embodiments of this disclosure, a method for crop type classification is provided, comprising: receiving an input including one or more search parameters, wherein the one or more search parameters include one or more of county, size, shape, and identifier; and presenting a crop indicator image depicting a portion of a geographic region, wherein the portion of the geographic region is selected based on the one or more search parameters, and wherein the crop indicator image includes at least one image in an image set associated with the geographic region, overlaid with an indication of a crop type classification determined for a specific location depicted in at least one image; The crop type classification determined for a specific location depicted in the at least one image is determined by: obtaining multiple image sets associated with a geographic region and a time period, wherein each image set in the multiple image sets includes multispectral and time-series images depicting a corresponding specific portion of the geographic region during the time period; predicting one or more crop types growing in each specific location within the specific portion of the geographic region associated with the image sets in the multiple image sets; and determining a crop type classification for each specific location based on the predicted one or more crop types for the corresponding specific location; wherein predicting the one or more crop types growing in each specific location includes applying the image sets to one or more machine learning systems, wherein the one or more machine learning systems include convolutional neural networks (CNNs); and wherein the one or more machine learning systems are configured to predict the one or more crop types growing in each specific location after supervised training on ground-based real data.

[0031] Optionally, the instructions also enable the device to: provide a user interface through which a user can manually modify the crop type classification; receive crop type classification modifications from the user; and update a database storing the at least one image and crop type classification using the crop type classification modifications.

[0032] Optionally, predicting the one or more crop types to grow at each specific location includes: predicting the presence of crops at the specific location; determining crop boundary locations within a specific part of a geographic area based on the predicted presence of crops at the specific location; and predicting the one or more crop types to grow within each determined crop boundary location.

[0033] Optionally, determining the crop type classification for each specific location includes: for each specific location, selecting the majority of predicted crop types from the crop types predicted for that specific location, wherein the majority of predicted crop types are crop type classifications.

[0034] Optionally, determining the crop type classification for each specific location includes: for each specific location, if there is no major majority predicted crop type, dividing the corresponding specific location into multiple sub-specific locations, and classifying each of the multiple sub-specific locations as a corresponding crop type among the predicted crop types for the specific location.

[0035] Optionally, determining the crop type classification for each specific location includes determining the crop type classification to a sub-meter ground resolution for each specific location.

[0036] Optionally, the ground-based real data includes one or more of the following: government crop data, publicly available crop data, images of crop areas identified at low ground resolution, images of crop types identified at low ground resolution, images of manually identified crop boundaries, images of manually identified crop boundaries and crop types, crop survey data, sampled crop data, and farmer reports.

[0037] Optionally, the first resolution of the first image of the image set is different from the second resolution of the second image of the image set, the first resolution is lower than the third resolution of the crop indicator image, and the fourth resolution of at least a portion of the ground truth data is lower than the third resolution of the crop indicator image.

[0038] Optionally, predicting the one or more crop types growing at each specific location includes: for each specific location, analyzing the changes over time of pixels in a time series image associated with the corresponding specific location, wherein a specific pattern of change of the pixels is associated with at least one crop type.

[0039] Optionally, the method may further include presenting an estimate of crop yield for each specific location based on a crop type classification determined for that specific location, or presenting guidance on crop management practices for each specific location based on a crop type classification determined for that specific location.

[0040] According to embodiments of this disclosure, an apparatus for crop type classification is provided, including one or more processors and a computer-readable storage medium storing a plurality of instructions, the instructions being executed by the one or more processors to cause the apparatus to perform the aforementioned method. Attached Figure Description

[0041] Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein, unless otherwise specified, the same reference numerals refer to the same parts throughout the various views. Not all instances of elements are necessarily labeled to avoid confusing the figures in appropriate places. The figures are not necessarily drawn to scale, but rather focus on illustrating the described principles.

[0042] Figure 1 A block diagram illustrating a network view of an example system incorporating the crop type classification techniques of this disclosure, according to some embodiments, is depicted.

[0043] Figure 2 The illustrations depicting some embodiments can be derived from Figure 1 The flowchart shows an example of the system implementation's processing.

[0044] Figures 3A-3B Example images depict crop type classification techniques according to this disclosure based on some embodiments.

[0045] Figure 4 The illustrations depicting some embodiments can be derived from Figure 1 Another example of a system implementation processing flowchart.

[0046] Figure 5 The illustrations depicting some embodiments can be derived from Figure 1 This is another example of a process flowchart implemented by the system.

[0047] Figure 6 This disclosure describes some embodiments that may be implemented in this way. Figure 1 Example devices implemented in the system. Detailed Implementation

[0048] This document describes embodiments of systems, apparatuses, and methods for classifying crop types in images. In some embodiments, a method includes: obtaining a plurality of image sets associated with a geographic region and a time period, each of the plurality of image sets comprising multispectral and time-series images depicting a corresponding specific portion of the geographic region during the time period; predicting one or more crop types growing in each specific location within the specific portion of the geographic region associated with the image sets in the plurality of image sets; determining a crop type classification for each specific location based on the predicted one or more crop types for the corresponding specific location; and generating a crop indication image comprising at least one image from the multispectral and time-series images of the image sets overlaid with an indication of the crop type classification determined for the corresponding specific location.

[0049] In the following description, numerous specific details are set forth to provide a thorough understanding of the embodiments. However, those skilled in the art will recognize that the techniques described herein can be practiced without one or more of these specific details, or through other methods, components, materials, etc. In other instances, well-known structures, materials, or operations have not been shown or described in detail to avoid obscuring certain aspects.

[0050] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the phrases "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, in one or more embodiments, particular features, structures, or characteristics may be combined in any suitable manner.

[0051] Figure 1 A block diagram illustrating a network view of an example system 100 incorporating crop type classification techniques of the present disclosure, according to some embodiments, is depicted. System 100 may include a network 102, a server 104, a database 106, a server 108, a database 110, a device 112, and an aerial image capture device 116. One or more of the server 104, database 106, server 108, database 110, device 112, and aerial image capture device 116 may communicate with the network 102. At least server 108 may include the crop type classification techniques of the present disclosure to facilitate the automatic identification of one or more crop types in an image at sub-meter resolution, as described more fully below.

[0052] Network 102 may include one or more wired and / or wireless communication networks. Network 102 may include one or more network elements (not shown) to physically and / or logically connect computer devices to exchange data with each other. In some embodiments, network 102 may be the Internet, a wide area network (WAN), a personal area network (PAN), a local area network (LAN), a campus network (CAN), a metropolitan area network (MAN), a virtual local area network (VLAN), a cellular network, a carrier network, a WiFi network, a WiMax network, and so on. Furthermore, in some embodiments, network 102 may be a private, public, and / or secure network that can be used by a single entity (e.g., a business, school, government agency, household, individual, etc.). Although not shown, network 102 may include, but is not limited to, servers, databases, switches, routers, gateways, base stations, repeaters, software, firmware, intermediate servers, and / or other components to facilitate communication.

[0053] Server 104 may include one or more computers, processors, cellular infrastructure, network infrastructure, backhaul infrastructure, hosting servers, servers, workstations, personal computers, general-purpose computers, laptops, internet-connected appliances, handheld devices, wireless devices, Internet of Things (IoT) devices, portable devices, etc., configured to facilitate the collection, management, and / or storage of aerial imagery (also referred to as land surface images, land images, picture images, or pictures) of a land surface at one or more resolutions. For example, server 104 may command device 116 to acquire images of one or more specific geographic areas, to traverse a specific orbit, to acquire images at a specific resolution, to acquire images at a specific frequency, or to acquire images of a specific geographic area over a specific time period. As another example, server 104 may communicate with device 116 to receive images acquired by device 116. As yet another example, server 104 may be configured to acquire / receive images from government sources, users (e.g., such as user 114), etc., with associated crop-related information including (e.g., crop type identification, crop boundaries, identified road locations, and / or other annotation information). As will be discussed in detail below, images containing associated crop-related information may include human-tagged images, USDA Farmland Data Layer (CDL) data, FSA Common Land Unit (CLU) data, ground truth data, and so on.

[0054] Servers 104 can communicate with each other directly and / or via network 102 with device 116. In some embodiments, server 104 may include one or more web servers, one or more application servers, one or more intermediate servers, etc.

[0055] Database 106 may include one or more storage devices to store data and / or instructions for use by server 104, device 112, server 108, and / or database 110. For example, database 106 may include images and associated metadata provided by device 116. The content of database 106 may be accessed via network 102 and / or directly by server 104. The content of database 106 may be arranged in a structured format to facilitate selective retrieval. In some embodiments, database 106 may include more than one database. In some embodiments, database 106 may be included within server 104.

[0056] According to some embodiments, server 108 may include one or more computers, processors, cellular infrastructure, network infrastructure, backhaul infrastructure, hosting servers, servers, workstations, personal computers, general-purpose computers, laptops, internet-connected appliances, handheld devices, wireless devices, Internet of Things (IoT) devices, portable devices, etc., configured to implement one or more features of the crop type classification technology of this disclosure. Server 108 may be configured to train and generate a machine learning-based model using images and potentially associated data provided by server 104 / database 106, capable of automatically detecting crop boundaries and classifying (one or more) crop types within crop boundaries in multiple images of the land surface. Crop type classification may be at sub-meter granularity or ground resolution. The "trained" machine learning model may be configured to identify crop boundaries and classify crop types in images without human supervision. The model may be trained by implementing supervised machine learning techniques. Server 108 may also facilitate access to and / or use of images with crop type classifications.

[0057] Server 108 may communicate directly or via network 102 with one or more of server 104, database 106, database 110, and / or device 112. In some embodiments, server 108 may also communicate with device 116 to facilitate the integration of one or more functions of server 104 as described above. In some embodiments, server 108 may include one or more web servers, one or more application servers, one or more intermediate servers, etc.

[0058] Server 108 may include hardware, firmware, circuitry, software, and / or combinations thereof to facilitate various aspects of the techniques described herein. In some embodiments, server 108 may include, but is not limited to, image filtering logic 120, crop type prediction logic 122, training logic 124, crop type classification logic 126, post-detection logic 128, and crop boundary detection logic 130. As will be described in detail below, image filtering logic 120 may be configured to apply one or more filtering, “cleaning,” or denoising techniques to an image to remove artifacts and other unwanted data from the image. Crop type prediction logic 122 may be configured to predict one or more crop types growing within each crop region defined by crop boundaries. Crop type prediction logic 122 may include at least a portion of a “trained” machine learning-based model. Training logic 124 may be configured to facilitate supervised learning, training, and / or refinement using one or more machine learning techniques to generate / configure crop type prediction logic 122. Alternatively, training logic 124 may be configured to support unsupervised learning, semi-supervised learning, reinforcement learning, computer vision techniques, and so on.

[0059] Crop type classification logic 126 can be configured to classify or identify (one or more) crop types within each crop region associated with a crop boundary based on (one or more) crop types predicted by crop type prediction logic 122. Post-detection logic 128 can be configured to perform one or more post-crop type classification activities, such as, but not limited to, determining crop yields for different crop types, determining crop management practices / strategies, assigning unique identifiers to each crop field (or crop subfield) associated with the detected crop boundary, providing crop field (or subfield) search capabilities, and so on.

[0060] Crop boundary detection logic 130 can be configured to detect crop boundaries within an image. In some embodiments, crop boundary detection logic 130 can be used to generate at least a portion of ground truth data. Alternatively or additionally, crop boundary detection logic 130 can include part of a “trained” machine learning-based model for performing crop type classification, wherein the “trained” model detects crop boundaries (to identify crop regions / fields / subfields) and then classifies crops located within those crop regions / fields / subfields according to their (one or more) crop types. Similar to crop type prediction logic 122, training logic 124 can be configured to facilitate supervised learning, training, and / or refinement using one or more machine learning techniques to generate / configure crop boundary detection logic 130.

[0061] In some embodiments, one or more (or a portion thereof) of logic 120-130 may be implemented as software comprising one or more instructions to be executed by one or more processors included in server 108. In alternative embodiments, one or more (or a portion thereof) of logic 120-130 may be implemented as firmware or hardware, such as, but not limited to, application-specific integrated circuits (ASICs), programmable array logic (PALs), field-programmable gate arrays (FPGAs), etc., included in server 108. In other embodiments, one or more (or a portion thereof) of logic 120-130 may be implemented as software, while other logic (or a portion thereof) of logic 120-130 may be implemented as firmware and / or hardware.

[0062] Although Figure 1 Server 108 may be depicted as a single device, but it is contemplated that server 108 may include one or more servers and / or one or more of logics 120-130 may be distributed across multiple devices. In some embodiments, depending on computing resources or limitations, one or more of logics 120-130 may be implemented in multiple instances.

[0063] Database 110 may include one or more storage devices for storing data and / or instructions used by server 108, device 112, server 104, and / or database 110. For example, database 110 may include images provided by server 104 / database 106 / device 116; ground-based real-world data for constructing and / or training crop type prediction logic 122; crop type heatmaps generated by crop type prediction logic 122; crop type classifications, identifiers, and other associated images and / or crop type information generated by crop type classification logic 126; data used by any of logics 120-130; data generated by any of logics 120-130; data to be accessed by user 114 via device 112; and / or data to be provided by user 114 via device 112. The content of database 110 may be accessed via network 102 and / or directly by server 108. The content of database 110 may be arranged in a structured format to facilitate selective retrieval. In some embodiments, database 110 may include more than one database. In some embodiments, database 110 may be included within server 108.

[0064] Device 112 may include one or more computers, workstations, personal computers, general-purpose computers, laptops, internet-connected appliances, handheld devices, wireless devices, Internet of Things (IoT) devices, portable devices, smartphones, tablets, etc. In some embodiments, user 114 may interface with device 112 to provide data to be used by one or more of logics 120-130 (e.g., manually identifying crop boundaries and crop types on a selected image for use as ground truth data) and / or request data associated with classified crop types (e.g., searching for a specific crop field (or subfield), requesting a visual display of a specific image overlaid with crop type information). At least training logic 124 and / or detection logic 128 may facilitate the functionality associated with device 112. The user 114 providing data for crop type classification may be the same as or different from the user requesting data already generated according to the execution of a crop type classification model.

[0065] Device 116 may include one or more of the following: satellites, aircraft, drones, hot air balloons, and / or other devices capable of capturing multiple aerial or airborne photographs of the land surface. The multiple aerial photographs may include multiple multispectral time-series images. Device 116 may include one or more location tracking mechanisms (e.g., Global Positioning System (GPS)), multispectral imaging mechanisms (all frequency bands), weather condition detection mechanisms, timestamp generation mechanisms, mechanisms for detecting distances from the land surface, and / or associated image metadata generation capabilities to provide associated image information for each of the multiple captured images. Device 116 may operate manually and / or automatically, and the captured images may be provided to server 104, server 108, or other devices via wired or wireless connections. Device 116 may also be deployed multiple times at the same location within a specific time period to capture time-series images of the same location. Examples of images provided by or generated from images provided by device 116 (associated with ground reality data, or potentially intended for automatic crop type classification) include, but are not limited to, Landsat 7 satellite images, Landsat 8 satellite images, Google Earth images, and so on.

[0066] Although the above combination Figure 1 Discrete components have been discussed, but the components can be combined. For example, servers 104 and 108 may include a single component, databases 106 and 110 may include a single component, and / or device 112 may be combined with server 108.

[0067] Figure 2 A flowchart of an illustrated example process 200 according to some embodiments is depicted, which can be implemented by system 100 to generate a crop type classification model, perform crop type classification using the generated crop type classification model, and various uses of crop type classification information.

[0068] At box 202, training logic 124 can be configured to acquire or receive ground truth data, which includes multiple land surface images with identified crop boundaries (or corresponding crop regions) and the crop types classified therein. Multiple images including ground truth data can be selected to cover those with various land features, crop boundaries, crop types, etc., in order to train / generate a detection model capable of handling multiple land features, crop boundaries, or crop types that may exist in unknown images to be classified.

[0069] In some embodiments, multiple images may include images containing multispectral data (e.g., red-green-blue (RGB) spectrum, visible spectrum, near-infrared (NIR), normalized difference vegetation index (NDVI), infrared (IR), all spectral bands, etc.) (also referred to as multispectral images or picture images). Multiple images may also include time-series images, where the same geographic location may be imaged multiple times within a specific time period. A specific time period may include, but is not limited to, the crop growing season (e.g., May to October), a year, multiple years, 2008 to 2016, and / or other predetermined times. The imaging frequency may be hourly, daily, weekly, bi-weekly, monthly, quarterly, or annually. Images associated with a specific geographic location and optionally within a specific time period may be referred to as an image set. Multiple image sets may be included in ground reality data.

[0070] Ground-based real data may include, but is not limited to: (1) images with identified crop boundaries (or crop areas)—such images may be manually identified by the user and / or as the result of automatic crop boundary detection (examples of which are described in Figure 3); (2) images with classified / identified / specified crop types—such images may be manually identified by the user and / or obtained from government or publicly available sources; and / or (3) images with identified crop boundaries and crop types—such images may be manually identified by the user and / or obtained from government or publicly available sources.

[0071] Image features manually identified by users may also be referred to as human-labeled data or human-labeled images. For example, one or more users (such as user 114) may annotate and select images via a graphical user interface (GUI) mechanism provided on device 112. Images with identified crop boundaries and / or crop types obtained from government or publicly available sources may provide such identification at a lower ground resolution or accuracy than that provided by the crop type classification scheme of this disclosure. For example, the ground resolution may be 30 meters, greater than 1 meter, etc. Crop boundary and / or type identification from government or publicly available sources may also be provided as farmer reports, sample-based data, survey-based data, extrapolation, etc. An example of government / publicly available geoidentified crop boundary and type data may be USDA CDL data from 2008–2016 at a resolution of 30 meters per pixel (ground).

[0072] Training logic 124 can facilitate image selection, presentation of selected images for human labeling, use of human-labeled images, acquisition of government / publicly available crop boundary and / or crop type identification data, etc. Ground-based data can also be referred to as training data, model building data, model training data, etc.

[0073] In some embodiments, the time period and / or (one or more) geographic regions associated with the ground truth data may be the same (or approximately the same) as the time period and / or (one or more) geographic regions associated with the image for which crop type is to be identified (at box 216). For example, for images taken between 2008 and 2016 for which operation is to be performed at box 216, CLU data from 2008 may be used, CDL data from 2008–2016 may be used, and human-tagged data may include images taken between 2008 and 2016. The CLU and CDL data may include image data of the United States, and the images in the human-tagged data may also include image data of the United States.

[0074] Next, at box 204, image filtering logic 120 can be configured to perform preliminary filtering on one or more images including ground reality data. In some embodiments, preliminary filtering may include monitoring for clouds, shadows, haze, fog, atmospheric obstructions, and / or other land surface obstructions included in the image on a per-pixel basis. On a per-pixel basis, if such an obstruction is detected, then image filtering logic 120 can be configured to determine whether to resolve the obstruction, how to correct the obstruction, and whether to omit image information associated with the pixel of interest when constructing the model at box 206. For example, if a first pixel does not include land surface information due to clouds, but the geographic location associated with a second pixel adjacent to the first pixel is imaged because it is not obscured by clouds, then image filtering logic 120 can be configured to change the first pixel value to the second pixel value. As another example, a known incorrect pixel value in a given image can be replaced by the pixel value of the corresponding pixel from another image within the same image set (e.g., from different images for the same geographic location in the same time series). In other embodiments, for example, if the image is known to be cloudless and free of atmospheric obstructions, then box 204 may be optional.

[0075] At box 206, using the obtained and optionally pre-filtered or corrected ground truth data, the resulting ground truth data can be applied to one or more machine learning techniques / systems to generate or construct a crop type model. In some embodiments, the crop type model may include crop type prediction logic 122. The machine learning techniques / systems may include, for example, convolutional neural networks (CNNs) or supervised learning systems. The crop type model can be configured to provide a probabilistic prediction of one or more crop type classifications for each pixel corresponding to a specific geographic location associated with the image set provided as input. Crop types may include, but are not limited to, rice, wheat, corn / grain, soybean, sorghum, legumes, fruits, vegetables, oilseeds, nuts, pasture, etc.

[0076] Because ground-based data includes images with accurately identified crop boundaries and crop types, machine learning techniques / systems can learn which land surface features in the images indicate crop areas, and the types (one or more) of crops growing within those areas. When such knowledge is sufficiently detailed and accurate, it can then be used to automatically identify crop types in images where the crop types might be unknown.

[0077] To make predictions about (one or more) crop types within a crop region, it may be necessary to predict the existence of the crop region, such that the portion of the image to be analyzed for crop type prediction can be reduced or minimized. Therefore, in some embodiments, crop boundary detection logic 130 and crop type prediction logic 122 can be considered part of a crop type model. Crop boundary detection logic 130 is discussed in conjunction with Figure 3. The crop type model may also be referred to as a crop type classification model.

[0078] In some embodiments, a crop type model may be associated with a specific geographic region—the same geographic region captured in an image that includes ground-based data. For example, a crop type model may be specific to a particular county within the United States. Similarly, a crop type model may be associated with a specific time period—the same time period associated with an image that includes ground-based data. As the geographic region becomes larger, data inconsistencies or regional differences may arise, which could lead to a decrease in the accuracy of the crop type model.

[0079] Next, training logic 124 can be configured to determine whether the accuracy of the crop type model is equal to or exceeds a predetermined threshold. The predetermined threshold could be 70%, 80%, 85%, 90%, etc. If the model's accuracy is less than the predetermined threshold (the negative branch of box 208), then process 200 can return to box 202 to obtain / receive additional ground truth data to apply to machine learning techniques / systems to refine the current crop type model. Providing additional ground truth data to the machine learning techniques / systems includes providing additional supervised learning data, enabling better configuration of the crop type model to predict which(one) types of crops are / have already grown in the crop region. One or more iterations of boxes 202-208 can be performed until a sufficiently accurate crop type model can be built.

[0080] If the model's accuracy equals or exceeds a predetermined threshold (the branch is in box 208), then the crop type model can be considered acceptable for unsupervised or automatic crop type classification of images in which the crop type (and crop boundaries) are unknown. At box 210, multiple images to be applied to the crop type model for automatic classification can be acquired or received. These multiple images can be images captured by device 116.

[0081] In some embodiments, the multiple images may include multiple image sets, wherein each image set in the multiple image sets may be associated with a corresponding part / region (e.g., a county in the United States) of multiple parts / regions (e.g., all counties in the United States) of a geographic area (e.g., the United States) that may collectively include a classification of crop types for all crop fields / subfields located therein. For each of the multiple parts / regions, the associated image set may include: (1) at least one image at each of multiple time points (e.g., May 1, June 1, July 1, August 1, September 1, October 1), and (2) for the corresponding time point of the multiple time points, there may also be one or more images, wherein each image can provide specific / different spectral information from another image taken at the same time point (e.g., the first image taken on May 1 includes an RGB image, the second image taken on May 1 includes an NIR image, the third image taken on May 1 includes an NDVI image, etc.).

[0082] The entire geographic area covered by multiple images can be the same (or nearly the same) geographic area associated with the image used to generate the crop type model in box 202. In other words, the crop type model generated in box 206 may have been specifically developed and customized for use on the image in box 210. Such a crop type model can also be referred to as a localized or localized crop type model. The multiple images obtained in box 210 can also be associated with the same time period as the crop type model. Continuing the example above, the crop type model generated in box 206 can be associated with the United States and the years 2008–2016 (because the images used to train and build the model are images of the United States taken during the years 2008–2016), and the multiple images in box 210 can similarly be images of the United States taken during the years 2008–2016.

[0083] Each image within the image set may depict the same land location (at the same orientation and distance from the surface), differing only in that the images differ from each other in multispectral and / or time-series content. Therefore, each image within the image set can be the “same” image, differing only in that land surface features may vary at different times and / or in different spectral / color composition schemes. In some embodiments, images in the image set that include ground real data in block 202 may have similar characteristics.

[0084] Then, at box 212, the image of box 210 can be initially filtered by image filtering logic 120. In some embodiments, box 212 may be similar to box 204, except that the image being processed is the image of box 210 instead of the image of box 202. In other embodiments, box 212 may be optional if the image is captured (or recaptured as needed) to ensure that there are no clouds or other obstacles in the image.

[0085] Next, in box 214, crop type prediction logic 122 (in some embodiments, with the assistance of crop boundary detection logic 130) can be configured to determine a crop type heatmap for each (filtered) image set in the plurality of image sets obtained in box 210. For each image set in the plurality of image sets, that image set can be provided as input to the crop type model generated in box 206, and in response, the crop type model can provide a prediction / determination of (one or more) crop types within each crop region based on each pixel or each crop region. Each pixel (or crop region) of the heatmap can indicate the relative or absolute probability of a particular (one or more) crop type. In some embodiments, the heatmap can be vectorized according to a raster format.

[0086] A single crop region may have one or more predicted crop types. If a crop type heatmap is visually presented, then each of the multiple crop types can be assigned a different color, and the intensity / shading of a particular color overlaid on the image can indicate, for example, the statistical probability of the accuracy of the crop type prediction. As another example, crop type and / or prediction intensity / accuracy can be expressed as text in the image.

[0087] Multispectral and time-series imagery, including image sets targeting the same geographic region, allows for the detection of changes in specific land surface features over time. This helps determine whether a particular area is more likely to be a crop region and what crop(s) might be growing within that region. For example, crop color may change throughout the growing season. Crop fields may look different before planting, during the growing season, and after harvest. Specific patterns of crop color change over time can indicate the type of crop planted (e.g., wheat, soybeans, corn, etc.). When a crop is planted and / or harvested can indicate the type of crop planted. If a first type of crop is planted in a given crop field in the first year, and then a second type of crop, different from the first type, is planted in the same crop field in the second year, then the changes detected between those two years can indicate that the geographic location associated with that crop field is likely a crop region. Different crop types may have different planting pattern characteristics (e.g., the distance between adjacent planting rows may differ for different crop types).

[0088] Next, at box 216, crop type classification logic 126 can be configured to classify crop types of a crop region based on crop type heatmaps for each of the multiple image sets in box 210. In some embodiments, if more than one crop type is predicted for a given crop region, a majority voting rule can be applied, whereby the crop type with the highest probability among the predicted crop types can be selected as the crop type for the given crop region. If no dominant majority crop type is predicted (e.g., a crop type is predicted with a probability of 70% or higher), then the given crop region can be divided into multiple crop sub-regions, where each crop sub-region is assigned a corresponding crop type from the multiple crop types predicted for the given crop region. For example, if a given crop region has a 30% probability prediction of a first crop type, a 40% probability prediction of a second crop type, and a 30% probability prediction of a third crop type, then the margin of error for the probability could be the absence of a dominant crop type prediction. In this case, the given crop region can be subdivided into first, second, and third crop sub-regions, and assigned the first, second, and third crop types, respectively.

[0089] In an alternative embodiment, supplementary knowledge can be associated with crop type heat. Figure 1 This is used to make a final classification of crop types for crop regions. For example, if certain crop types do not or cannot grow simultaneously in the same geographical location, then if such incompatible crop types are predicted for the same crop region, then one or more crop types that are less likely to be planted in that geographical location can be ignored.

[0090] Crop boundaries associated with each crop region / field / subfield can be determined or identified at sub-meter (ground) resolution, approximately 0.15 to 0.2 meters resolution, less than 0.5 meters resolution, less than approximately 0.2 meters resolution, and so on. By extension, the crop type classification for each crop region / field / subfield can also be considered to be classified at sub-meter ground resolution, approximately 0.15 to 0.2 meters resolution, less than 0.5 meters resolution, less than approximately 0.2 meters resolution, and so on.

[0091] In some embodiments, at least some of the images in the image set associated with a specific portion of the overall geographic area of ​​interest (e.g., the images obtained in box 210) may have different resolutions from each other and / or lower resolutions than those associated with the crop type classification output by crop type classification logic 126. For example, even if at least some of the images provided as input have a ground resolution of 5 meters, the output containing crop types may be classified at a ground resolution of less than one meter (less than one meter per pixel) or 0.1 meters (0.1 meters per pixel).

[0092] Crop boundaries can define tightly formed areas. Crop boundaries can include crop field boundaries, or, where sufficient information and / or prior knowledge exists in the image set, crop subfield boundaries. Crop field boundaries can define crop fields, which can include physical areas demarcated by fences, permanent waterways, woodlands, roads, etc. Crop subfields can include subsets of crop fields, where a portion of the physical area of ​​a crop field primarily contains a specific crop type that differs from the primary crop type in another portion of the physical area of ​​the crop field. Each distinct crop type portion of a physical area can be considered a crop subfield. Therefore, a crop field can contain one or more crop subfields. For example, a crop field can include a first crop subfield of maize and a second crop subfield of soybeans.

[0093] In some embodiments, the crop type heatmap provided by the crop type prediction logic 122 can indicate the probability of one or more crop types for each crop region, and the crop type classification logic 126 can be configured to make a final determination of which crop type among the one or more crop types predicted for the crop region to assign to that pixel associated with the crop region.

[0094] In the case of classifying crop types into crop subfield levels for all image sets, processing 200 can proceed to box 218, where post-detection logic 128 can be configured to perform one or more post-detection activities based on the classified crop / crop subfields for all image sets (e.g., the entire geographic region). For each crop field / subfield with a classified crop type, post-detection activities may include, but are not limited to, calculating the area of ​​the crop field / subfield, assigning a unique identifier to the crop field / subfield (e.g., a unique computer-generated identifier (GUID) that will never be reused in another crop field / subfield), classifying the crop field / subfield within a classification system (e.g., crop fields / subfields may be classified, assigned, labeled, or associated with a specific continent, country, state, county, etc.), and / or generating associated metadata for storage, retrieval, search, and / or update activities. In some embodiments, post-detection activities may also include overlaying indications of the identified crop fields / subfields and crop types onto the original image to visually represent the crop type classification results, or otherwise visually enhance the original image with the detected information. The data generated by the post-detection activities can be maintained in database 110.

[0095] In some embodiments, for each image set, the post-detection logic 128 can be configured to generate a new image (also referred to as a crop indicator image) that depicts an original image (e.g., including at least one image from a plurality of images in the image set) that covers an indicator of a defined crop type (one or more). Figure 3A Image 340 shown is an example of a new image.

[0096] Figure 3A Various images that can be used or generated in the crop type classification process of this disclosure, according to some embodiments, are depicted. Image 300 may include examples of low-resolution ground reality data. Image 300 may have a resolution of 30 meters per pixel, be an example of a USDA CDL data image, etc. Image 300 may include indicators 302 and 304, which indicate the location of crop areas and optionally indicate one or more crop types of the crop areas. Because image 300 includes a low-resolution image, the location of crop areas and the crop type classification for a particular location may be at most approximate.

[0097] Image 310 may include examples of images from multiple image sets (e.g., images acquired / received within box 210). Image 310 may include high-resolution images acquired annually, etc., with a resolution of, for example, 0.15 meters per pixel. In some embodiments, images 300 and 310 may be associated with the same geographic location. Image 330 may also include examples of images from multiple image sets. Images 310 and 330 may include images from the same image set. Image 330 may include examples of low-resolution images, time-series images, monthly acquired images, etc.

[0098] As described above, crop boundaries can be identified during crop type classification. Image 320 depicts a visual illustration of the crop boundaries that can be identified in image 310. Image 320 may include image 310, which is overlaid with indications of the identified crop boundaries 322, 324, 326, and 328. Image 320 may include a high-resolution image at, for example, a resolution of 0.15 meters per pixel.

[0099] Using crop boundaries identified for the image set, the images in the image set can be additionally used to determine one or more crop types for each identified crop boundary. Image 340 may include images 310 or 320, which have indications of crop type classification for the included individual crop boundaries. The crop types for crop boundaries 322, 324, 326, and 328 are “grapes,” “corn,” “soybeans,” and “grapes,” respectively. Image 340 may include high-resolution images (e.g., at 0.15 m / pixel).

[0100] If you are performing a viewing, searching, or other activity involving a specific crop field / subfield or crop type, this newly generated image can be displayed to the user.

[0101] Next, at box 220, post-detection logic 128 can be configured to determine whether to update the crop type classification. The update can be triggered based on factors such as the availability of new images (e.g., near real-time changes to one or more crop boundaries, new growth periods, etc.), time / date events (e.g., New Year, new growth period), sufficient time elapsed since the last update, or some preset time period (e.g., regularly, weekly, bi-weekly, monthly, quarterly, annually, etc.). If an update is to be performed (the "no" branch at box 220), then process 200 can return to box 210. If no update is to be performed (the "no" branch at box 220), then process 200 can proceed to boxes 222, 224, and 226.

[0102] At box 222, the post-detection logic 128 can be configured to provide crop type data viewing and search functionality. Application programming interfaces (APIs), websites, applications, etc., can be implemented to allow users to access crop type data in various ways. For example, users can search for all crop regions classified as a specific crop type, crop regions within a specific country classified as a specific crop type, crop region sizes by crop type, crop yields of different crop types, crop management practices for different crop types, or any other search parameters. Images overlaid with crop boundaries and crop type indicators can be displayed to the user. For example, users can perform searches and view crop type data via device 112.

[0103] At box 224, post-detection logic 128 can be configured to facilitate modifications by authorized users to the crop type classification of a specific crop field / subfield that has been automatically identified. Farmers growing crops in a specific crop field / subfield may notice that the crop types in the database for that crop field / subfield are incorrect or incomplete, and may manually label the images with the correct crop type(s)(s). The modification capability can be similar to generating human-labeled images in box 202. The provided modifications (which may require approval) can then be used to update database 110. The provided modifications can also be used as ground-based data to refine the crop type model.

[0104] The determined crop type classification can be extended for a variety of uses. At box 226, the post-detection logic 128 can be configured to perform one or more of the following based on the crop type classification and / or crop characteristics detected during the execution of the crop type classification: estimating crop yield for each crop type (e.g., each crop type, each county, each crop type and county, each crop type and country, etc.); determining crop management practices for each crop type (e.g., estimating harvest date, determining when to fertilize, determining the type of fertilizer); diagnosing leaf drop disease; controlling or curing crop diseases; identifying different varieties within a crop type; determining crop attributes (e.g., based on the orientation of the planted crop); and so on.

[0105] Figure 3B An example representation of crop yield estimates calculated based on crop type classification data according to some embodiments is depicted. Image 350 may include the same image as image 340, supplemented with crop yield estimates for each crop field / subfield. As shown, corn yields are higher per acre than grape or soybean yields. If similar estimates are calculated for each different crop type for all crop fields / subfields within a geographic region (e.g., the United States), then the total crop yield for each crop type can be known.

[0106] In this way, a complete database of crop fields / subfields (or crop boundaries) with categorized crop types for a given geographic region (e.g., county, state, country, continent, planet) can be automatically generated at sub-meter resolution, and it can remain up-to-date over time with minimal supervision. For multiple geographic regions, assuming the existence of ground-based real-world data for each geographic region within the multiple geographic regions, then processing 200 can be performed for each geographic region across the multiple geographic regions.

[0107] Figure 4 A flowchart illustrating an example process 400 according to some embodiments is provided, which can be implemented by system 100 to automatically detect crop boundaries (and correspondingly, crop regions / fields / subfields) in an image. Figure 4 The crop boundaries detected in box 416 can include the above-mentioned targets Figure 2 The crop boundary detection results mentioned in the ground truth data in box 202. In some embodiments, the crop boundary detection performed by the crop type model during the generation of the crop type heatmap may include at least the following: Figure 4 Boxes 414 and 416.

[0108] At box 402, training logic 124 can be configured to acquire or receive ground truth data, which includes multiple land surface images with identified crop boundaries. Multiple images including ground truth data can be selected to cover images with various land features, crop boundaries, etc., thereby training / generating a detection model capable of handling different land features and crop boundaries that may exist in the images to be detected. In some embodiments, the multiple images can be similar to those described above for... Figure 2 The difference between the images discussed in box 202 and those in the box is that they identify crop boundaries rather than classify crop types.

[0109] In some embodiments, ground-based data for crop boundary detection may include existing images with identified crop boundaries (or crop areas), wherein the crop boundaries (or crop areas) may be identified at a low (ground) resolution (e.g., a resolution greater than 1 meter, a resolution of 3 to 250 meters, a resolution of 30 meters, etc.). Such images may be refreshed frequently, such as daily to every two weeks. Because crop boundary identification is at a low resolution, such identification may be considered “noisy,” approximate, or inaccurate. Examples of existing images with low-resolution identified crop boundaries may include, but are not limited to, USDA CDL data, FSA CLU data, government-collected data, data based on sampling or surveys, farmer reports, etc. Existing images with identified crop boundaries may be obtained by server 104, stored in database 106, and / or provided to server 108.

[0110] In some embodiments, ground-based data may include CDL and CLU data (as discussed above) and / or human-labeled data. Human-labeled data may include crop boundaries in images that have been manually identified, labeled, or annotated by, for example, a user 114 via a graphical user interface (GUI) mechanism provided on device 112. Such manual annotations may have a higher (ground) resolution than the resolution that may be associated with the CDL and / or CLU data. For example, manually labeled images may be obtained from device 116. Training logic 124 may facilitate image selection, presentation of selected images, use of human-labeled images, and so on. Ground-based data may also be referred to as training data, model building data, model training data, and so on.

[0111] In some embodiments, the time period and / or (one or more) geographic regions associated with the ground truth data may be the same (or approximately the same) as the time period and / or (one or more) geographic regions associated with the image for which crop boundaries are to be detected (at box 216). For example, for images taken between 2008 and 2016 for which operations are to be performed at box 216, CLU data from 2008 may be used, CDL data from 2008–2016 may be used, and human-labeled data may include images taken between 2008 and 2016. The CLU and CDL data may include image data of the United States, and the human-labeled data may also include image data of the United States.

[0112] Next, at box 404, image filtering logic 120 can be configured to perform preliminary filtering on one or more images that include ground reality data. In some embodiments, preliminary filtering may include monitoring for clouds, shadows, haze, fog, atmospheric obstacles, and / or other land surface obstacles included in the image on a per-pixel basis. Box 404 may be similar to box 204, except that the image being filtered is an image that includes ground reality data from box 402.

[0113] At box 406, using the obtained and optionally pre-filtered or corrected ground truth data, the resulting ground truth data can be applied to one or more machine learning techniques / systems to generate or construct a crop / non-crop model. The machine learning techniques / systems may include, for example, convolutional neural networks (CNNs) or supervised learning systems. The crop / non-crop model can be configured to provide a probabilistic prediction of crop or non-crop for each pixel corresponding to a specific geographic location associated with the image set provided as input. The crop / non-crop model may include a portion of crop boundary detection logic 130. Since the ground truth data includes images with accurately identified crop boundaries, the machine learning techniques / systems can learn which land surface features in the images indicate crops or non-crops. When such knowledge is sufficiently detailed and accurate, it can then be used to automatically identify crop boundaries in images where crop boundaries may be unknown.

[0114] In some embodiments, crop / non-crop models may be associated with a specific geographic area—the same geographic area captured in an image that includes ground-based data. For example, a crop / non-crop model may be specific to a particular county within the United States. Similarly, crop / non-crop models may be associated with a specific time period—the same time period associated with an image that includes ground-based data. As the geographic area increases, data inconsistencies or regional differences may arise, which could lead to a decrease in the accuracy of crop / non-crop models.

[0115] Next, training logic 124 can be configured to determine whether the accuracy of the crop / non-crop model is equal to or exceeds a predetermined threshold. The predetermined threshold could be 70%, 80%, 85%, 90%, etc. If the model's accuracy is less than the predetermined threshold (the negative branch of box 408), then process 400 can return to box 402 to obtain / receive additional ground truth data to apply to the machine learning technique / system to refine the current crop / non-crop model. Providing additional ground truth data to the machine learning technique / system includes providing additional supervised learning data so that the crop / non-crop model can be better configured to predict whether a pixel depicts a crop (or is located within a crop field) or a non-crop (or is not located within a crop field). One or more iterations of boxes 402-408 can be performed until a sufficiently accurate crop / non-crop model can be built.

[0116] If the model's accuracy equals or exceeds a predetermined threshold (the branch in box 408), then the crop / non-crop model can be considered acceptable for unsupervised or automatic crop / non-crop detection in images where crop boundaries (or crop fields) are unknown. At box 410, multiple images to be applied to the crop / non-crop model for automatic detection can be acquired or received. Multiple image sets can be acquired, each associated with the same (or nearly the same) geographic location and time period as the image in box 402. If the crop boundary detection results in box 202 are used as ground truth data, then the acquired image in box 410, the image in box 402, and the image in box 202 can all be associated with the same (or nearly the same) geographic location and time period. The multiple images can be those captured by device 116, images from Landsat 7, images from Landsat 8, Google Earth images, images at one or more different resolutions, and / or images acquired at one or more different frequencies.

[0117] Then, at box 412, the image of box 410 can be initially filtered by image filtering logic 120. In some embodiments, box 412 may be similar to box 404, except that the image being processed is the image of box 410 instead of the image of box 402. In other embodiments, box 412 may be optional if the image is captured (or recaptured as needed) to ensure that there are no clouds or other obstacles in the image.

[0118] Next, at box 414, crop boundary detection logic 130 can be configured to determine a crop / non-crop heatmap for each (filtered) image set in the plurality of image sets obtained in box 410. For each image set in the plurality of image sets, that image set can be provided as input to the crop / non-crop model generated in box 406, and in response, the crop / non-crop model can provide a prediction / determination of whether a crop is depicted on a per-pixel basis. In other words, the presence (or absence) of a crop is predicted at a specific location within a specific portion of a geographic area associated with the corresponding image set. Each pixel of the heatmap can indicate the relative or absolute probability of a crop or the absence of a crop. In some embodiments, the probability prediction of a crop / non-crop provided by the heatmap can be indicated by using specific colors, patterns, shadows, hues, or other indicators overlaid on the original image. For example, the zero probability of a crop can be indicated by the absence of an indicator, the highest probability of a crop can be indicated by the darkest or brightest red shadow, and probabilities in between can be appropriately graded using colors, shadows, hues, patterns, etc., between no indicator and the darkest / brightest red.

[0119] At box 416, crop boundary detection logic 130 can be configured to determine crop boundaries based on crop / non-crop heatmaps for each of the multiple image sets in box 410. Besides using crop / non-crop heatmaps, crop boundary location determination can also be based on prior knowledge, the application of denoising techniques, clustering and region growing techniques, and so on.

[0120] In some embodiments, crop boundary detection logic 130 can be configured to use prior knowledge information when determining crop boundaries. Prior knowledge information may include, but is not limited to, known locations of roads, waterways, woodlands, buildings, parking lots, fences, walls, and other physical structures; known information about agricultural or farm practices, such as specific boundary shapes resulting from specific agricultural / farm practices near a geographic location associated with the image set (e.g., straight or circular boundaries in cases where pivot irrigation is known to be used); crop type; and so on. Denoising or filtering techniques can be implemented to determine and / or refine crop boundaries. Applicable denoising or filtering techniques may include, but are not limited to, techniques for smoothing initially determined crop boundaries (e.g., since boundaries tend to be linear or follow geometric shapes in the absence of physical barriers). Similarly, clustering and region growing techniques can be employed to determine or refine crop boundaries. Unsupervised clustering and region growing techniques can be used to reclassify stray pixels from non-crops to crops and vice versa in regions where the number of pixels differs significantly from the number of pixels around them. For example, if several pixels are classified as non-crops within a larger region classified as crops, then those pixels can be reclassified as crops.

[0121] Crop boundaries can be determined or identified at sub-meter (ground) resolution, approximately 0.15 to 0.2 meters resolution, less than 0.5 meters resolution, less than approximately 0.2 meters resolution, and so on. Crop boundaries can define tightly formed areas. Crop boundaries can include crop field boundaries, or, where sufficient information and / or prior knowledge exists in the image set, crop subfield boundaries. Crop field boundaries can define crop fields, which can include physical areas demarcated by fences, permanent waterways, woodlands, roads, etc. Crop subfields can include subsets of crop fields, where a portion of the physical area of ​​a crop field primarily contains a specific crop type that differs from the dominant crop type in another portion of the physical area of ​​the crop field. Each distinct crop type portion of a physical area can be considered a crop subfield. Therefore, a crop field can contain one or more crop subfields. For example, a crop field can include a first crop subfield of maize and a second crop subfield of soybeans.

[0122] In some embodiments, the crop / non-crop heatmap can indicate the likelihood of crop regions, and the crop boundary detection logic 130 can be configured to make a final determination of which pixels in the crop / non-crop heatmap, which are indicated to likely depict crops, include crop field(s) or crop subfield(s). The perimeter of a crop field or subfield defines the boundary of the associated crop field or subfield.

[0123] In this way, crop boundaries at the level of crop subfields can be automatically detected. Such crop boundary detection results can be used as ground truth data in box 202. In some embodiments, the crop boundary detection techniques (or portions thereof) discussed herein can be included in the crop type model generated in box 206.

[0124] Figure 5 A flowchart illustrating an example process 500, which, according to some embodiments, can be implemented by system 100 to perform crop type classification using an existing crop type classification model and modify the crop type classification model as needed, is provided. In some embodiments, blocks 502, 504, 506, and 508 may be similar to... Figure 2 The differences between the boxes 210, 212, 214, and 216 are that the image sets for which crop type classification is performed can be associated with geographic regions and / or time periods that are different from those associated with the crop type model used in box 506.

[0125] Continuing the example above, the crop type model used in box 506 is generated based on images of the United States taken between 2008 and 2016, while the image set in box 502 could be images of the United States taken between 2000 and 2007. As another example, the image set in box 502 could be images of geographic regions other than the United States (e.g., foreign countries, China, Mexico, Canada, Africa, Eastern Europe, etc.). As yet another example, the image set in box 502 could be images of a specific geographic region taken over several years other than 2008-2016. Even if the crop type model may not be precisely cropped for the images being processed, it can be used as a starting point because the model already exists. For countries outside the United States, there may not be sufficient publicly available ground-based data to easily generate crop type models.

[0126] In some embodiments, blocks 510-512 may be executed concurrently with, before, or after blocks 502-508. Blocks 510 and 512 may be similar to... Figure 2 The respective boxes 202 and 204. The ground truth data obtained in box 510 may be associated with the same (or nearly the same) geographic area and / or time period as the image set in box 502. In some embodiments, the amount of ground truth data in box 510 may differ from the amount of ground truth data in box 202. Because there may be little or no government / publicly available crop data for countries outside the United States or for earlier years, only a small amount of ground truth data may be available.

[0127] At box 514, training logic 124 can be configured to evaluate the accuracy of at least one subset of crop types predicted using existing crop type models in box 508 by comparing them with crop types identified in (filtered) ground-based data provided in boxes 510, 512. In some embodiments, one or more crop types classified for the same (or nearly the same) geographic region from two sets of identified crop type data can be compared with each other.

[0128] If the accuracy of the predicted crop type is equal to or exceeds a threshold (box 514 is the branch), then processing 500 can proceed to boxes 516-522. This threshold may include preset thresholds such as 75%, 80%, 85%, 90%, etc. An existing crop type model can be considered suitable (or sufficiently accurate) for the specific geographic area and time period associated with the image of interest in box 502. In some embodiments, boxes 516, 518, 520, 522, and 524 can be similar to... Figure 2The differences between boxes 218, 220, 222, 224, and 226 lie in the fact that the crop type classifications of interest are those identified in box 508. In box 518 (a branch), if the crop type classifications need to be updated, then process 500 can return to box 502. For crop type classification updates, once the model's applicability / accuracy has been initially confirmed, it is not necessary to repeat boxes 510, 512, and 514.

[0129] If the accuracy of the predicted crop boundary is less than a threshold (the negative branch of box 514), then processing 500 can proceed to box 524. A new crop type model can be generated that is associated with the same (or nearly the same) geographic region and time period as the image obtained in box 502. The new crop type model may include modifications to an existing crop type model, or it may include a model trained only with data corresponding to the geographic region and time period matching the image of interest. At box 524, training logic 124 can be configured to generate a new crop type model based on (filtered) ground truth data applied to one or more machine learning techniques / systems in box 512. Box 524 can be similar to... Figure 2 The box 206.

[0130] Next, at box 526, the accuracy of the new crop / non-crop model can be evaluated. If the accuracy is less than the threshold (the negative branch of box 526), ​​then additional ground truth data can be obtained or received at box 528, and training / refinement / construction of the new crop type model can continue by returning to box 524. If the accuracy is equal to or exceeds the threshold (the positive branch of box 526), ​​then process 500 can proceed to box 506 to use the new crop type model with the (filtered) image set from box 504 to generate a crop type heatmap associated with the (filtered) image set. In cases where a new crop type model is generated due to insufficient accuracy of the existing crop type model, it is not necessary to repeat boxes 510, 512, and 514.

[0131] In this way, it is also possible to cost-effectively, accurately, and automatically classify crop types in crop fields / subfields located outside the United States and / or for periods other than the most recent few years. Current and past (to the extent that aerial imagery data is available) crop fields / subfields across the planet can be classified by crop type. Depending on the availability of aerial imagery, historical aerial imagery potentially dating back 20 to 40 years can be applied to crop type models to retrospectively classify the crop types in these images. The ability to retrospectively classify historical imagery can facilitate the identification of various trends (e.g., crop land use, crop yield, etc.).

[0132] Figure 6 Example devices that can be implemented in the system 100 of this disclosure according to some embodiments are depicted. Figure 6 The devices may include at least a portion of any one of server 104, database 106, server 108, database 110, device 112, and / or device 116. Platform 600, as shown, includes a bus or other internal communication device 615 for transmitting information, and a processor 610 coupled to the bus 615 for processing information. The platform also includes random access memory (RAM) or other volatile storage device 650 (alternately referred to herein as main memory) coupled to the bus 615 for storing information and instructions to be executed by the processor 610. Main memory 650 may also be used to store temporary variables or other intermediate information during instruction execution by the processor 610. Platform 600 also includes read-only memory (ROM) and / or static storage device 620 coupled to the bus 615 for storing static information and instructions for the processor 610, and data storage device 625, such as a disk, optical disk and its corresponding disk drive, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card). Data storage device 625 is coupled to the bus 615 for storing information and instructions.

[0133] Platform 600 may also be coupled to display device 670, such as a cathode ray tube (CRT) or liquid crystal display (LCD) coupled to bus 615 via bus 665 to display information to a computer user. In embodiments where platform 600 provides computing power and connectivity to the created and installed display device, display device 670 may display an image overlaid with crop field / subfield information as described above. Alphanumeric input device 675, including alphanumeric and other keys, may also be coupled to bus 615 via bus 665 (e.g., via infrared (IR) or radio frequency (RF) signals) to transmit information and command selections to processor 610. Additional user input devices are cursor control devices 680, such as a mouse, trackball, stylus, or cursor arrow keys, which are coupled to bus 615 via bus 665 to transmit directional information and command selections to processor 610 and to control cursor movement on display device 670. In embodiments utilizing a touchscreen interface, it should be understood that the display 670, input device 675, and cursor control device 680 can all be integrated into the touchscreen unit.

[0134] Another component that may optionally be coupled to platform 600 is communication device 690 for accessing other nodes of the distributed system via a network. Communication device 690 may include any of many commercially available networking peripherals, such as those used for coupling to Ethernet, Token Ring, the Internet, or a wide area network. Communication device 690 may also be a null-modem connection, or any other mechanism providing connectivity between platform 600 and the outside world. Note that... Figure 6 Any or all components of such a system, as well as associated hardware, may be used in various embodiments of this disclosure.

[0135] The above-described process is based on computer software and hardware. The described techniques can be configured as machine-executable instructions contained in a tangible or non-transitory machine-readable storage medium, which, when executed by a machine, will cause the machine to perform the described operations. Furthermore, the process can be implemented in hardware, such as application-specific integrated circuits (ASICs) or other hardware.

[0136] Tangible machine-readable storage media include any mechanism that provides (e.g., stores) information in a non-transitory form accessible to a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device having one or more processors, etc.). For example, machine-readable storage media include recordable / non-recordable media (e.g., read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory devices, etc.).

[0137] The above description of illustrative embodiments of the invention, including those described in the abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments and examples of the invention have been described herein for illustrative purposes, various modifications can be made within the scope of the invention, as will be recognized by those skilled in the art.

[0138] These modifications can be made to the invention based on the above detailed description. The terminology used in the appended claims should not be construed as limiting the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention will be determined entirely by the appended claims, which will be interpreted according to the established principles of claim interpretation.

Claims

1. A method for crop type classification, comprising: The computing device receives input containing one or more search parameters, wherein the one or more search parameters include one or more of county, size, shape, and identifier; as well as The computing device presents a crop indicator image depicting a portion of a geographic region, wherein the portion of the geographic region is selected based on the one or more search parameters, and wherein the crop indicator image comprises at least one image from a set of multispectral and time-series images overlaid with an indication of a crop type classification determined for a specific location depicted in at least one image; The crop type classification for a specific location depicted in the at least one image is determined by the following: Obtain multiple image sets associated with a geographic region and a time period, wherein each image set in the multiple image sets includes multispectral and time-series images that depict a specific portion of the geographic region during the time period; Predict one or more crop types to grow in each specific location within a specific portion of a geographic region associated with an image set in the plurality of image sets; as well as Based on one or more crop types predicted for a specific location, determine the crop type classification for each specific location; Determining the crop type classification for each specific location involves classifying crop types in crop regions based on images using the following methods: In response to determining that the crop types predicted for a specific location include the majority of predicted crop types, the majority of predicted crop types are selected as the crop type classification; and In response to the determination that the crop types predicted for a specific location do not include the major majority of predicted crop types: Divide the corresponding specific location into multiple sub-specific locations; and Each corresponding sublocation is classified into the corresponding crop type in the crop type predicted for that location.

2. The method of claim 1, further comprising providing a user interface through which a user can manually modify the crop type classification; Receive crop type classification modifications from users; and The database storing the at least one image and its crop type classification is updated using crop type classification modification.

3. The method of claim 1, wherein predicting the one or more crop types growing at each specific location comprises: Predicting the presence of crops at a specific location; Based on the predicted existence of crops at specific locations, determine the location of crop boundaries within a specific part of a geographic area; as well as Predict the one or more crop types that will grow within each defined crop boundary location.

4. The method of claim 1, further comprising the computing device presenting an estimate of crop yield for each specific location based on a crop type classification determined for that specific location.

5. The method of claim 1, further comprising the computing device presenting guidance on crop management practices for each specific location based on a crop type classification determined for that specific location.

6. The method of claim 1, wherein determining the crop type classification for each particular location includes determining the crop type classification to sub-meter ground resolution for each particular location.

7. The method of claim 1, wherein predicting the one or more crop types growing at each particular location comprises applying the image set to one or more machine learning systems.

8. The method of claim 7, wherein the one or more machine learning systems are configured to predict the one or more crop types to grow in each particular location after supervised training on real-world ground data.

9. The method of claim 8, wherein the ground real data includes one or more of the following: government crop data, publicly available crop data, images of crop areas identified at low ground resolution, images of crop types identified at low ground resolution, images of manually identified crop boundaries, images of manually identified crop boundaries and crop types, crop survey data, sampled crop data, and farmer reports.

10. The method of claim 8, wherein a first resolution of a first image of the image set is different from a second resolution of a second image of the image set, the first resolution is lower than a third resolution of the crop indicator image, and a fourth resolution of at least a portion of the ground truth data is lower than the third resolution of the crop indicator image.

11. The method of claim 1, wherein predicting the one or more crop types growing at each specific location comprises: For each specific location, the changes in pixels associated with that specific location over time are analyzed in the time series image, where a specific pattern of pixel change is associated with at least one crop type.

12. A method for classifying crop types, comprising: Obtain multiple image sets associated with a geographic region and a time period, wherein each image set in the multiple image sets includes multispectral and time-series images that depict a specific portion of the geographic region during the time period; Predict one or more crop types to grow in each specific location within a specific portion of a geographic region associated with an image set in the plurality of image sets; Based on one or more crop types predicted for a specific location, determine the crop type classification for each specific location; and Generate a crop indication image comprising at least one image from the multispectral and time-series images of the image set, overlaid with an indication of crop type classification determined for a specific location; Provides a user interface that allows users to manually modify crop type classifications; Predicting the one or more crop types that will grow at each specific location includes applying the image set to one or more machine learning systems, wherein the one or more machine learning systems include convolutional neural networks (CNNs); and The one or more machine learning systems are configured to predict the one or more crop types to grow at each specific location after being trained in a supervised manner on real-world ground data. The determination of crop type classification for each specific location includes: for each specific location, selecting the majority of predicted crop types from the crop types predicted for the corresponding specific location, wherein the majority of predicted crop types is the crop type classification; for each specific location, if there is no majority of predicted crop type, the corresponding specific location is divided into multiple sub-specific locations, and each of the multiple sub-specific locations is classified as the corresponding crop type among the crop types predicted for the specific location.

13. The method of claim 12, wherein predicting the one or more crop types growing at each specific location comprises: Predicting the presence of crops at a specific location; Based on the predicted existence of crops at specific locations, determine the location of crop boundaries within a specific part of a geographic area; as well as Predict the one or more crop types that will grow within each defined crop boundary location.

14. The method of claim 12, wherein determining the crop type classification for each particular location includes determining the crop type classification to sub-meter ground resolution for each particular location.

15. The method of claim 12, wherein the ground real data includes one or more of the following: government crop data, publicly available crop data, images of crop areas identified at low ground resolution, images of crop types identified at low ground resolution, images of manually identified crop boundaries, images of manually identified crop boundaries and crop types, crop survey data, sampled crop data, and farmer reports.

16. The method of claim 12, wherein a first resolution of a first image of the image set is different from a second resolution of a second image of the image set, the first resolution is lower than a third resolution of the crop indicator image, and a fourth resolution of at least a portion of the ground truth data is lower than the third resolution of the crop indicator image.

17. The method of claim 12, wherein predicting the one or more crop types growing at each specific location comprises: For each specific location, the changes in pixels associated with that specific location over time are analyzed in the time series image, where a specific pattern of pixel change is associated with at least one crop type.

18. The method of claim 12, further comprising estimating crop yield for each specific location based on a crop type classification determined for that specific location.

19. The method of claim 12, further comprising determining crop management practices for each specific location based on a crop type classification determined for that specific location.

20. The method of claim 12, further comprising: Display crop indicator images on user-accessible devices; as well as Modification of a specific instruction received from a user for a crop type classification determined for a specific location, wherein the modification includes manual reclassification of the crop type for the specific location associated with the specific instruction.

21. A method for classifying crop types, comprising: Receive input containing one or more search parameters, wherein the one or more search parameters include one or more of county, size, shape and identifier; as well as Presenting crop indicator images depicting a portion of a geographic region, wherein said portion of the geographic region is selected based on said one or more search parameters, and wherein the crop indicator images comprise at least one image from a set of images associated with the geographic region, overlaid with an indication of a crop type classification determined for a specific location depicted in at least one image; The crop type classification for a specific location depicted in the at least one image is determined by the following: Obtain multiple image sets associated with a geographic region and a time period, wherein each image set in the multiple image sets includes multispectral and time-series images that depict a specific portion of the geographic region during the time period; Predict one or more crop types to grow in each specific location within a specific portion of a geographic region associated with an image set in the plurality of image sets; as well as Based on one or more crop types predicted for a specific location, determine the crop type classification for each specific location; Predicting the one or more crop types that will grow at each specific location includes applying the image set to one or more machine learning systems, wherein the one or more machine learning systems include convolutional neural networks (CNNs); and The one or more machine learning systems are configured to predict the one or more crop types to grow at each specific location after being trained in a supervised manner on real-world ground data. The determination of crop type classification for each specific location includes: for each specific location, selecting the majority of predicted crop types from the crop types predicted for the corresponding specific location, wherein the majority of predicted crop types is the crop type classification; for each specific location, if there is no majority of predicted crop type, the corresponding specific location is divided into multiple sub-specific locations, and each of the multiple sub-specific locations is classified as the corresponding crop type among the crop types predicted for the specific location.

22. The method of claim 21, wherein the method further comprises: Provides a user interface that allows users to manually modify crop type classifications; Receive crop type classification modifications from users; as well as The database storing the at least one image and its crop type classification is updated using crop type classification modification.

23. The method of claim 21, wherein predicting the one or more crop types growing at each specific location comprises: Predicting the presence of crops at a specific location; Based on the predicted existence of crops at specific locations, determine the location of crop boundaries within a specific part of a geographic area; as well as Predict the one or more crop types that will grow within each defined crop boundary location.

24. The method of claim 21, wherein determining the crop type classification for each particular location includes determining the crop type classification to sub-meter ground resolution for each particular location.

25. The method of claim 21, wherein the ground real data includes one or more of the following: government crop data, publicly available crop data, images of crop areas identified at low ground resolution, images of crop types identified at low ground resolution, images of manually identified crop boundaries, images of manually identified crop boundaries and crop types, crop survey data, sampled crop data, and farmer reports.

26. The method of claim 21, wherein a first resolution of a first image of the image set is different from a second resolution of a second image of the image set, the first resolution is lower than a third resolution of the crop indicator image, and a fourth resolution of at least a portion of the ground truth data is lower than the third resolution of the crop indicator image.

27. The method of claim 21, wherein predicting the one or more crop types growing at each specific location comprises: For each specific location, the changes in pixels associated with that specific location over time are analyzed in the time series image, where a specific pattern of pixel change is associated with at least one crop type.

28. The method of claim 21, further comprising presenting an estimate of crop yield for each specific location based on a crop type classification determined for that specific location, or presenting guidance on crop management practices for each specific location based on a crop type classification determined for that specific location.

29. An apparatus for crop type classification, comprising one or more processors and a computer-readable storage medium storing a plurality of instructions, the instructions being responsive to execution by the one or more processors to cause the apparatus to perform the method of any one of claims 1 to 28.

Images