Method, apparatus, and computer-readable medium for extracting plant features from non-hyperspectral images

Abstract

A method, apparatus, and computer-readable medium for extracting plant features from non-hyperspectral images, including receiving non-hyperspectral images of plants captured by non-hyperspectral image capture devices, segmenting the non-hyperspectral images to identify image portions corresponding to the one or more plants, generating non-hyperspectral plant images based on the identified image portions, applying a specialized machine learning model (MLM) to the non-hyperspectral plant images to generate hyperspectral plant images corresponding to a spectral range, each hyperspectral plant image corresponding to a spectral subrange, the specialized MLM being trained based on pairs of hyperspectral and non-hyperspectral plant images and configured to perform hyperspectral conversion of non-hyperspectral plant images, extracting plant features based on applying at least one image processing technique to at least one hyperspectral plant image

Description

METHOD, APPARATUS, AND COMPUTER-READABLE MEDIUM FOR EXTRACTING PLANT FEATURES FROM NON-HYPERSPECTRAL IMAGESRELATED APPLICATION DATA

[0001] This application claims priority to U.S. Provisional Application No. 63 / 608,466, filed December 11, 2023, the disclosure of which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] Hyperspectral imaging is an imaging technique that gathers data across the electromagnetic spectrum. Hyperspectral sensors collect data as a set of images, with each image representing a wavelength range of the spectrum. The hyperspectral images can then be combined to form a “hyperspectral cube” which contains information across the entire spectrum for each pixel of an image.

[0003] Depending on the subject matter being imaged, hyperspectral data and images can be used to extract useful information. However, deployment of hyperspectral imaging devices and sensors is not efficient in most scenarios. In addition to added complexity and cost, hyperspectral systems require greater resources. Since hyperspectral imaging records information across a wide spectral range, the amount of data produced is much greater than simpler methods, such as RGB (Red-Green-Blue) imaging. This additional data must be stored and transmitted across information networks, consuming much more storage space and network bandwidth than traditional image capture devices.

[0004] Accordingly, improvements are needed in systems for hyperspectral analysis which do not require resource-intensive hyperspectral imaging.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Fig. 1 illustrates a flowchart for extracting plant features from non-hyperspectral images according to an exemplary embodiment.

[0006] Fig. 2 illustrates an example of an image capture environment according to an exemplary embodiment.

[0007] Fig. 3 illustrates an example of a non-hyperspectral image 300 captured by a non- hyperspectral image capture device according to an exemplary embodiment.

[0008] Fig. 4 illustrates a flowchart for segmenting the one or more non-hyperspectral images to identify a plurality of image portions corresponding to the one or more plants according to an exemplary embodiment.

[0009] Fig. 5 illustrates an example of instance segmentation according to an exemplary embodiment.

[0010] Fig. 6 illustrates an example of semantic segmentation according to an exemplary embodiment.

[0011] Figs. 7A-7B illustrate an example of the non-hyperspectral plant image generation process according to an exemplary embodiment.

[0012] Fig. 8 illustrates a system chart for the hyperspectral conversion process according to an exemplary embodiment.

[0013] Fig. 9 illustrates an example of hyperspectral conversion of a non-hyperspectral plant image according to an exemplary embodiment.

[0014] Fig. 10 illustrates the spectral range and subranges of the hyperspectral images according to an exemplary embodiment.

[0015] Fig. 11 illustrates a flow of pre-processing steps and post-processing steps according to an exemplary embodiment.

[0016] Fig. 12 illustrates a flowchart and system diagram of the plant feature extraction process according to an exemplary embodiment.

[0017] Figs. 13A-13E illustrate different plant parameters and / or metrics that can be extracted from the hyperspectral plant images according to an exemplary embodiment.

[0018] Fig. 14 illustrates an example of a determined chlorophyll index levels over time in one or more plants according to an exemplary embodiment.

[0019] Figs. 15A-15B illustrate graphs showing correlations for harvest weights of plants according to an exemplary embodiment.

[0020] Fig. 16 illustrates a flowchart for modifying plant growth according to an exemplary embodiment.

[0021] Fig. 17 illustrates examples of plant environment systems according to an exemplary embodiment.

[0022] Fig. 18 illustrates examples of plant growth modification actions 1800 according to an exemplary embodiment.

[0023] Fig. 19 illustrates a system flow diagram of the plant analysis, prediction, modification system according to an exemplary embodiment.

[0024] Fig. 20 illustrates a specialized computing environment configured to perform the described methods and implement the described systems according to an exemplary embodiment.DETAILED DESCRIPTION

[0025] While methods, systems, and computer-readable media are described herein by way of examples and embodiments, those skilled in the art recognize that methods, apparatuses, and computer-readable media for extracting plant features from non-hyperspectral images are not limited to the embodiments or drawings described. The drawings and description are not intended to be limited to the particular form disclosed. Rather, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the present disclosure and claims. Any headings used herein are for organizational purposes only and are not meant to limit the scope of the description or the claims. As used herein, the word “can” is used in a permissive sense (i.e., meaning having the potential to) rather than the mandatory sense (i.e., meaning must). Similarly, the words “include,” “including,” and “includes” mean including, but not limited to.

[0026] Hyperspectral analysis can be used to understand a plant’s features and state. This information can be leveraged to determine current plant health or future predicted plant health and current or future plant diseases, stresses, and growth trajectory. However, as discussed above, hyperspectral imaging of plants is resource-intensive, making it difficult and inefficient to deploy in most real-world greenhouses or grow environments.

[0027] Applicant has discovered a method, apparatus, and computer-readable medium for extracting plant features from non-hyperspectral images that solves the above-mentioned problems. The system disclosed herein utilizes non-hyperspectral images captured by non- hyperspectral imaging devices to minimize the resources, storage, and bandwidth required to capture images of plants. A novel method is further disclosed herein to convert the non- hyperspectral images into hyperspectral images. Plant metrics and features can then be extracted from the converted hyperspectral images.

[0028] Fig. 1 illustrates a flowchart for extracting plant features from non-hyperspectral images according to an exemplary embodiment. The steps shown in the flowchart can be performed by one or more computing devices of a hyperspectral plant image conversion and analysis platform (the “platform”). As discussed further in this disclosure, the process can include one or more additional steps. Additionally, the steps shown in the flowchart can be performed in any order, unless otherwise indicated.

[0029] At step 101 one or more non-hyperspectral images of one or more plants are received, the one or more non-hyperspectral images being captured by one or more non- hyperspectral image capture devices. The non-hyperspectral images can be received directly from the image capture devices or from an intermediary computer system on the network, such as a controller for a greenhouse, farm, or grow site. The one or more non-hyperspectral images can include a multiple non-hyperspectral images and multiple types of non-hyperspectral images.

[0030] As used herein, the term “non-hyperspectral images” refers to images that are not hyperspectral, meaning the images are not captured across a continuous spectral range using hyperspectral sensors or imagers. Non-hyperspectral images can include RGB (Red-Green- Blue) images, CMYK (Cyan-Magenta- Yellow-Key) images, IR (infrared) images, or multispectral images that utilize multiple discrete and / or discontinuous wavelengths. IR images can be captured, for example, with a camera having a long pass filter that only lets in light above a certain wavelength. When multiple or a plurality of non-hyperspectral images are received, the non-hyperspectral images can include one or more RGB images, one or more IR images, and / or other types of non-hyperspectral images.

[0031] Non-hyperspectral image capture devices include any image capture device configured to capture non-hyperspectral images, such as an RGB image capture device, CMYKimage capture device. Common RGB image capture devices include cameras, videocameras, digital cameras, photosensors, etc. The non-hyperspectral image capture devices can be any type of camera or sensor, such as an IR sensor.

[0032] Each non-hyperspectral image can include one or more channels, with each channel corresponding to a different (and discrete) wavelength. One of the distinctions between hyperspectral and non-hyperspectral images is that non-hyperspectral images have channels (also referred to as “bands”) that correspond to discrete spectral wavelengths, whereas hyperspectral images have channels that corresponds to a continuous spectral range. In other words, each pixel of a non-hyperspectral image has a discrete spectrum corresponding to one or more discrete wavelengths, whereas each pixel of a hyperspectral image has a continuous spectrum corresponding to a range of wavelengths. For example, an RGB image has three channels corresponding to red, green, and blue. A CMYK image has four channels corresponding to cyan, magenta, yellow, and the key color (usually black).

[0033] Fig. 2 illustrates an example of an image capture environment according to an exemplary embodiment. The environment 200 shown in figure is a greenhouse, but it is understood that the present system can be utilized in any type of grow environment, such as a greenhouse, outdoor farm, hydroponic farm, etc. The environment 200 includes one or more plants, such as plant 201. The plants can be any kind of plant raised in agriculture or otherwise, such as lettuce, spinach, tomatoes, microgreens, fruits, etc.

[0034] As shown in Fig. 2, one or more image capture devices, such as devices 202A, 202B, and 202C are used to capture the non-hyperspectral images 203. The image capture devices can be any type of image capture device discussed previously, such as RGB cameras, IR sensors, etc. The image capture devices can also include multiple different types of image capture devices, such as one or more RGB cameras and one or more IR sensors. The image capture devices can be within a grow environment, such as cameras within a greenhouse, or external to the grow environment, such as cameras outside a greenhouse. The non-hyperspectral images can be satellite images, in which case the image capture devices would be satellites in orbit.

[0035] Fig. 3 illustrates an example of a non-hyperspectral image 300 captured by a non- hyperspectral image capture device according to an exemplary embodiment. The image 300includes several plants, such as plant 300A. The image 300 also includes several non-plant features, such as pathway 300B, hose bib 300C, and house 300D.

[0036] Returning to Fig. 1, at step 102 the one or more non-hyperspectral images are segmented to identify a plurality of image portions corresponding to the one or more plants. As discussed below, this segmentation can be performed using a variety of techniques or combination of techniques.

[0037] Fig. 4 illustrates a flowchart for segmenting the one or more non-hyperspectral images to identify a plurality of image portions corresponding to the one or more plants according to an exemplary embodiment. The process of segmenting the one or more non- hyperspectral images can include one or more of the steps illustrated in Fig. 4.

[0038] At step 401 an instance segmentation model and / or a semantic segmentation model is selected based at least in part on a plant type, a grow environment, and / or other parameters or features of the plant or environment. This step can include selecting between an instance segmentation model or a semantic segmentation model. For example, certain plants and / or grow environments can be more suitable for instance segmentation and other plants and / or grow environments can be suitable for semantic segmentation. This step can additionally, or alternatively, include selecting an instance segmentation model from a plurality of instance segmentation models or selecting a semantic segmentation model from a plurality of semantic segmentation models. Different segmentation models can be used for different plants and / or grow environments. For example, lettuce grown on floating rafts can utilize a different segmentation model than lettuce grown in the ground.

[0039] Step 401 can optionally include applying a classifier, machine learning model, other image analysis algorithm to the non-hyperspectral images to automatically classify the plant and / or grow environment. An appropriate segmentation model type (instance or semantic) and / or segmentation model (i.e., plant or grow environment specific) can then be identified based on the classification.

[0040] The segmentation model and / or segmentation model type can also be selected based at least in part on user input. For example, a user or administrator can review the non- hyperspectral image and manually select a segmentation model type and / or segmentation modelbased on user preferences. Optionally, the system can default to predetermined segmentation model and utilize a secondary segmentation model of segmentation is unsuccessful or unsatisfactory with the first model. For example, the image can first be segmented using instance segmentation, but if instance segmentation fails or produces an unsatisfactory result, then the image can be segmented using semantic segmentation.

[0041] At step 402 an instance segmentation model is applied to the one or more non- hyperspectral images to identify one or more instances of plants in the one or more non- hyperspectral images. Optionally, if semantic segmentation is selected for the particular plant and / or grow environment, then step 402 can be omitted.

[0042] Fig. 5 illustrates an example of instance segmentation according to an exemplary embodiment. As shown in the figure, the non-hyperspectral image 500 is segmented using instance segmentation to identify all plant instances in the image. Image 501 shows the non- hyperspectral image 500 with each instance of a plant shown in dashed lines.

[0043] Returning to Fig. 4, at step 403, a semantic segmentation model is applied to the one or more non-hyperspectral images to identify one or more pixels corresponding to plants in the one or more non-hyperspectral images. As discussed previously, this step can be performed instead of step 402, or in addition to step 402 if the results of instance segmentation are unsatisfactory.

[0044] Fig. 6 illustrates an example of semantic segmentation according to an exemplary embodiment. As shown in the figure, the non-hyperspectral image 600 is segmented using semantic segmentation to determine a semantic label or tag associated with each pixel in the image. Semantic segmentation identifies pixels in the image which are plant pixels, and optionally, can also identify other semantic categories, such as floor, wall, etc. Optionally, the semantic segmentation can identify just plant and non-plant pixels. Box 601 shows a portion of the semantically segmented image, indicating plant pixels and floor pixels.

[0045] Returning to Fig. 1, at step 103 one or more non-hyperspectral plant images are generated based at least in part on the identified plurality of image portions. The non- hyperspectral plant images are non-hyperspectral images which include only the portions of the image that correspond to the one or more plant.

[0046] Figs. 7A-7B illustrate an example of the non-hyperspectral plant image generation process according to an exemplary embodiment. As shown in Fig. 7A, image segmentation 701 is performed on the non-hyperspectral image 701. This segmentation can be instance segmentation or semantic segmentation, as discussed previously. A plant mask 703 is then generated based at least in part on the segmentation. An example plant mask 703 is shown in Fig. 7A. As shown in the figure, the plant mask 703 flags all pixels corresponding to plants, which are shown in black. All non-plant pixels are shown as white.

[0047] Turning to Fig. 7B, the plant mask 703 is then applied to the non-hyperspectral image to generate the non-hyperspectral plant image 704. As shown in Fig. 7B, the non- hyperspectral plant image 704 includes only the portions of the image corresponding to plants.

[0048] Returning to Fig. 1, at step 104 a specialized machine learning model (MLM) is applied to the one or more non-hyperspectral plant images to generate a plurality of hyperspectral plant images corresponding to a spectral range, each hyperspectral plant image in the plurality of hyperspectral plant images corresponding to a spectral subrange that is a subset of the spectral range.

[0049] The process of segmenting the non-hyperspectral images and generating non- hyperspectral plant images increases the overall efficiency of the system by reducing the size of each image that is processed by the system and the specialized MLM. A non-hyperspectral image that includes only plant portion requires less resources, computational power, and time during conversion to hyperspectral images than a non-hyperspectral image that includes additional unrelated features, such as walls, floors, accessories, and other non-plant details. Additionally, several of the pre-processing and post-processing steps described in greater detail below (such as variable sorting for normalization or extended multiplicative scatter correction) are more effective when operating on segmented plant images and would have degraded performance if applied to whole images, including non-plant portions.

[0050] However, steps 102 and 103 can optionally be omitted. While omitting these steps reduces the overall efficiency and effectiveness of the present system, the entire non- hyperspectral images can be passed directly from step 101 to step 104. In this case, the specialized machine learning model (MLM) can be applied to the one or more non-hyperspectral images to generate a plurality of hyperspectral images corresponding to a spectral range, eachhyperspectral image in the plurality of hyperspectral images corresponding to a spectral subrange that is a subset of the spectral range. Together, the plurality of hyperspectral images can form a “hyperspectral cube,” which can then be used for the data analysis processes described below.

[0051] The specialized machine learning model can be trained based at least in part on pairs of hyperspectral and non-hyperspectral plant images and can be configured to perform hyperspectral conversion of a non-hyperspectral plant image.

[0052] Fig. 8 illustrates a system chart for the hyperspectral conversion process according to an exemplary embodiment. As shown in Fig. 8, the non-hyperspectral plant image 801 is provided as input to the specialized machine learning model (MLM) 801. The specialized MLM can include input / output interfaces 802A to receive the input image(s) and to output the resulting hyperspectral images.

[0053] The specialized MLM 802 can additionally include a convolutional neural network 802B. The convolutional neural network can be, for example, a convolutional neural network with a U-Net architecture or a convolutional neural network with attention. Of course, these networks are provided as examples only, and the MLM can utilize an alternative or additional architectures. The specialized MLM 802 can utilize encoder / decoder architectures, transformers, transformers with multi-head spectral attention layers, etc.

[0054] The specialized MLM 802 further include MLM training software used to train the specialized MLM to accurately convert plant non-hyperspectral images into hyperspectral images. The training software 802C can communicate with a training database 803. The training database 803 A can include training data pairs 803 A corresponding to pairs of hyperspectral and non-hyperspectral images, as well as plant data training data pairs 803B corresponding to pairs of hyperspectral and non-hyperspectral plant images.

[0055] For example, the specialized MLM can be trained initially on RGB / hyperspectral pairs of random objects. After this, transfer learning can be performed using pre-trained weights and performing a secondary training with a plant-specific dataset that includes RGB / hyperspectral pairs of plants. The plant specific datasets can include different plants in different environments, such as leafy greens (basil and lettuce) in natural lighting or crops (suchas tomatoes) in additional lighting conditions. The various lighting conditions can include natural light, natural light plus artificial lights (of various different types) or solely artificial lights.

[0056] The specialized MLM 802 further includes hyperspectral conversion software 802D. The hyperspectral conversion software 802D leverages the convolutional neural network 802B and converts the input non-hyperspectral plant image 801 into a plurality of hyperspectral plant images 804, which are then output via the input / output interfaces 802A.

[0057] Fig. 9 illustrates an example of hyperspectral conversion of a non-hyperspectral plant image according to an exemplary embodiment. As shown in Fig. 9, the non-hyperspectral plant image 901 includes only the plant portions of the original captured image. Application of the specialized machine learning model 902 to the non-hyperspectral plant image 901 results in hyperspectral plant images 903.

[0058] Due to the nature of the drawings, it is not possible to show the different spectral ranges corresponding to each of the hyperspectral images. However, the hyperspectral images shown in Fig. 9 have different shading pattern to reflect that each hyperspectral image corresponds to a different spectral range of wavelengths.

[0059] Fig. 10 illustrates the spectral range and subranges of the hyperspectral images according to an exemplary embodiment. As discussed previously, the plurality of hyperspectral plant images correspond to a spectral range, with each hyperspectral plant image corresponding to a spectral subrange. The spectral range can be, for example, the entire visible spectrum, or a larger range that includes portions of the spectrum outside of the visible range (such as those captured by IR sensors, which can be used to extend the spectral range). The spectral range of the plurality of hyperspectral plant images can be, for example, a range between wavelengths of 200 nanometers to 15,000 nanometers, inclusive. Each spectral subrange of each hyperspectral plant image can be, for example, between 400-700 nanometers wide, inclusive. As shown in Fig. 10, hyperspectral image 1001 A correspond to spectral subrange 1002A, hyperspectral image 1001B correspond to spectral subrange 1002B, hyperspectral image 1001C correspond to spectral subrange 1002C, and hyperspectral image 1001D correspond to spectral subrange 1002D.

[0060] Optionally, one or more pre-processing steps can be performed prior to hyperspectral conversion and one or more post-processing steps can be performed after hyperspectral conversion and prior to plant feature extraction. Fig. 11 illustrates a flow of preprocessing steps and post-processing steps according to an exemplary embodiment.

[0061] Pre-processing steps 1101 can be performed prior to hyperspectral conversion 1102 of the non-hyperspectral images. At step 1101A a white balance algorithm is applied to the one or more non-hyperspectral images prior to segmenting the one or more non-hyperspectral images. The white balance algorithm can be applied to images, such as RGB images, to adjust and normalize lighting. The result of this algorithm is that even artificial lighting is adjusted to appear as natural lighting. This provides consistency in image lighting across different images.

[0062] Post-processing steps 1103 can be performed after hyperspectral conversion 1102 and prior to plant feature extraction 1104. At step 1103 A a variable sorting for normalization algorithm is applied to the plurality of hyperspectral plant images prior to extracting the one or more plant features. This algorithm removes the effects in the image of different angles of lighting and corrects for changes in lighting.

[0063] At step 1103B an extended multiplicative scatter correction algorithm is applied to normalize values in one or more spectral channels of the plurality of hyperspectral plant images prior to extracting the one or more plant features. This algorithm is a normalization method that normalizes the values in each spectral channel.

[0064] Returning to Fig. 1, at step 105 one or more plant features of the one or more plants are extracted based at least in part on applying at least one image processing technique to at least one hyperspectral plant image in the plurality of hyperspectral plant images.

[0065] Fig. 12 illustrates a flowchart and system diagram of the plant feature extraction process according to an exemplary embodiment. At step 1201 one or more image processing techniques are applied to one or more hyperspectral plant images. Boxes 1201A, 1202B, and 1202C illustrate examples of image processing techniques that can be applied.

[0066] At step 1201A a reflectance estimation algorithm is applied to at least one hyperspectral plant image to determine a reflectance level in the corresponding spectralsubrange. For example, the reflectance level in the IR range can provide information about water stress levels in a plant.

[0067] At step 1201B a pixel signature is determined based at least in part on the at least one hyperspectral plant image. For example, a pixel signature can be compared with expected pixel signatures to identify signs of pest infestation in a plant.

[0068] At step 1201C a partial least square regression analysis is applied to the plurality of hyperspectral plant images. This analysis can identify deficiencies in nutrient amounts in the plant, such as deficiencies in nitrogen, phosphorous, or potassium.

[0069] At step 1202 plant features are extracted based at least in art on applying the one or more image processing techniques to the one or more hyperspectral plant images. The extracted plant features can include, for example, current features of the plant(s) 1202A, predicted features of the plant(s) 1202B, pest infestation 1202C, nutrient levels 1202D, water stress 1202E, plant weight 1202F, chlorophyll content 1202G, or photosynthetic efficiency 1202H.

[0070] At step 1203 plant growth, plant health, and / or other plant parameters can be analyzed and / or displayed. This step can include displaying results and corresponding charts on user interface, such as a dashboard, or in digital reports or web pages. This step can also be performed to determine corrective or plant environment modification instructions to address any issues or problems detected in the plants, as discussed further below.

[0071] Figs. 13A-13E illustrate different plant parameters and / or metrics that can be extracted from the hyperspectral plant images according to an exemplary embodiment.

[0072] Fig. 13A illustrates the detected photosynthesis levels in plants. The upper box illustrates an image of the plants, and the lower box illustrates the detected photosynthesis levels for different portion of the plants in the image.

[0073] Fig. 13B illustrates the photosynthetic efficiency of the plants shown in Fig. 13A over time. Photosynthetic efficiency is an index measuring light conversion efficiency to biomass. It is assessed by examining specific spectral characteristics of the leaves. Photosynthetic efficiency can be determined through analysis of the xanthophyll cycle.

[0074] Fig. 13C illustrates light levels of the plants shown in Fig. 13A over time. This is the photosynthetic photon flux density (PPFD) that measures the quantity of light suitable for photosynthesis.

[0075] Fig. 13D illustrates the daily light integral of the plants shown in Fig. 13 A over time. The daily light integral is the total amount of light a plant receives in a day, measured in moles / meters square / day.

[0076] Fig. 13E illustrates vapor pressure deficit of the plants shown in Fig. 13 A over time. Vapor pressure deficit (VPD) measures the difference between the moisture in the air and its saturation point, influencing plant transpiration and water stress.

[0077] Fig. 14 illustrates an example of a determined chlorophyll index levels over time in one or more plants according to an exemplary embodiment. The chlorophyll index is derived from the generated hyperspectral images and shows periodic behavior over the course day-night cycles (shown with dashed lines).

[0078] Figs. 15A-15B illustrate graphs showing correlations for harvest weights of plants according to an exemplary embodiment. The first column of graphs shows the correlation between the visible size of a plant and the actual weight of the plant at harvest and the second column of graphs shows the correlation between cumulative photosynthetic efficiency determined by the present process and the harvested weight of a plant. Each row corresponds to a different point in time, with the top row being just prior to harvest, the second row being 75% into the growth cycle, the third row being halfway through the growth cycle, and the bottom row being 25% into the growth cycle.

[0079] As discussed previously, the present techniques can be used to create plant growth models and controllers to predict future outcomes for plants and to determine and implement environmental modification actions that alter a growth rate, trajectory, or predicted future condition of a plant.

[0080] Fig. 16 illustrates a flowchart for modifying plant growth according to an exemplary embodiment.

[0081] At step 1601 a current state of the one or more plants and at least one predicted future state of the one or more plants is determined based at least in part on the plurality of hyperspectral plant images and a second specialized MLM trained on historical plant data.

[0082] At step 1602 at least one plant growth modification action is determined based at least in part on the current state of the one or more plants, a desired future state of the one or more plants, and the second MLM, wherein the desired future state is different than the at least one predicted future state. The system can also treat one or more plants like a path variable, using historically similar plant data to predict the future growth of the plant of interest.

[0083] At step 1603 one or more modification instructions corresponding to the at least one plant growth modification action are transmitted to a one or more plant environment systems, the one or more plant environment systems being configured to implement the one or more modification instructions to alter an environment of the one or more plants.

[0084] Fig. 17 illustrates examples of plant environment systems according to an exemplary embodiment. Plant environment systems 1700 can include lighting systems 1700A, heating, ventilation, and air conditioning systems 1700B, carbon dioxide regulators (1700C), irrigation systems 1700D, and / or one or more other systems 1700E.

[0085] Fig. 18 illustrates examples of plant growth modification actions 1800 according to an exemplary embodiment. Plant growth modification actions 1800 can include adjusting blue light treatment 1800 A, adjusting red light treatment 1800B, increasing light intensity 1800C, decreasing light intensity 1800D, modifying light spectrum 1800E, increasing a lighting time window 1800F, decreasing a lighting time window 1800G, adjusting a ventilation setting 1800H, adjusting a temperature setting 18001, adjusting an air supply setting 1800J, or adjusting an irrigation setting 1800K, and / or other adjustments 1800L. The modification actions and modification instructions can be sent to different systems and subsystems, such as controllers, lenses, light sources, hoses, pumps, etc.

[0086] Fig. 19 illustrates a system flow diagram of the plant analysis, prediction, modification system according to an exemplary embodiment.

[0087] At step 1900 raw data is captured, including images (such as RGB images), sensor data (such as IR sensors, humidity, temperature, etc ), and other data such as weather data. Rawdata can include RGB camera output, IR camera / sensor output, ambient temperature, ambient humidity, sunlight, and / or predicted weather. Raw data also include carbon dioxide or oxygen content in the environment, soil moisture level, soil nutrient content, water nutrient content, etc. Raw data can be received via one or more APIs from other computer systems, such as weather data from the national weather service.

[0088] At step 1901 hyperspectral images are generated using one or more of the techniques described herein.

[0089] At step 1902 plant and / or environmental metrics and / or features are determined based on the hyperspectral images and / or captured raw data. As discussed earlier, this can be plant / lead metrics (growth rate, size, etc.), a plant health spectral fingerprint (chlorophyll content, diseases, etc.), a computed / predicted daily light integral (DLI), temporal photosynthetic photon flux density prediction.

[0090] At step 1903 the plant and / or environment metrics and / or features are analyzed. This step can include inputs from step 1901 and / or step 1902, as well as historic data 1904 and / or goals / preferences 1905 provided by users or administrators. Historic data 1904 can include historic plant growth data. Goals / preferences can include a farmer’s desired output (increased yield, reduced expenses, changes in flavor / sugar content, etc.).

[0091] At step 1906 results and potential outcomes are determined based at least in part on the analysis in step 1903. Potential outcomes can include, for example, disease stress, light stress, insufficient light, deficient growth trajectory, small leaf size, etc.

[0092] At step 1907 recommendations and / or corrective actions or adjustments are determined. As discussed earlier, these can include blue light treatment, decreasing light intensity, modifying a spectrum, increasing intensity or number of hours of light, adjusting red light density or far-red light density, etc.

[0093] At step 1908 the output of different steps can be transmitted in a user interface or dashboard. This can include the output of step 1902, step 1906, and step 1907.

[0094] One or more of the above-described techniques can be implemented in or involve one or more special-purpose computer systems having computer-readable instructions loaded thereon that enable the computer system to implement the above-described techniques. Fig. 20illustrates the specialized computing environment 2000 of the plant analysis and prediction platform that is used to perform the above-described methods and implement the abovedescribed systems according to an exemplary embodiment.

[0095] With reference to Fig. 20, the computing environment 2000 includes at least one processing unit / controller 2002 and memory 2001. The processing unit 2002 executes computerexecutable instructions. In a multi-processing system, multiple processing units execute computer-executable instructions to increase processing power. The memory 901 can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory 2001 can store software implementing the above-described techniques and data structures, including image storage 2001 A, environmental data and analysis software 2001B, image segmentation software 2001C, hyperspectral conversion machine learning model 200 ID, plant feature and data analysis software 2001E, white balance adjustment software 2001F, variable sorting for normalization software 2001G, extended multiplicative scatter correction software 2001H, historical plant data 20011, plant growth and prediction machine learning model 2001 J, growth modification analysis software 2001K, environmental system controllers 2001L, or any other software described herein.

[0096] All of the software stored within memory 2001 can be stored as a computer- readable instructions, that when executed by one or more processors 2002, cause the processors to perform the functionality described with respect to Figs. 1-19.

[0097] Processor(s) 2002 execute computer-executable instructions. In a multi-processing system, multiple processors or multicore processors can be used to execute computer-executable instructions to increase processing power and / or to execute certain software in parallel.

[0098] Specialized computing environment 2000 additionally includes a communication interface 2003, such as a network interface, which is used to communicate with devices, applications, or processes on a computer network or computing system, collect data from devices on a network, such as legacy systems, destination systems, or other network systems, and implement encryption / decry ption actions on network communications within the computer network or on data stored in databases of the computer network. The communication interface conveys information such as computer-executable instructions, audio or video information, orother data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired or wireless techniques implemented with an electrical, optical, RF, infrared, acoustic, or other carrier.

[0099] Specialized computing environment 2000 further includes input and output interfaces 2004 that allow users (such as system administrators) to provide input to the system to set parameters, to edit data stored in memory 2001, or to perform other administrative functions.

[0100] An interconnection mechanism (shown as a solid line in Fig. 20), such as a bus, controller, or network interconnects the components of the specialized computing environment 2000.

[0101] Input and output interfaces 2004 can be coupled to input and output devices. For example, Universal Serial Bus (USB) ports can allow for the connection of a keyboard, mouse, pen, trackball, touch screen, or game controller, a voice input device, a scanning device, a digital camera, remote control, or another device that provides input to the specialized computing environment 2000.

[0102] Specialized computing environment 2000 can additionally utilize a removable or non-removable storage, such as magnetic disks, magnetic tapes or cassettes, CD-ROMs, CD- RWs, DVDs, USB drives, or any other medium which can be used to store information and which can be accessed within the specialized computing environment 2000.

[0103] Having described and illustrated the principles of our invention with reference to the described embodiment, it will be recognized that the described embodiment can be modified in arrangement and detail without departing from such principles. Elements of the described embodiment shown in software can be implemented in hardware and vice versa.

[0104] In view of the many possible embodiments to which the principles of our invention can be applied, we claim as our invention all such embodiments as can come within the scope and spirit of the following claims and equivalents thereto.

Claims

We Claim:

1. A method executed by one or more computing devices for extracting plant features from non-hyperspectral images: receiving, by at least one of the one or more computing devices, one or more non- hyperspectral images of one or more plants, the one or more non-hyperspectral images being captured by one or more non-hyperspectral image capture devices, wherein each non- hyperspectral image comprises one or more channels, each channel corresponding to a different wavelength; segmenting, by at least one of the one or more computing devices, the one or more non- hyperspectral images to identify a plurality of image portions corresponding to the one or more plants; generating, by at least one of the one or more computing devices, one or more non- hyperspectral plant images based at least in part on the identified plurality of image portions; applying, by at least one of the one or more computing devices, a specialized machine learning model (MLM) to the one or more non-hyperspectral plant images to generate a plurality of hyperspectral plant images corresponding to a spectral range, each hyperspectral plant image in the plurality of hyperspectral plant images corresponding to a spectral subrange that is a subset of the spectral range, wherein the specialized machine learning model is trained based at least in part on pairs of hyperspectral and non-hyperspectral plant images and configured to perform hyperspectral conversion of a non-hyperspectral plant image; and extracting, by at least one of the one or more computing devices, one or more plant features of the one or more plants based at least in part on applying at least one image processing technique to at least one hyperspectral plant image in the plurality of hyperspectral plant images.

2. The method of claim 1, wherein one or more non-hyperspectral images comprise one or more of: a Red-Green-Blue (RGB) image, a Cyan-Magenta- Yellow-Key (CMYK) image, or an infrared (IR) image.

3. The method of any one of claim 1, wherein the spectral range comprises a range between wavelengths of 200 nanometers to 15,000 nanometers, inclusive.

4. The method of claim 3, wherein each spectral subrange is between 400-700 nanometers wide, inclusive.

5. The method of claim 1, wherein segmenting the one or more non-hyperspectral images to identify a plurality of image portions corresponding to the one or more plants comprises one or more: applying an instance segmentation model to the one or more non-hyperspectral images to identify one or more instances of plants in the one or more non-hyperspectral images; or applying a semantic segmentation model to the one or more non-hyperspectral images to identify one or more pixels corresponding to plants in the one or more non-hyperspectral images.

6. The method of claim 5, wherein one or more of the instance segmentation model or the semantic segmentation model are selected based at least in part on one or more of a plant type of the one or more plants or a grow environment of the one or more plants.

7. The method of claim 1, wherein the specialized MLM comprises one of: a convolutional neural network with a U-Net architecture or a convolutional neural network with attention.

8. The method of claim 1, further comprising one or more of: applying, by at least one of the one or more computing devices, a white balance algorithm to the one or more non-hyperspectral images prior to segmenting the one or more non- hyperspectral images; applying, by at least one of the one or more computing devices, a variable sorting for normalization algorithm to the plurality of hyperspectral plant images prior to extracting the one or more plant features; or applying, by at least one of the one or more computing devices, an extended multiplicative scatter correction algorithm to normalize values in one or more spectral channels of the plurality of hyperspectral plant images prior to extracting the one or more plant features.

9. The method of claim 1, wherein applying at least one image processing technique to at least one hyperspectral plant image in the plurality of hyperspectral plant images comprises one or more of: applying a reflectance estimation algorithm to the at least one hyperspectral plant image to determine a reflectance level in the corresponding spectral subrange; determining a pixel signature based at least in part on the at least one hyperspectral plant image; applying a partial least square regression analysis to the plurality of hyperspectral plant images.

10. The method of claim 1, wherein the one or more plant features comprise one or more current features of the one or more plants or one or more predicted features of the one or more plants.

11. The method of claim 1, wherein the one or more plant features comprise one or more of water stress, pest infestation, nutrient level, plant weight, chlorophyll content, or photosynthetic efficiency.

12. The method of claim 1, further comprising: determining, by at least one of the one or more computing devices, a current state of the one or more plants and at least one predicted future state of the one or more plants based at least in part on the plurality of hyperspectral plant images and a second specialized MLM trained on historical plant data; determining, by at least one of the one or more computing devices, at least one plant growth modification action based at least in part on the current state of the one or more plants, a desired future state of the one or more plants, and the second MLM, wherein the desired future state is different than the at least one predicted future state; and transmitting, by at least one of the one or more computing devices, one or more modification instructions corresponding to the at least one plant growth modification action to a one or more plant environment systems, the one or more plant environment systems beingconfigured to implement the one or more modification instructions to alter an environment of the one or more plants.

13. The method of claim 12, wherein the one or more plant environment systems comprise one or more of a lighting system, a heating-ventilation-air-conditioning (HVAC) system, a carbon dioxide regulator, or an irrigation system.

14. The method of claim 12, wherein the at least one plant growth modification action comprises one or more of: adjusting blue light treatment, adjusting red light treatment, decreasing light intensity, increasing light intensity, modifying light spectrum, increasing a lighting time window, decreasing a lighting time window, adjusting a ventilation setting, adjusting a temperature setting, adjusting an air supply setting, or adjusting an irrigation setting.

15. At least one non-transitory computer-readable medium storing computer-readable instructions for extracting plant features from non-hyperspectral images that, when executed by one or more computing devices, cause at least one of the one or more computing devices to carry out a method according to any one of the preceding claims.

16. An apparatus for extracting plant features from non-hyperspectral images, the apparatus comprising: one or more processors; and one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to carry out a method according to any one of claims 1-14.

Images