System and method for monitoring plants

Abstract

Examples described herein provide a computer-implemented method for training a machine learning model to determine whether to water a plant of interest. The method includes receiving training data, the training data including a set of time-lapse images for each of a plurality of plants. The method further includes, for each of the plurality of plants: determining leaf displacement for the plant using the set of time-lapse images for the plant; predicting, based on the leaf displacement, whether to water the plant; and comparing a result of the prediction with a ground truth soil moisture value taken by a soil moisture sensor associated with the plant. The method further includes training a machine learning model based on each of the results of the comparison for each of the plurality of plants.

Description

SYSTEM AND METHOD FOR MONITORING PLANTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of United States Provisional Application Serial No. 63 / 609308 filed December 12, 2023, the contents of which is incorporated herein by reference.BACKGROUND

[0002] Gardening, including the care of houseplants, has become a popular hobby. Care of houseplants involves providing proper light exposure, nutrition (e.g., fertilization), and watering in order for houseplants to survive, grow, and thrive. Different types of houseplants have different needs in terms of light exposure, nutrition, and watering.

[0003] Automated systems for performing some types of gardening maintenance, such as watering, have been developed to reduce the effort needed by the owner of the houseplants. Some of these systems use a timer that opens a valve to dispense a predetermined amount of water to the plant. While these systems can be effective, they do not adapt to changing environments and the plant may at any point in time be overwatered or underwatered.

[0004] Accordingly, while existing systems for maintaining houseplants are suitable for their intended purposes the need for improvement remains, particularly in providing a system and a method having the features described herein.SUMMARY

[0005] In one exemplary embodiment, a computer-implemented method for training a machine learning model to determine whether to water a plant of interest is provided. The method includes receiving training data, the training data including a set of time-lapse images for each of a plurality of plants. The method further includes, for each of the plurality of plants: determining leaf displacement for the plant using the set of time-lapse images for the plant; predicting, based on the leaf displacement, whether to water the plant; and comparing a result of the prediction with a ground truth soil moisture value taken by a soil moisture sensor associated with the plant. The method further includes training a machine learning model based on each of the results of the comparison for each of the plurality of plants. The trainingidentifies optimal parameters of a machine learning prediction algorithm that uses the leaf displacement across time as an input and outputs a predicted value that can be used to determine whether or not the plant should be watered.

[0006] In one exemplary embodiment, a computer-implemented method for determining whether to water a plant of interest is provided. The method includes receiving a set of timelapse images of the plant of interest. The method further includes detecting features of the plant of interest using the set of time-lapse images. The method further includes determining leaf displacement for the plant of interest using the features. The method further includes determining, using a trained machine learning model, whether to water the plant of interest, wherein the trained machine learning model takes as input the leaf displacement and generates as output a prediction of whether to water the plant of interest. The method further includes generating a watering instruction responsive to determining to water the plant of interest.

[0007] In another exemplary embodiment a system for determining whether to water a plant of interest is provided. The system includes a display, a camera to capture a set of timelapse images of the plant of interest, a memory comprising computer readable instructions, and a processing device for executing the computer readable instructions. The computer readable instructions control the processing device to perform operations. The operations include detecting features of the plant of interest using the set of time-lapse images. The operations further include determining leaf displacement for the plant of interest using the features. The operations further include determining, using a trained machine learning model, whether to water the plant of interest, wherein the trained machine learning model takes as input the leaf displacement and generates as output a prediction of whether to water the plant of interest. The operations further include generating a watering instruction responsive to determining to water the plant of interest. The operations further include presenting the watering instruction to a user on the display.

[0008] Other embodiments described herein implement features of the above-described method(s) in computer systems and computer program products.

[0009] The above features and advantages, and other features and advantages, of the disclosure are readily apparent from the following detailed description when taken in connection with the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The specifics of the exclusive rights described herein are particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other features and advantages of one or more embodiments described herein are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:

[0011] FIG. 1A depicts a block diagram of components of a machine learning training and inference system according to one or more embodiments described herein;

[0012] FIG. IB depicts a block diagram of a system for monitoring plants using machine learning according to one or more embodiments described herein;

[0013] FIG. 1C depicts the system of FIG IB for monitoring plants using machine learning according to one or more embodiments described herein;

[0014] FIG. 2 depicts a flow diagram of a method for training a machine learning model for monitoring plants according to one or more embodiments described herein;

[0015] FIG. 3A, FIG.3 B, and FIG. 3C depict a flow diagram of a method for training a machine learning model for monitoring plants according to one or more embodiments described herein;

[0016] FIG. 4 depicts a graph of a distribution plot of a probability density of a number of days until a plant is predicted to need watering according to one or more embodiments described herein;

[0017] FIG. 5 depicts a flow diagram of a method for performing inference using a machine learning model for monitoring plants according to one or more embodiments described herein;

[0018] FIG. 6A, 6B, and 6C depict a flow diagram of a method for performing inference using a machine learning model for monitoring plants according to one or more embodiments described herein;

[0019] FIG. 7A, 7B, and 7C together depict a set of time-lapse images respectively of a plant of interest according to one or more embodiments described herein;

[0020] FIG. 8 depicts an example of an image of a set of time-lapse images according to one or more embodiments described herein;

[0021] FIG. 9 depicts a block diagram of a processing system for implementing one or more embodiments described herein;

[0022] FIG. 10A, FIG. 10B, and FIG. 10C depict a flow diagram of a method for training a machine learning model for monitoring plants which do not exhibit movement responses to plant water stress according to one or more embodiments described herein; and

[0023] FIG. 11 A and FIG. 1 IB depict a flow diagram of a method for performing inference using a machine learning model for monitoring plants which do not exhibit movement responses to plant water stress according to one or more embodiments described herein.

[0024] The diagrams depicted herein are illustrative. There can be many variations to the diagram or the operations described therein without departing from the scope of the embodiments described herein. For instance, the actions can be performed in a differing order or actions can be added, deleted or modified. Also, the term “coupled” and variations thereof describes having a communications path between two elements and does not imply a direct connection between the elements with no intervening elements / connections between them. All of these variations are considered a part of the specification.DETAILED DESCRIPTION

[0025] One or more embodiments described herein provide for monitoring plants, such as houseplants.

[0026] Watering houseplants at the right time can be challenging for the inexperienced or temporarily careless plant owner. Watering too early or too often can cause root rot.However, watering too late or not often enough causes stress, wilting, and eventually death of the plant in acute situations. Complicating matters is that moisture in the soil drops at a variable rate depending on a number of factors including temperature, humidity in the air, metabolic rate of the plants in the soil, etc., so a simple regular schedule to water each plant is often inadequate resulting in periods of drought or overwatering.

[0027] Often, to determine whether to water a plant, a plant owner waters on a set schedule or on an as-needed basis, such as when the soil feels dry to the touch. For example, if the soil feels dry, the plant owner may water, but if not, the plant owner waits to water. According to horticulturalists, plants can experience plant water stress even while the soil feels damp, and actually waiting until the soil is dry to the touch can be dangerous for the plant. In some situations, an experienced and particularly attentive plant owner may use visual cues, such as plant droop, to instruct watering habits rather than relying on touch alone. However, this visual intuition approach requires close attention to be paid to each individual plant. As plant movements are often subtle, visual intuition is inaccessible to novice or more casual plant owners.

[0028] Another approach to understanding soil moisture for plants, and when to water, is to use impedance-based soil moisture sensors. These sensors use the electrical impedance of the soil to gauge the amount of water in the soil. The primary drawback of this approach is that the sensors can corrode due to having metals, electrical current, and water in close proximity, thus necessitating frequent replacement or infrequent usage of the impedance-based sensors. Another drawback is that the sensors are sensitive to the composition of the soil. Particularly, common soil additives such as perlite and vermiculite alter the response of the sensor, making typical consumer grade impedance sensors inaccurate for soils containing such additives. A further drawback is that, when used as an “always-on” solution to detect soil moisture, these sensors can be visually distracting (e.g., sticking out of the plant pot with wires trailing out) and / or inconvenient to use (e.g., requiring frequent recharging or changing of a battery). Impedance soil moisture sensors can also be used on an ad-hoc basis, but this approach is more time-intensive because the plant owner must manually place and use the sensor, compared to an “always-on” sensor that simply alerts the plant owner when a plant may need water.

[0029] Another approach to understanding soil moisture for plants is to use soil tensiometers. Soil tensiometers employ a water filled tube with a porous tip to gauge the amount of moisture in the soil. As the soil becomes drier, capillary action generates vacuum pressure on the contents of the tensiometer. The vacuum pressure can then be read by a vacuum sensor. Due to the fact that there must be enough water present inside of the tensiometer to generate the capillary action, tensiometers tend to be fairly large and unsuitable for indoor use. Another drawback is that the tensiometers must be regularlyrefilled with water. Currently available tensiometers are also quite expensive with few consumer options. As such, tensiometers are much more prevalent in agricultural applications than in consumer usage, such as the care of houseplants.

[0030] Yet another approach to understanding when to water plants is to use remote sensing solutions. Remote sensing is the process of monitoring and measuring characteristics at a distance without physical contact with the object being measured (e.g., without a moisture sensor). Plants experiencing water stress display observable characteristics different to plants not experiencing water stress. A number of imaging and reflectance based remote sensing techniques have been developed for detecting plant water stress. Remote sensing to detect plant water stress is possible due to observable reactions in the crown / canopy of many plants. A plant crown is the above ground portion of a plant, and a canopy is a collection of plant crowns. Water stress has been linked to several observable variations in canopy properties, such as: changes in leaf angle due to water stress causing leaves to wilt in broad leaved species and leaves to roll in grass species; increases in leaf temperature due to stomatai closure to limit transpiration; and leaf movement

[0031] Another approach to understanding when to water plants is to measure reflectance to estimate canopy properties. For example, one approach of remote sensing to detect plant water stress is to take overhead measurements of light reflectance across a field of plants. Shaded and sunlit soil and foliage have different reflectance properties and thus can be used to estimate leaf angle across a crop. This type of measurement is usually collected at a large scale (e.g. for agricultural crops via airplanes and satellites) with spectrometers or digital cameras. It is less appropriate for individual plants, such as house plants, when higher resolution methods are available.

[0032] Yet another approach to understanding when to water plants is to directly measure leaf angles. Instead of using indirect methods to estimate leaf angle, a number remote sensing techniques have been developed to measure leaf angle directly to detect plant water stress. These include using single lens photography, stereo lens photography, photogrammetry, and light detection and ranging (LiDAR). Each approach has proven effective in measuring the distribution of leaf angles in plant crowns / canopies. However, each approach also has its own drawbacks. In the case of single lens photography, suitable leaves for calculation of leaf angle must be chosen manually. For stereo lens photography, no manual selection of suitable leaves is required; however, inaccurate camera calibration (e.g., focal length, aperture, and lightingconditions) negatively affect the quality of 3D reconstruction from the stereo imaging. LiDAR sidesteps both these issues by using lasers to measure distances and reconstruct leaf angles. However, a single stationary LiDAR system (e.g., one that does not utilize LiDAR measurements from multiple perspectives) has strict requirements on angle and placement to achieve maximum accuracy, and thus can be difficult to implement.

[0033] Thermal imaging can also be used to measure leaf temperature to understand when to water plants. Studies using thermal imaging to detect plant water stress have shown promise. However, this approach requires the use of expensive thermal imaging cameras, and thus may not be suitable for houseplant owners.

[0034] One or more embodiments described herein uses machine learning methods to monitor plants. For example, one or more embodiments measures leaf movement in a plant of interest using time-lapse images of the plant of interest and analyzes the images using a trained machine learning model to determine whether to water the plant of interest. Thus, the movement trends (e.g., from hour to hour, or day to day) of individual plants can be determined and used to predict when to water each plant. As a result, plant owners are provided with information regarding when to water their houseplants. One or more embodiments described herein detects when a plant has been watered and notifies a plant owner in a timely manner. To do this, a machine learning model is trained and used. The machine learning model analyzes leaf displacement information to predict whether to water a plant. As used herein, leaf displacement means movement data on any visible part of a plant, including, but not limited to: leaves, stems, stalks shoots, buds, flower, and fruit.Displacement relates to the change in physical location of a point on the plant itself (a keypoint) as seen across images in a timelapse. For example, a keypoint may be identified as a point on the tip of a particular leaf of a particular plant. The distance and direction that the point on the plant moves in subsequent images can be measured and expressed as a vector of movement, or displacement.

[0035] One or more embodiments described herein uses remote sensing that does not require direct contact to the soil of each plant being monitored. Thus, the one or more embodiments can aid a plant owner to track the watering status and needs of multiple plants at once using only a single, low-cost device (e.g., a camera), without the need of dedicated impedance-based soil moisture sensors and / or soil tensiometers. Further, one or more embodiments described herein has very low ongoing maintenance requirements after initialsetup compared to direct sensing approaches. In contrast, impedance-based soil moisture sensors) require frequent replacement due to corrosion, and tensiometers must be frequently refilled and their vacuum seal must be checked to be intact. Moreover, the camera used in the one or more embodiments described herein can be placed a distance away from the plants being monitored. Thus, the camera can be placed discretely and near a wall outlet for power. Getting electricity to the impedance-based soil moisture sensors and, if powered (e.g. to send live data to a server), the soil tensiometer, can be inconvenient in the context of houseplants as these approaches require either running a wire to each sensor, which can be visually unattractive, or relying on battery power, which requires regular replacement of batteries or recharging.

[0036] Further, one or more embodiments described herein are less expensive in terms of requirements and easier to use than other remote sensing approaches. Existing remote sensing solutions are either much more expensive in terms of equipment, as in the cases of thermal imaging and LiDAR, not suitable for consumer usage, as in the cases of measuring canopy reflectance and direct leaf angle measurement with single lens photography, or have stringent requirements for placement, as in the case of direct leaf angle measurement with LiDAR and stereo photography. The embodiments described herein are lower cost because they can use a single camera as the sensor, are easy to use for individual plants in the home, and are relatively flexible in placement.

[0037] One or more embodiments described herein can utilize machine learning techniques to perform tasks, such as monitoring plants. For example, a machine learning model can be trained to predict when and / or how much to water a plant using time-lapse images of the plant. More specifically, one or more embodiments described herein can incorporate and utilize rule-based decision making and artificial intelligence (Al) reasoning to accomplish the various operations described herein, namely monitoring plants. The phrase “machine learning” broadly describes a function of electronic systems that learn from data. A machine learning system, engine, or module can include a trainable machine learning algorithm that can be trained, such as in an external cloud environment, to learn functional relationships between inputs and outputs, and the resulting model (sometimes referred to as a “trained neural network,” “trained model,” and / or “trained machine learning model”) can be used for monitoring plants, for example. In one or more embodiments, machine learning functionality can be implemented using an artificial neural network (ANN) having the capability to betrained to perform a function. In one or more embodiments, machine learning functionality can be implemented using a random forest, the XGBoost algorithm, or another suitable approach to implementing machine learning techniques, such as to predict whether to water a plant. In machine learning and cognitive science, ANNs are a family of statistical learning models inspired by the biological neural networks of animals, and in particular the brain. ANNs can be used to estimate or approximate systems and functions that depend on a large number of inputs. Convolutional neural networks (CNN) are a class of deep, feed-forward ANNs that are particularly useful at tasks such as, but not limited to analyzing visual imagery and natural language processing (NLP). Recurrent neural networks (RNN) are another class of deep, feed-forward ANNs and are particularly useful at tasks such as, but not limited to, unsegmented connected handwriting recognition and speech recognition. Other types of neural networks are also known and can be used in accordance with one or more embodiments described herein.

[0038] ANNs can be embodied as so-called “neuromorphic” systems of interconnected processor elements that act as simulated “neurons” and exchange “messages” between each other in the form of electronic signals. Similar to the so-called “plasticity” of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in ANNs that carry electronic messages between simulated neurons are provided with numeric weights that correspond to the strength or weakness of a given connection. The weights can be adjusted and tuned based on experience, making ANNs adaptive to inputs and capable of learning. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by the pixels of an input image. After being weighted and transformed by a function determined by the network’s designer, the activation of these input neurons are then passed to other downstream neurons, which are often referred to as “hidden” neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character was input. It should be appreciated that these same techniques can be applied in the case of monitoring plants as described herein.

[0039] Systems for training and using a machine learning model are now described in more detail with reference to FIG. 1A. Particularly, FIG. 1A depicts a block diagram of components of a machine learning training and inference system 100 according to one or more embodiments described herein. The machine learning training and inference system 100 performs training 102 and inference 104. During training 102, a training engine 116 trains amodel (e.g., the trained ML model 118) to perform a task, such as to monitor plants. Inference 104 is the process of implementing the trained ML model 118 to perform the task, such as to monitor plants, in the context of a larger system (e.g., a system 126). All or a portion of the machine learning training and inference system 100 shown in FIG. 1A can be implemented, for example by all or a subset of the system 126 of FIGS. 1A-1C.

[0040] The training 102 begins with training data 112, which may be structured or unstructured data. According to one or more embodiments described herein, the training data 112 includes a set of time-lapse images for each of a plurality of plants. According to one or more embodiments described herein, the set of time-lapse images can each include one or more ground truth soil moisture values. For example, in an embodiment the camera 146 can capture images of a plant on a periodic basis (e.g., every 5 minutes), and the soil moisture sensor 160 can collect ground truth soil moisture values at the same time as when the images are captured (i.e. performed substantially simultaneously). According to one or more embodiments described herein, information about the plants, which can be used to determine leaf displacement as described herein, can be captured in other ways. For example, multiple cameras (e.g., multiple instances of the camera 146) can be used to capture time-lapse images of one or more plants. Using multiple cameras provides a larger field of view as compared to a single camera and can provide different points of view depending on the location and orientation of the multiple cameras. As another example, LIDAR data can be captured and used to verify or supplement the time-lapse images. As yet another example, photogrammetry techniques can be applied to one or more of the time-lapse images to extract 3D data about plant leaves. For example, photogrammetry can determine 3D coordinates of points on leaves of a plant, and the 3D coordinates can be used to determine leaf displacement.

[0041] The training engine 116 receives the training data 112 and a model form 114. The model form 114 represents a base model that is untrained. For example, the model form 114 can be a region-based CNN (R-CNN) algorithm or another suitable architecture. The model form 114 can have preset weights and biases, which can be adjusted during training. It should be appreciated that the model form 114 can be selected from many different model forms with the final model form selected via empirical testing of predictive power (i.e. how often the model correctly predicts when to water target plants). The training 102 can be supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and / or the like, including combinations and / or multiples thereof. For example, supervised learning canbe used to train a machine learning model to make predictions about object of interest in a set of time-lapse images (i.e., whether to water a plant). To do this, the training data 112 includes a set of time-lapse images for a plurality of plants and associated ground truth soil moisture values (e.g., measured by a soil moisture sensor). In this example, the training engine 116 takes as input a set of time-lapse training images of a plant from the training data 112, makes a prediction for whether to water a plant, and compares the prediction to the ground truth soil moisture value measured at the time of the last image in the time-lapse. The training engine 116 then adjusts weights and / or biases of the model based on results of the comparison, such as by using backpropagation. The training 102 may be performed multiple times (referred to as “epochs”) until a suitable model is trained (e.g., the trained ML model 118).

[0042] Once trained, the trained ML model 118 can be used to perform inference 104 to monitor plants. The inference engine 120 applies the trained ML model 118 to new data 122 (e.g., real-world, non-training data). For example, the new data 122 can be a set of time-lapse images of a plant of interest (where such images were not part of the training data 112). In this way, the new data 122 represents data to which the trained ML model 118 has not been exposed. The inference engine 120 makes a prediction 124 (i.e., whether to water a plant of interest in an image of the new data 122). The prediction can then be used by a system 126 to take an action, perform an operation, perform an analysis, and / or the like, including combinations and / or multiples thereof. According to one or more embodiments described herein, as shown in FIG. IB, the system 126 can include the inference engine 120 such that the system 126 makes the prediction 124 using the new data 122.

[0043] In accordance with one or more embodiments, the predictions 124 generated by the inference engine 120 are periodically monitored and verified to ensure that the inference engine 120 is operating as expected. Based on the verification, additional training 102 may occur using the trained ML model 118 as the starting point. The additional training 102 may include all or a subset of the original training data 112 and / or new training data 112. In accordance with one or more embodiments, the training 102 includes updating the trained ML model 118 to account for changes in expected input data.

[0044] FIG. IB depicts a block diagram of the system 126 for monitoring plants using machine learning according to one or more embodiments described herein. The system 126 can include a processing device 142, a memory 144, a camera 146, the trained model 118, and the inference engine 120.

[0045] The processing device 142 is any suitable device for executing instructions and / or performing processing functions. The processing device 142 can be a single core or multicore processor. The processing device 142 can be a microprocessor, central processing unit (CPU), special-purpose processing hardware, and / or the like including combinations and / or multiples thereof. The processing device 142 is an example of the processor 921 of FIG. 9.

[0046] The memory 144 is any suitable device for storing data and / or machine executable instructions. The memory 144 can be a volatile and / or a non-volatile memory. According to one or more embodiments described herein, the memory 144 can represent multiple memories and / or multiple types of memories. According to one or more embodiments described herein, the memory 144 can store the trained ML model 118, which is trained to monitor plants. The memory 144 is an example of the read only memory 922 and / or the random-access memory 924 of FIG. 9. The memory 144 can also store the inference engine 120, which can make the predictions 124 described herein.

[0047] The camera 146 is any suitable device for capturing images and / or video. For example, the camera 146 can be a visible or non- visible (e.g., infrared, ultraviolet) light camera and can capture still images and / or video. According to one or more embodiments described herein, the camera 206 can capture a set of time-lapse images of the plant of interest. For example, the camera 206 can capture multiple images of a plant of interest over a period of time (e.g., every 10 minutes each day). In an embodiment, the camera 208 includes a photosensitive array. In another embodiment, the camera 208 may be configured to simultaneously acquire an image and measure a distance to the plant of interest (e.g. a camera with an RGB-D sensor).

[0048] The various components, modules, engines, etc. described regarding FIG. IB can be implemented as instructions stored on a computer-readable storage medium, as hardware modules, as special-purpose hardware (e.g., application specific hardware, application specific integrated circuits (ASICs), application specific special processors (ASSPs), field programmable gate arrays (FPGAs), as embedded controllers, hardwired circuitry, etc.), or as some combination or combinations of these. According to aspects of the present disclosure, the engine(s) described herein can be a combination of hardware and programming. The programming can be processor executable instructions stored on a tangible memory, and the hardware can include the processing device 142 for executing those instructions. Thus, a system memory (e.g., the memory 144) can store program instructions that when executed bythe processing device 142 implement the engines described herein. Other engines can also be utilized to include other features and functionality described in other examples herein.

[0049] According to one or more embodiments described herein, the system 126 may be implemented in a distributed computing environment with the processing device 142, memory 144, trained model 118, and inference engine spanning across multiple processing systems in different physical locations (e.g. in cloud computing or distributed computing infrastructure). In an embodiment, the camera 146 will be co-located with the plant(s) being monitored.

[0050] FIG. 1C depicts a block diagram of the system 126 for monitoring plants using machine learning according to one or more embodiments described herein. In this example, the system 126 captures the images and stores the images. For example, the system 126 includes the camera 146 and a controller 150, which functions as a combination of a processing device (e.g., the processing device 142) and a memory (e.g., the memory 144). The system 126 may also include a communications circuit / module that allows for network accessed (e.g. via cloud providers) non-volatile storage 170, additional processing capacity, and / or capabilities on distributed systems 172.

[0051] The camera 146 captures time-lapse images of a plant of interest 162 and stores the images to a memory of the controller 150 and / or to a remote storage device (e.g., storage device 170). For example, the time-lapse images captured using the camera 146 and are then transferred to a storage device, such as a local storage (e.g., the memory 144) or a cloud storage provider. The controller 150 also receives soil moisture values from a soil moisture sensor 160. The soil moisture sensor 160 measures moisture levels in the soil of the plant of interest 162. According to one or more embodiments described herein, the soil moisture sensor 160 is a direct sensing soil moisture sensor (e.g. an impedance-based sensor or a tensiometer) for each plant that is being monitored (e.g., the plant(s) of interest 162). The soil moisture sensor 160 can be read by a device with suitable inputs (such as a microcontroller 164) and sent to the system 126 connected to the internet local to the location of the plants, for example. More particularly, the soil moisture sensor 160 can be connected by a wire and / or wirelessly (e.g., via a wired analog wire) to a microcontroller 164, which can then transmit the soil moisture values to the system 126. According to one or more embodiments described herein, a temperature and humidity sensor 166 and / or a light sensor 168 can also be connected to the microcontroller 164, which can be used to collect local environmental data(e.g., temperature data, light data, humidity data, and / or the like including combinations and / or multiples thereof). The environmental data can receive temperature and / or humidity data from the temperature and humidity sensor 166 and light data from the light sensor 168. In some embodiments, the temperature and humidity sensor 166 and / or the light sensor 168 can be omitted. According to one or more embodiments described herein, the local environmental data can also be received from another source, such as a public weather database.

[0052] The controller 150 receives the soil moisture values, the temperature data, the humidity data, and / or the light data from the microcontroller 164 via a wired and / or wireless connection (e.g., a USB connection). According to one or more embodiments described herein, the system 126 can store data (e.g., the images, soil moisture values, the temperature data, the humidity data, and / or the light data) locally (e.g., in the memory 144) and / or remotely, in a storage device 170 (e.g., cloud storage).

[0053] According to one or more embodiments described herein, the system 126 can perform inference 104 to make predictions 124 regarding whether to water the plant of interest 162. According to one or more embodiments described herein, the system 126 can utilize one or more remote processing systems (e.g., server(s) 172) to perform the inference 104 and / or to train the machine learning model for monitoring plants. For example, the system 126 can capture sets of time-lapse images of a plurality of plants to be used as training data 112, and the server(s) 172 can perform the training 102 to train the machine learning model for monitoring plants as described herein.

[0054] According to an embodiment, the system 126 can perform object detection, feature detection, object tracking, data pre-processing, and / or the like including combinations and / or multiples thereof, as described herein (see, e.g., FIGS. 3A-C and FIGS. 6A-6C). Such processes can be executed locally on the system 126 (e.g., using the processing device 142, the memory 144, and / or the controller 150) and / or can be executed remotely (e.g., using one or more distributed server(s) 172).

[0055] According to an embodiment, the system 126 can execute a prediction algorithm and / or an optimization algorithm as described herein (see, e.g., FIGS. 3A-3C). These algorithms can be executed locally on the controller 150 and / or remotely on one or more of the server(s) 172.

[0056] FIG. 2 depicts a flow diagram of a method 200 for training a machine learning model to monitor plants according to one or more embodiments described herein. The method 200 can be implemented by any suitable system or device, such as the system 126 of FIGS. 1A-1C, the processing system 900 of FIG. 9, and / or the like including combinations and / or multiples thereof. In an embodiment, the method 200 performs the training 102 of the machine learning training and inference system 100 of FIG. 1A.

[0057] At block 202, the training engine 116 receives training data 112. The training data 112 can include a set of time-lapse images for each of a plurality of plants. According to one or more embodiments described herein, the set of time-lapse images can each include one or more ground truth soil moisture values. For example, the camera 146 can capture images of a plant on a periodic basis (e.g., every 5 minutes), and the soil moisture sensor 160 can collect ground truth soil moisture values associated with each of the images.

[0058] For each of the plurality of plants, blocks 204, 206, and 208 of the method 200 are performed. At block 204, training engine 116 determines leaf displacement for the plant using the set of time-lapse images for the plant. At block 206, the training engine 116 predicts, based on the leaf displacement, whether to water the plant. At block 208, the training engine 116 compares a result of the prediction with a ground truth value derived from a measurement taken by a soil moisture sensor associated with the plant.

[0059] At block 210, it is determined whether more plants of the plurality of plants exist within the training data 112, and if so, the method 200 repeats blocks 204, 206, and 208. Once no more plants exist within the training data 112, the method 200 proceeds to block 212.

[0060] At block 212, the training engine 116 trains a machine learning model (e.g., the trained ML model 118) based on each of the results of the comparison for each of the plurality of plants. The training 102 identifies optimal parameters of a machine learning prediction algorithm that uses the leaf displacement across time as an input and outputs a predicted value that can be used to determine whether or not the plant should be watered.

[0061] Additional processes also may be included, and it should be understood that the process depicted in FIG. 2 represents an illustration, and that other processes may be added or existing processes may be removed, modified, or rearranged without departing from the scope of the present disclosure. It should also be understood that the processes depicted inFIG. 2 may be implemented as programmatic instructions stored on a non-transitory computer-readable storage medium that, when executed by a processor (e.g., the processing device 142) of a computing system (e.g., the system 126), cause the processor to perform the processes described herein.

[0062] The training 102 is now described in more detail with reference to FIG. 3A-3C. In an embodiment, FIG. 3A, FIG. 3B and FIG. 3C depict a flow diagram of a method 300 for training a machine learning model for monitoring plants according to one or more embodiments described herein. The method 300 can be implemented by any suitable system or device, such as the system 126 of FIGS. 1A-1C, the processing system 900 of FIG. 9, and / or the like including combinations and / or multiples thereof. In particular, the method 300 provides for optimizing parameters of a machine learning prediction algorithm that takes leaf movement patterns as inputs and outputs a value that can be used to determine whether or not plants should be watered.

[0063] A set of time-lapse images are obtained (block 302, FIG. 3B), such as from the camera 146. The set of time-lapse images can be captured periodically, such as at intervals (e.g., every 5 to 10 minutes). A ground truth soil moisture value (block 332) is measured at substantially the same time each of the set of time-lapse images are captured, such as using the soil moisture sensor 160.

[0064] The set of time-lapse images (block 302) are input into an object detection algorithm (block 304). For each image, the object detection algorithm (block 304) detects each plant in the image and associates a set of pixels with each detected plant (block 306). In an embodiment, the object detection algorithm is a trained object detection algorithm. This set of pixels may be expressed, for example, as a bounding box or mask. Examples of object detection algorithms include the R-CNN algorithm, the “you only look once” (YOLO) algorithm, and / or the like including combinations and / or multiples thereof. According to one or more embodiments described herein, the object detection algorithm can be trained to detect plants using images of plants and associated labels, where the associated labels indicate whether a detected object is a plant and / or a class or type of the plant.

[0065] Each plant representation (i.e. set of pixels determined to be representing a plant) in each image is input into a re-identification algorithm (block 307) which assigns an identity value to each representation (block 342). Identity values of representations of the same plantacross images in the time-lapse should be consistent to allow for tracking of movement of an individual plant over time. This re-identification algorithm may take various inputs in order to determine the likelihood / probability that a set of pixels represents the same plant across images, including, but not limited to: location of the representation in the image and embeddings of the representation from a hidden layer of a CNN algorithm.

[0066] Points of interest, or keypoints, for individual plants across images in the time-lapse are then identified using a feature detection algorithm (block 308). Keypoints can be any point identified on a plant that can be tracked across subsequent frames in a time-lapse. Examples of a feature detection algorithm include, but are not limited to, those described by J. Shi and Tomasi in “Good Features to Track” (1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Jun. 1994, pp. 593-600. doi:10.1109 / CVPR.1994.323794), features from accelerated segment test (FAST), and / or the like including combinations and / or multiples thereof. The feature detection algorithm (block 308) outputs coordinates of keypoints for each plant (block 310) in each of the cropped images. Examples of keypoints include, but are not limited to the tips of leaves, edges and comers of leaves, edges and comers of distinguishing marks on the plant (e.g. stripes, spots, variegations), and a location where the stem meets joins the leaf.

[0067] A feature tracking algorithm (block 312) is then used to track the location of each keypoint associated with an individual plant across a plurality of images in the set of timelapse images. Examples of an object tracking algorithm include, but are not limited to, those described by B. Lucas and T. Kanade in “An Iterative Image Registration Technique with an Application to Stereo Vision (UCAI)” (UCAI'81: Proceedings of the 7th international joint conference on Artificial intelligence - Volume 2 August 1981, Pages 674-679), simple online and real-time tracking (SORT), and / or the like including combinations and / or multiples thereof.

[0068] The output of the feature tracking algorithm (block 312) are pairs of coordinates of the same point of interest identified in a pair of images in the time-lapse. For example, the feature tracking algorithm may output two coordinates pertaining to the same leaf tip in a pair of images taken 5 minutes apart. By subtracting the later coordinate from the earlier, we arrive at a movement vector (in pixels) for each point of interest (block 314). The set of movement vectors relating to all, or substantially all, keypoints tracked on one plant across a pair of images in a time-lapse is referred to as the leaf displacement for that plant.

[0069] Leaf displacement of each plant (block 314, FIG. 3B) is then used to find optimal parameters of a ‘When to water’ machine learning prediction algorithm (block 320, FIG. 3C). Particularly, a training loop 301 is performed to find a desired or optimal prediction algorithm parameters (block 322) of a machine learning prediction algorithm (block 320) that uses leaf displacement data across time as an input and outputs a predicted value (block 324) that can be used to determine whether or not a plant should be watered. The type of prediction algorithm (block 320) can be any suitable classification or regression algorithm, including decision trees, neural networks, support vector machines, linear methods, and their variants, for example.

[0070] The format of the predicted value (block 324) can be any suitable format and can be further interpreted as a recommendation for a user as to whether or not to water the plant. For example, the prediction algorithm (block 320) can predict a probability on a scale between 0 and 1 that the plant should be watered, where 1 indicates the plant should be watered and 0 indicates the plant should not be watered. Another example of an output of the prediction algorithm might be some number on a continuous scale indicating the level of soil moisture in the plant pot, then a set of ranges can be used to segment the scale so if the output falls within or without a certain range, the prediction might be ‘water’, or ‘don’t water’. In an embodiment, the threshold probability for watering the plant may be user defined. Yet another example, the prediction algorithm (block 320) can predict a number of days until watering is needed. As yet another example, the prediction algorithm (block 320) can predict inputs to a probability density function (PDF) with “days until watering needed” as an independent variable. An illustration of a probability density that the prediction algorithm (block 320) can predict is shown in FIG. 4, which shows a graph 400 of number of days “X” until the plant is predicted to need watering is plotted in terms of probability density.

[0071] In order to optimize or train the ‘When to water’ prediction algorithm (block 301) within a desired accuracy, the outputs of the prediction algorithm are compared to a ‘ground truth’ value, i.e. a value representing whether the plant needed to be watered in that moment in reality. This ‘ground truth’ value is initially collected by a direct sensing soil moisture sensor 160 (FIG. 1C). Further processing of the data from the soil moisture sensor is performed (block 330, FIG. 3A) to align the format of the ground truth data to the format of the outputs of the prediction algorithm (block 320). For example, if the output of theprediction algorithm is a probability between 0 and 1 (inclusive), the ground truth data will be transformed into a value within the range of 0 and 1 (inclusive).

[0072] With continued reference to FIG. 3A-3C, the training loop 301 (which is an example of the training 102 of FIG. 1A) uses supervised training to optimize prediction algorithm parameters (block 322) that are used by the prediction algorithm (block 320) using an optimization algorithm (block 326). The optimization algorithm (block 326) calculates a loss based on the difference between the ground truth value (block 328) and the prediction algorithm output (block 324) and changes the prediction algorithm parameters (block 322) in order to reduce or minimize the loss. An example of an optimization algorithm is described by D. P. Kingma and J. Ba in “Adam: A Method for Stochastic Optimization” (arXiv, Jan. 29, 2017. doi: 10.48550 / arXiv.1412.6980). The training loop 301 is run in a loop until some stopping mechanism is triggered. A non-limiting example of a stopping mechanism is a process that records / stores loss values and stops the training when the loss value plateaus (i.e. the loss value has not appreciably dropped or decreased within a predetermined threshold in the last few iterations).

[0073] Additional environment data (block 316) may also be an optional input into the ‘When to water’ prediction algorithm (block 320) to increase prediction accuracy. One example of additional environment data is local climate data which may be collected by a temperature and humidity sensor 166. Another example of additional environment data is ambient light information collected by a light sensor 168.

[0074] Prior to sending inputs to the ‘When to water’ prediction algorithm (block 320), the input data may be further preprocessed (block 318) to increase or improve algorithm performance, or fit the format desired by the algorithm for example. Examples of further data processing include, but are not limited to: centering data by subtracting by the mean, binning continuous data into categories, and aggregation (e.g. averaging the displacement of each keypoint for each plant in a certain time period).

[0075] Additional processes also may be included, and it should be understood that the process depicted in FIGS. 3A-3C represents an illustration, and that other processes may be added or existing processes may be removed, modified, or rearranged without departing from the scope of the present disclosure. It should also be understood that the processes depicted in FIGS. 3A-3C may be implemented as programmatic instructions stored on a non-transitorycomputer-readable storage medium that, when executed by a processor (e.g., the processing device 142) of a computing system (e.g., the system 126), cause the processor to perform the processes described herein.

[0076] In an embodiment, once the training 102 is performed (e.g., once the methods 200 and / or 300 are performed), inference 104 can begin. An example of inference 104 is now described with reference to FIGS. 5 and 6A-6C. FIG. 5 depicts a flow diagram of a method 500 for performing inference using a machine learning model for monitoring plants according to one or more embodiments described herein. The method 500 can be implemented by any suitable system or device, such as the system 126 of FIGS. 1A-1C, the processing system 900 of FIG. 9, and / or the like including combinations and / or multiples thereof. In particular, the method 500 performs the inference 104 of the machine learning training and inference system 100 of FIG. 1A.

[0077] At block 502, the system 126 receives a set of time-lapse images of an individual plant or set of plant of interest. The set of time-lapse images can be received, for example, from the camera 146 or another suitable device or data store. It should be appreciated that for inference 104, the time-lapse images of the plant(s) of interest may or may not be similar to the training data 112 and may represent new plants unseen to the trained ML model 118. It should be further appreciated that the time-lapse images of the plant of interest do not have ground truth soil moisture values associated therewith.

[0078] At block 506, the system 126 determines leaf displacement for each plant. The process for determining leaf displacement for each plant can be performed using the same or similar technique for determining leaf displacement during training 102 (see, e.g., block 204 of FIG 2.). By using the leaf displacement data as inputs to a prediction algorithm, the advantages of a remote sensing solution described herein are realized and improved. As examples: the cost advantages of tracking leaf displacement using a single camera compared to other remote sensing solutions such as LiDAR and thermal imaging; the convenience advantages compared to using a single camera to measure leaf angles directly (as described herein), where existing prior art solutions require a human to pre-select leaves to be judged / evaluated (and even then, the margin for error is high); and the ease of deployment advantages compared to 3D capture methods for measuring leaf angle (e.g., LiDAR, photogrammetry, stereoscopic imaging), which are comparatively difficult to set up, use, and more computationally expensive at the point of capture. The one or more embodimentsdescribed herein, such as the method 500 of FIG. 5, eliminate the need to judge / identify leaf angle and instead track leaf movement over time. Particularly, plant movement patterns, which can be determined using leaf displacement, across a 2D plane is sufficient to provide accurate information about the moisture conditions of the soil that the plant is in.

[0079] At block 508, the system 126 determines whether to water the plant(s) of interest by supplying the leaf displacement data to the trained ML model 118. For example, the trained ML model 118 takes as input the leaf displacement and generates as output a prediction of whether to water the plant of interest.

[0080] At block 510, the system 126 generates a watering instruction responsive to determining to water the plant of interest. For example, the watering instruction can indicate how much water to give the plant (e.g., 1 cup, 200 milliliters, etc.) and / or when to water (e.g., now, tomorrow, next week, etc.). The watering instruction can be presented to the plant owner via a text and / or graphical interface, such as on a display (not shown) of the system 126 or another system, such as a user computing device (e.g., a smartphone, a wearable computing device, a tablet computer, and / or the like including combinations and / or multiples thereof). The watering instruction can be presented in an application executing on the system 126, as a text-message, as an e-mail message, and / or the like including combinations and / or multiples thereof.

[0081] Additional processes also may be included, and it should be understood that the process depicted in FIG. 5 represents an illustration, and that other processes may be added or existing processes may be removed, modified, or rearranged without departing from the scope of the present disclosure. It should also be understood that the processes depicted in FIG. 5 may be implemented as programmatic instructions stored on a non-transitory computer-readable storage medium that, when executed by a processor (e.g., the processing device 142) of a computing system (e.g., the system 126), cause the processor to perform the processes described herein.

[0082] The inference 104 is now described in more detail with reference to FIGS. 6A-6C. In particular, FIGS. 6A-6C depicts a flow diagram of a method 600 for performing inference using a machine learning model for monitoring plants according to one or more embodiments described herein. The method 600 can be implemented by any suitable system or device, such as the system 126 of FIGS. 1A-1C, the processing system 900 of FIG. 9, and / or the likeincluding combinations and / or multiples thereof. In particular, the method 600 performs the inference 104 of the machine learning training and inference system 100 of FIG. 1 A.

[0083] A set of time-lapse images are obtained (block 602), such as from the camera 146. The set of time-lapse images can be captured periodically, such as at intervals (e.g., every 5 to 10 minutes).

[0084] The set of time-lapse images (block 602) are input into an object detection algorithm (block 604, FIG. 6B). For each image, the object detection algorithm (block 604) detects each plant in the image and associates a set of pixels with each detected plant (block 606). This set of pixels may be expressed, for example, as a bounding box or mask.

[0085] Each plant representation in each image is input into a re-identification algorithm (block 607) which assigns an identity value to each representation (block 644). In an embodiment, identity values of representations of the same plant across images in the timelapse should be consistent to allow for tracking of movement of that plant over time.

[0086] Points of interest, or keypoints, for individual plants across images in the time-lapse are then identified using a feature detection algorithm (block 608). Keypoints can be any point identified on a plant that can be tracked across subsequent frames in a time-lapse. The feature detection algorithm (block 608) outputs coordinates of keypoints for each plant (block 610).

[0087] A feature tracking algorithm (block 612) is then used to track the location of each keypoint across images of the set of time-lapse images.

[0088] The output of the feature tracking algorithm (block 612) are pairs of coordinates of the same point of interest identified in a pair of images in the time-lapse. By subtracting the later coordinate from the earlier, we arrive at a movement vector (in pixels) for each point of interest (block 614). The set of movement vectors relating to all keypoints tracked on one plant across a pair of images in a time-lapse is referred to as the leaf displacement of that plant.

[0089] Leaf displacement of each plant (block 614) is then fed into a prediction algorithm(block 622), which represents the trained ML model 118, and is used to predict whether to water the plant of interest by generating a predicted value (block 624). According to one ormore embodiments described herein, the optimized prediction algorithm (block 622) can receive the leaf displacement (block 614) and / or local environment data (block 616, e.g. weather data), which can be transformed or pre-processed (block 618) prior to being fed into the prediction algorithm (block 622).

[0090] The predicted value (block 624) is converted to a recommendation to water or not to water (block 625). For example, the predicted value can be a predicted soil moisture value. If the predicted value (block 624) predicts the soil moisture level, watering can be recommended based on a predetermined threshold. The predicted soil moisture value can be compared to a threshold, historical trend, or other such data to determine whether to water the plant or not water the plant. For example, if the soil moisture value is below a first threshold, the recommendation may be to water now, while if the soil moisture value is above the first threshold but below a second threshold, the recommendation may be to water tomorrow. Additional thresholds can also be used. According to one or more embodiments described herein, the predicted value (block 624) can output a category (e.g., “needs water,” “does not need water,” etc.) directly, and watering can be recommended according to the category predicted. According to one or more embodiments described herein, the predicted value (block 624) can include an indication of days until watering (block 626). For example, the predicted value (block 624) can indicate to water in a certain number of days (e.g., 2 days, 4 days, etc.). The predicted days until watering can be added to data for the plant of interest (block 628). If the predicted value (block 624) is days until watering is recommended, watering can then be recommended if the value is below a certain number (e.g., below “1”). If the predicted value (block 624) is parameters of a PDF as described herein of the days until watering is recommended, watering is recommended responsive to an area of the predicted function around 0 “days to watering needed” is above a threshold.

[0091] A set of data can be produced that represents a watering instruction to instruct the user (e.g., plant owner) about watering (block 630). For example, the watering instruction can indicate how much water to give the plant (e.g., 1 cup, 200 milliliters, etc.) and / or when to water (e.g., now, tomorrow, next week, etc.).

[0092] If watering is recommended (block 632), the user (e.g., plant owner) can be provided with the watering instruction (block 634) (e.g., to water the plant now, to water the plant on a certain day, etc.).

[0093] Watering instructions can be displayed to the plant owner (i.e., user) (block 640). For example, an in-application notification, mobile alert, e-mail alert, text message alert, and / or the like including combinations and / or multiples thereof, can be used to provide information, such as a watering instruction. For example, the watering instruction can include a number of days until watering is recommended, an amount of water to provide, and / or the like including combinations and / or multiples thereof.

[0094] Additional processes also may be included, and it should be understood that the process depicted in FIG. 6 represents an illustration, and that other processes may be added or existing processes may be removed, modified, or rearranged without departing from the scope of the present disclosure. It should also be understood that the processes depicted in FIG. 6 may be implemented as programmatic instructions stored on a non-transitory computer-readable storage medium that, when executed by a processor (e.g., the processing device 142) of a computing system (e.g., the system 126), cause the processor to perform the processes described herein.

[0095] According to one or more embodiments described herein, method 300 of FIG. 3 and 600 of FIG. 6 can be changed to detect when a plant has been watered rather than when it needs watering. This is achieved by altering the pre-processing of ground truth soil moisture values 328 of FIG. 3 to indicate a watering event. A watering event might be indicated when a sudden increase in soil moisture is detected by the soil moisture sensor 160. The output of the prediction algorithm 320 and 622 (of FIG. 3 and 6 respectively) may also be altered to match the format of pre-processing ground truth moisture value 328 so that the outputs can be compared and losses can be calculated. Other processes within method 300 and method 600 would remain the same or may be very similar. Detecting watering events can be used to assure the plant owner that the system 126 is monitoring the plant of interest accurately.

[0096] FIGS. 7A, 7B, and 7C together depict a set of time-lapse images 701, 702, 703 respectively of a plant of interest 710 according to one or more embodiments described herein. The image 701 of FIG. 7A was captured at a first point in time, the image 702 of FIG. 7B was captured at a second point in time later than the first point in time, and the image 703 of FIG. 7C was captured at a third point in time later than the second point in time. In the images 701-703, multiple feature points are shown, including feature points 711, 712, and 713. As can be observed, the feature points 711-713 each move over time as shown by the set of time-lapse images 701-703. For example, the feature point 711 is in a first position in theimage 701, a second position in the image 702 that is relatively lower than the first position, and a third position in the image 703 that is relatively lower than the second position. The same is true of the feature points 712 and 713. Thus, over time, it can be observed that the leaves of the plant of interest 710 are “drooping” or otherwise showing signs that watering may be desired. The set of time-lapse images 701-703 are representative of both training data 112 and new data 122. The set of time-lapse images 701-703 can be captured by any suitable camera, such as the camera 146.

[0097] FIG. 8 depicts an example of an image 800 of a set of time-lapse images according to one or more embodiments described herein. That is, the image 800 is one of a set of timelapse images. As can be seen, the camera 146 can capture a single image that shows multiple plants, and the system 126 can isolate and identify individual plants (e.g., block 306 and block 342 of FIG. 3 and / or block 606 and block 644 of FIG. 6) to identify individual plants, for performing training 102 and / or inference 104.

[0098] It is understood that one or more embodiments described herein is capable of being implemented in conjunction with any other type of computing environment now known or later developed. For example, FIG. 9 depicts a block diagram of a processing system 900 for implementing the techniques described herein. In accordance with one or more embodiments described herein, the processing system 900 is an example of a cloud computing node of a cloud computing system. In examples, processing system 900 has one or more central processing units (“processors” or “processing resources” or “processing devices”) 921a, 921b, 921c, etc. (collectively or generically referred to as processor(s) 921 and / or as processing device(s)). In aspects of the present disclosure, each processor 921 can include a reduced instruction set computer (RISC) microprocessor. Processors 921 are coupled to system memory (e.g., random access memory (RAM) 924) and various other components via a system bus 933. Read only memory (ROM) 922 is coupled to system bus 933 and may include a basic input / output system (BIOS), which controls certain basic functions of processing system 900.

[0099] Further depicted are an input / output (I / O) adapter 927 and a network adapter 926 coupled to system bus 933. I / O adapter 927 may be a small computer system interface (SCSI) adapter that communicates with a hard disk 923 and / or a storage device 925 or any other similar component. I / O adapter 927, hard disk 923, and storage device 925 are collectively referred to herein as mass storage 934. Operating system 940 for execution on processingsystem 900 may be stored in mass storage 934. The network adapter 926 interconnects system bus 933 with an outside network 936 enabling processing system 900 to communicate with other such systems.

[0100] A display 935 (e.g., a display monitor) is connected to system bus 933 by display adapter 932, which may include a graphics adapter to improve the performance of graphics intensive applications and a video controller. In one aspect of the present disclosure, adapters 926, 927, and / or 932 may be connected to one or more I / O busses that are connected to system bus 933 via an intermediate bus bridge (not shown). Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols, such as the Peripheral Component Interconnect (PCI). Additional input / output devices are shown as connected to system bus 933 via user interface adapter 928 and display adapter 932. A keyboard 929, mouse 930, and speaker 931 may be interconnected to system bus 933 via user interface adapter 928, which may include, for example, a Super I / O chip integrating multiple device adapters into a single integrated circuit.

[0101] In some aspects of the present disclosure, processing system 900 includes a graphics processing unit 937. Graphics processing unit 937 is a specialized electronic circuit designed to manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display. In general, graphics processing unit 937 is very efficient at manipulating computer graphics and image processing, and has a highly parallel structure that makes it more effective than general-purpose CPUs for algorithms where processing of large blocks of data is done in parallel. Thus, as configured herein, processing system 900 includes processing capability in the form of processors 921, storage capability including system memory (e.g., RAM 924), and mass storage 934, input means such as keyboard 929 and mouse 930, and output capability including speaker 931 and display 935. In some aspects of the present disclosure, a portion of system memory (e.g., RAM 924) and mass storage 934 collectively store the operating system 940 to coordinate the functions of the various components shown in processing system 900.

[0102] Referring now to FIGS. 10A-10C and FIGS. 11A-11C, methods are depicted for extending system 126 in order to predict when to a water a plant that does not exhibit movement in response to water stress. Some plants, such as cacti and succulents, are substantially stationary and do not exhibit movement in response to water stress. In these cases, monitoring movement directly would not produce suitable data for an algorithm todetermine ‘when to water’. By monitoring the movement of other plants (i.e. plants that do move) in the immediate vicinity (block 1046 of FIG. 10 and 1146 of FIG. 11), information on the ambient environment can be collected, such as how the ambient environment is affecting plant water usage. This information can then be used as an input into a ‘When to water stationary plant’ (block 1020 of FIG. 10 and block 1120 of FIG. 11) prediction algorithm to predict when to water a target stationary plant. As used herein the stationary plant and the non-stationary plant are in the same vicinity when the plants are located proximate to each other such that each plant is subject to substantially the same environmental parameters.

[0103] Training of such ‘When to water stationary plant’ algorithm is described in FIG. 10C. In this embodiment, direct sensing soil moisture sensors 160 are once again deployed, this time monitoring the soil moisture in plant pots occupied by plants that do not move in response to water stress. These soil moisture sensors provide the ground truth values (block 1028) that the prediction algorithm (block 1028) is evaluated against. The training loop (block 1001) is the same or similar to the training loop for the ‘When to water’ prediction algorithm (block 301 of FIG. 3). Inputs to the ‘When to water stationary plant’ prediction algorithm (block 1020) come from two non-optional sources: an ambient water loss factor (block 1056) derived from leaf displacement of non-stationary plants in the vicinity of the target plant (block 1046) and information on traits / parameters of the target (stationary) plant that can affect its water usage (block 1058). In an embodiment, the ambient loss factor is used as a proxy for how much water a plant is likely to lose just by being in the micro-climate of its immediate surroundings. The numerical expression of the ambient water loss factor can be a real number or vector of real numbers calculated using the prediction history of a trained ‘When to water’ model 118 (block 1050) utilizing as inputs the leaf displacement of non- stationary plants (block 1046) and traits / parameters affecting water usage of each such non- stationary plant (block 1052). The ‘When to water’ prediction history (block 1050) is calculated in the same or similar manner to those described in FIG. 5. The set of traits / parameters affecting water usage of the non-stationary plants (block 1052) are used in conjunction with the recent ‘When to water’ prediction history (block 1050) to isolate the effects of the local micro-climate on water loss (block 1054). The effect of soil moisture loss caused by the ambient environment is expressed as a universal ambient water loss factor (block 1056). Examples of traits / parameters affecting water usage of a plant may include, but are not limited to: type of plant, size of plant, size of plant pot, material of plant pot, and type of soil. In order to predict whether to water the non-stationary target plant, the calculatedambient water loss factor (block 1056) is combined with the traits / parameters affecting water usage of the target plant (block 1058) as inputs to a “When to water stationary plant” prediction algorithm (block 1020), which predicts a value that can be interpreted as a recommendation to water the plant or not. The ambient water loss factor (block 1056) may be calculated using any algorithm that takes both continuous and categorical variables as inputs and outputs a real number or vector of real numbers. Optimization of both the ambient water loss factor (block 1056) and the “When to water stationary plant” prediction algorithm (block 1020) may happen simultaneously if the loss from the prediction algorithm can be propagated to the algorithm calculating the ambient water loss factor (e.g. if both algorithms have differentiable loss functions, the algorithms can be stacked into the same training pipeline).

[0104] Inference of the ‘When to water stationary plant’ algorithm is detailed in FIGS. 11A-11B. The operation, its inputs, and outputs of the ‘When to water stationary plant’ algorithm (block 1120) is similar to the functioning of the ‘When to water stationary plant’ algorithm detailed in training (block 1020 of FIG. 10), the primary difference is that the algorithm is already trained and no loss is calculated, nor are the parameters of the algorithm altered. The target (stationary) plant being monitored and its associated data (1158), as well as the non- stationary plants and their associated data (block 1146 and 1152) may be new, unseen data, for the prediction algorithm (block 1120). The processes following the prediction, i.e. those following the determination of the predicted value (block 1124) follow the same, or similar processes to those described in the inference of the ‘When to water’ prediction algorithm (FIGS. 6A-6C).

[0105] Various embodiments are described herein with reference to the related drawings. Alternative embodiments can be devised without departing from the scope of the claims. Various connections and positional relationships (e.g., over, below, adjacent, etc.) are set forth between elements in the following description and in the drawings. These connections and / or positional relationships, unless specified otherwise, can be direct or indirect, and the embodiments described herein are not intended to be limiting in this respect. Accordingly, a coupling of entities can refer to either a direct or an indirect coupling, and a positional relationship between entities can be a direct or indirect positional relationship. Moreover, the various tasks and process steps described herein can be incorporated into a more comprehensive procedure or process having additional steps or functionality not described in detail herein.

[0106] The following definitions and abbreviations are to be used for the interpretation of the claims and the specification. As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” “contains” or “containing,” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a composition, a mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but can include other elements not expressly listed or inherent to such composition, mixture, process, method, article, or apparatus.

[0107] Additionally, the term “exemplary” is used herein to mean “serving as an example, instance or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. The terms “at least one” and “one or more” may be understood to include any integer number greater than or equal to one, i.e. one, two, three, four, etc. The terms “a plurality” may be understood to include any integer number greater than or equal to two, i.e. two, three, four, five, etc. The term “connection” may include both an indirect “connection” and a direct “connection.”

[0108] The terms “about,” “substantially,” “approximately,” and variations thereof, are intended to include the degree of error associated with measurement of the particular quantity based upon the equipment available at the time of filing the application. For example, “about” can include a range of ± 8% or 5%, or 2% of a given value.

[0109] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments described herein. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

[0110] The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments described herein.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for training a machine learning model to determine whether to water a plant of interest, the method comprising: receiving training data, the training data including a set of time-lapse images for each of a plurality of plants; for each of the plurality of plants: determining leaf displacement for the plant using the set of time-lapse images for the plant; predicting, based on the leaf displacement, whether to water the plant; and comparing a result of the prediction with a ground truth soil moisture value taken by a soil moisture sensor associated with the plant; and training the machine learning model based on each of the results of the comparison for each of the plurality of plants, wherein the training identifies optimal parameters of a machine learning prediction algorithm that uses the leaf displacement across time as an input and outputs a predicted value that can be used to determine whether or not the plant should be watered.

2. The computer- implemented method of claim 1, wherein the ground truth soil moisture value is one of a plurality of ground truth soil moisture values, wherein each of the plurality of ground truth soil moisture values is associated with at least one of the images of the set of time-lapse images for each of the plurality of plants.

3. The computer- implemented method of claim 1, wherein the plurality of ground truth soil moisture values are determined using one or more soil moisture sensors.

4. The computer- implemented method of claim 1, wherein predicting whether to water the plant is further based on environmental data.

5. The computer- implemented method of claim 4, wherein the environmental data comprises at least one of a temperature value, a humidity value, or a light level value.

6. The computer- implemented method of claim 4, wherein the environmental data is received from one or more sensors associated with at least one of the plurality of plants.

7. The computer- implemented method of claim 4, wherein the environmental data is received from public weather data.

8. The computer- implemented method of claim 1, wherein the leaf displacement is determined based on any visible feature on the plurality of plants.

9. The computer- implemented method of claim 8, wherein the visible feature of the plurality of plants comprise tips of the leaves.

10. The computer-implemented method of claim 8, wherein the visible features of the plurality of plants comprises at least one of leaves, stems, stalks, shoots, buds, flowers and fruit.

11. The computer- implemented method of claim 8, wherein the features are determined using a feature detection algorithm.

12. The computer- implemented method of claim 1, wherein the machine learning model has prediction algorithm parameters associated therewith, wherein the prediction algorithm parameters are optimized using an optimization algorithm that calculates loss and changes the prediction algorithm parameters based on the loss.

13. A computer-implemented method for determining whether to water a plant of interest, the method comprising: receiving a set of time-lapse images of the plant of interest; detecting features of the plant of interest using the set of time-lapse images; determining leaf displacement for the plant of interest using the features; determining, using a trained machine learning model, whether to water the plant of interest, wherein the trained machine learning model takes as input the leaf displacement and generates as output a prediction of whether to water the plant of interest; and generating a watering instruction responsive to determining to water the plant of interest.

14. The computer-implemented method of claim 13, wherein the set of time-lapse images of the plant of interest are captured using a camera in proximity to the plant of interest.

15. The computer- implemented method of claim 12, wherein the features of the plant of interest are any visible feature on the plurality of plants.

16. The computer- implemented method of claim 15, wherein the visible feature of the plurality of plants comprise tips of the leaves.

17. The computer-implemented method of claim 15, wherein the visible features of the plurality of plants comprises at least one of leaves, stems, stalks, shoots, buds, flowers and fruit.

18. The computer- implemented method of claim 13, wherein detecting the features of the plant of interest is performed using a feature detection algorithm.

19. The computer- implemented method of claim 13, further comprising: presenting the watering instruction to a user on a display of a user computing device.

20. The computer-implemented method of claim 13, wherein each of the leaf displacement indicates a displacement of one of the features of the plant of interest feature over time.

21. A system for determining whether to water a plant of interest, the system comprising: a display; a camera to capture a set of time-lapse images of the plant of interest; a memory comprising computer readable instructions; and a processing device for executing the computer readable instructions, the computer readable instructions controlling the processing device to perform operations comprising: detecting features of the plant of interest using the set of time-lapse images; determining leaf displacement for the plant of interest using the features; determining, using a trained machine learning model, whether to water theplant of interest, wherein the trained machine learning model takes as input the leaf displacement and generates as output a prediction of whether to water the plant of interest; generating a watering instruction responsive to determining to water the plant of interest; and presenting the watering instruction to a user on the display.

22. A system comprising: a display; a camera to capture a set of time-lapse images of a non- stationary plant of interest; a memory comprising computer readable instructions; and a processing device for executing the computer readable instructions, the computer readable instructions controlling the processing device to perform operations comprising: detecting features of the non- stationary plant of interest using the set of time-lapse images; determining leaf displacement for the plant of interest using the features; determining, using a trained first machine learning model, whether to water the non- stationary plant of interest, wherein the trained machine learning model takes as input the leaf displacement and generates as output a prediction of whether to water the non- stationary plant of interest; determining first parameters affecting rate of water usage in the non-stationary plant of interest; determining a ambient water loss factor based on the first parameters and historical data of predictions from the trained first machine learning model; determining second parameters affecting rate of water usage in a stationary plant of interest, the stationary plant being in the vicinity of the non-stationary plant; determining, using a trained second machine learning model, whether to water thestationary plant of interest, wherein the trained machine learning model takes as input the ambient water loss factor and second parameters and generates as output a prediction of whether to water the stationary plant of interest; generating a watering instruction responsive to determining to water the stationary plant of interest; and presenting the watering instruction to a user on the display.

23. A computer-implemented method for determining whether to water a stationary plant of interest, the method comprising: receiving a set of time-lapse images of a non-stationary plant of interest; determining leaf displacement for the plant of interest based at least in part on the set of time-lapse images; determining, using a trained first machine learning model, whether to water the nonstationary plant of interest, wherein the trained machine learning model takes as input the leaf displacement and generates as output a prediction of whether to water the plant of interest; determining first parameters affecting rate of water usage in the non-stationary plant of interest; determining a ambient water loss factor based on the first parameters and historical data of predictions from the trained first machine learning model; determining second parameters affecting rate of water usage in a stationary plant of interest, the stationary plant of interest being in the vicinity of the non-stationary plant of interest; determining, using a trained second machine learning model, whether to water the stationary plant of interest, wherein the trained machine learning model takes as input the ambient water loss factor and second parameters and generates as output a prediction of whether to water the stationary plant of interest; generating a watering instruction responsive to determining to water the stationaryplant of interest.

24. A computer-implemented method for training a machine learning model to determine whether to water a stationary plant of interest, the method comprising: receiving historical data on predictions of whether to water a non-stationary plant of interest; determining first parameters affecting a rate of water usage in the nonstationary plant; determining an ambient water loss factor based on the first parameters and the historical data; determining second parameters affecting rate of water usage in a stationary plant of interest receiving training data that includes a set soil of moisture data for the stationary plant of interest; predicting, based on the ambient water loss factor, the second parameters, whether to water the stationary plant; and comparing a result of the prediction with a ground truth soil moisture value taken by a soil moisture sensor associated with the stationary plant; and training the machine learning model based on each of the results of the comparison for the stationary plant, wherein the training identifies optimal parameters of a machine learning prediction algorithm that uses a leaf displacement of a non-stationary plant in the vicinity of the stationary plant across time as an input and outputs a predicted value that can be used to determine whether or not the stationary plant should be watered.

Images