System and method for crop assessment using computer vision

Abstract

According to an aspect, there is provided a systems and methods for assessing crops. The crop assessment includes, for each given plant of a plurality of plants, assigning a unique identifier to the given plant, emitting light toward the given plant using at least one light emitter, capturing an image using at least one imaging sensor, detecting the given plant and a predictor of a crop output of the plant in the image, estimating a location of the given plant based on a signal from a geo-positioning subsystem, and generating at least one crop metric predictive of yield of the given plant based on the crop output, and generating a crop output data structure with data defining the at least one crop metric and the location, in association with the unique identifier, for each of the plurality of plants.

Description

SYSTEM AND METHOD FOR CROP ASSESSMENT USING COMPUTER VISIONCROSS-REFERENCE

[0001] This application claims benefit and priority from US Provisional Patent Application No. 63 / 568,789, entitled “SYSTEM AND METHOD FOR CROP ASSESSMENT USING COMPUTER VISION”, filed on March 22nd, 2024, the entire contents of which are incorporated herein by reference.FIELD

[0002] Embodiments of the present disclosure generally relate to the field of crop assessment and / or management, and more specifically, embodiments relate to devices, systems and methods for improved crop assessment using computer vision.BACKGROUND

[0003] Agricultural production, such as fruit production, is harder than ever. Aggressive weather events can hurt or outright destroy crop yields. Pests and disease can spread through crops and severely reduce crop yields. All of these may impact the volume of crops produced (e.g., the total number of suitable crop units produced and / or a reduction in the size of suitable crop units).

[0004] Growers collect data manually to determine farm strategies. In this regime, the plants are inspected by agents in the field making data collection a manually intensive process. Only a small percentage of plants can be analysed. This may impact the accuracy and precision of crop estimates based on the few plants actually analyzed.

[0005] Post-harvest costs and sales are based on these estimates which may lead to misalignment of supply and demand. This may further misallocate food resources which may lead to wasted food in some places and too little food provided elsewhere.

[0006] Improvement in the area of crop assessment is desirable.SUMMARY

[0007] According to an aspect, there is provided a computer-implemented system for assessing crops. The system including at least one light emitter, at least one imaging sensor including an infrared imaging sensor configured to detect infrared light, a geo-positioning subsystem, and a processing subsystem that includes one or more processors and one or more memories coupled with the one or more processors. The processing subsystem configured to cause the system to, for each given plant of a plurality of plants, assign a unique identifier to the given plant, emit light toward the given plant using the at least one light emitter, capture an image using the at least one imaging sensor, detect the given plant and a predictor of a crop output of the plant in the image, estimate a location of the given plant based on a signal from the geo-positioning subsystem, and generate at least one crop metric predictive of yield of the given plant based on the crop output, and generate a crop output data structure with data defining the at least one crop metric and the location, in association with the unique identifier, for each of the plurality of plants.

[0008] According to a further aspect, there is provided a method of assessing crops. The method includes, for each given plant of a plurality of plants, assigning a unique identifier to the given plant, emitting light toward the given plant using at least one light emitter, capturing an image using at least one imaging sensor, detecting the given plant and a predictor of a crop output of the plant in the image, estimating a location of the given plant based on a signal from a geo-positioning subsystem, and generating at least one crop metric predictive of yield of the given plant based on the crop output, and generating a crop output data structure with data defining the at least one crop metric and the location, in association with the unique identifier, for each of the plurality of plants.

[0009] Many further features and combinations thereof concerning embodiments described herein will appear to those skilled in the art following a reading of the instant disclosure.DESCRIPTION OF THE FIGURES

[0010] In the figures, embodiments are illustrated by way of example. It is to be expressly understood that the description and figures are only for the purpose of illustration and as an aid to understanding.

[0011] Embodiments will now be described, by way of example only, with reference to the attached figures, wherein in the figures:

[0012] FIG. 1A is a perspective view of a crop assessment system, according to some embodiments.

[0013] FIG. 1B is a front view of the crop assessment system of FIG. 1A, according to some embodiments.

[0014] FIG. 1C is a side view of the crop assessment system of FIG. 1A, according to some embodiments.

[0015] FIG. 1D is a rear view of the crop assessment system of FIG. 1A, according to some embodiments.

[0016] FIG. 2 illustrates the crop assessment system of FIG. 1A mounted on a vehicle, according to some embodiments.

[0017] FIG. 3 is block schematic diagram of the system of the crop assessment system of FIG. 1A, according to some embodiments.

[0018] FIG. 4 is block schematic diagram of the interfaces of the crop assessment system of FIG. 1A, according to some embodiments.

[0019] FIG. 5A shows an RGB image next to a near-IR image captured under difficult lighting conditions using the crop assessment system of FIG. 1A, according to some embodiments.

[0020] FIG. 5B shows tissue damage in a near-IR / IR image using the crop assessment system of FIG. 1A, according to some embodiments.

[0021] FIG. 6 illustrates different information outputs provided by the crop assessment system of FIG. 1A, according to some embodiments.

[0022] FIG. 7 illustrates a dashboard output provided by the crop assessment system of FIG. 1A, according to some embodiments.

[0023] FIG. 8 is a process diagram of a method of assessing crops, according to some embodiments.

[0024] FIG. 9 is a schematic diagram of computing device, according to some embodiments.DETAILED DESCRIPTION

[0025] Provided herein are systems and methods to aide in crop assessment. In particular, provided herein is a system that is capable to scanning through a crop (e.g., an orchard) and assessing the crop yield (e.g., size of crop unit (e.g., fruit), number of crop units per plant, etc.). The system is configured to travel through a crop (e.g., mounted on a vehicle) and scan the crops for information. Such a system may use machine learning methods to identify individual plants and the crop units associated with each of the plants. The system may further be configured to obtain the location data of the plants and provide geographic representations (such as a 2D map) of the crops with information related to each of the plants provided thereon.

[0026] Such systems may be advantageous because the information from the crops can be obtained semi-automatically (without manual counting by agents) and can be carried out on most or all of the plants in the crop as opposed to only a few . This may help in developing more effective farming strategies than could be developed if only a few plants were assessed. This may enhance the accuracy and precision of crop estimates. Post-harvest costs and sales that are based on these enhanced estimates may lead to better alignment of supply and demand. This may further properly allocate food resources and reduce food waste.

[0027] The systems and methods described herein may make use of active lighting. Such systems may use light emitters on the system to illuminate the crops. This can serve the advantage of providing more consistent lighting conditions for the system’s imaging sensors. This may aide when outside lighting conditions are suboptimal. It may also ensure that the system’s imaging sensors are receiving more consistently lit images which may help any machine learning algorithms more accurately identify crop yields.

[0028] Active lighting may further make use of non-visible light. For example, the imaging sensor may be configured to receive multispectral lighting. Multispectral lighting may be provided by a light source emitting radiation in a few discrete (or approximately discrete) spectral bands (as opposed to a broadband light source). In some embodiments, a monochrome sensor can capture the scene illuminated with different light sources one after another (time-multiplexed) and generate a multispectral image. In some embodiments, a sensor equipped with a filter array could capture multiple bands at the same time. An advantage of using non-visible lighting may be that it may reflect or emit from the crop units, but transmitthrough other plant components (e.g., leaves) thereby making it possible for the system to detect crop units that are occluded by such plant components.

[0029] FIG. 1A is a perspective view of a crop assessment system 100, according to some embodiments. FIG. 1B is a front view of the crop assessment system 100, according to some embodiments. FIG. 1C is a side view of the crop assessment system 100, according to some embodiments. FIG. 1D is a rear view of the crop assessment system 100, according to some embodiments.

[0030] The crop assessment system 100 can include, for example, light emitters 102, imaging sensors 104, a protective glass 106, a positioning system antenna and receiver 108, visual indicators and antenna 110, heatsink 112, an enclosure 114, a display 118a, a button 118b (the display 118a and the display 118b together forming part of the buttons, LED, display 118), an ethernet connector 125, and a power connector 121. The crop assessment system 100 can be configured to survey and assess crops. For example, the crop assessment system 100 can use active lighting to image the crops and process the image data to generate a crop metric indicative of crop yield. This information may be combined with information from the positioning system (e.g., a geo-positioning subsystem) to associate plants with their locations and generate an output (e.g., a 2D output).

[0031] The light emitters 102 may be configured to emit active lighting. The light emitted by the light emitters 102 may be visible light. The light emitted by the light emitters 102 may be infrared light. The light emitted by the light emitters 102 may be selected to transmit through leaves or other occlusions to the crop units (e.g., fruits). In some embodiments, the light may be selected because it specifically reflects or is absorbed and reemitted by the crops in question. IR (or near-IR) light absorption and emission of plant tissue can be linked to its photosynthetic activity, and can be used to assess plant / tissue health and detect various issues. For example, the commonly used normalized difference vegetation index (NDVI) compares the relative amounts of red and near-IR light to assess photosynthetic activity or plant health. Tissue damage can be detected in near-IR, as it can lead to reduced near-IR emission / reflection. In indoor settings, near-IR / IR light might be desirable, as it doesn’t interfere with other (visible) light sources, so the system can acquire consistent images regardless of other light sources present in the room.

[0032] The crop assessment system 100 may have a plurality of light emitters 102. Each of the plurality of light emitters 102 may emit the same range of wavelengths of light or each light emitters 102 may emit a different range of wavelengths of light. Different light emitters 102 may be activated based on the type of crop being assessed (e.g., where different ranges may be optimal for different types of crops). While FIG. 1A shows four configurations of light emitters 102 and imaging sensors 104, other configurations and numbers of light emitters 102 and / or imaging sensors 104 can be used. For example, the front panel can be modified to accept more or less light emitters 102.

[0033] In some embodiments, the light emitters 102 that emit near-IR can be positioned in the central two positions and an imaging sensor 104 capable of receiving near-IR in the right-most position. Not aligning the light source with the imaging sensor 104 can mean better contrast in certain settings (if the light comes from an angle, it can be easier to see surface structure and distinguish between objects due to the slight shadow).

[0034] In some embodiments, the light emitters 102 may activate based on energy or imaging needs. For example, the crop assessment system 100 may be configured to determine whether the images received from the imagining sensors 104 meet a quality threshold and if not the crop assessment system 100 may be configured to adjust the lighting provided by the light emitters 102 until the images meet the quality threshold (or some other criteria is satisfied). In some embodiments, the crop assessment system 100 can control the direction of the light emitters 102.

[0035] In some embodiments, the light emitters 102 can be turned on and off with precise timing (e.g., for 1 ms when capturing a frame to overlap with the camera exposure interval). Pulsed operation can allow the system to put all the energy into the duration of the frame exposure and can achieve much higher “effective” output power, similar to a conventional camera flash.

[0036] In some embodiments, the light emitters 102 may include, for example, three 850 nm light emitters. Using 850 nm light may only affect the near-IR images and leave the RGB images unaffected. This can enable applications where the reflected / transmitted radiation of different light sources to be measured simultaneously, but independently (e.g., using natural sunlight and artificial 850 nm light).

[0037] The imaging sensors 104 can be configured to capture images of the crops. The imaging sensors 104 may be configured to capture images in the, for example, visible light spectrum or infrared light spectrum. The imaging sensors 104 may be configured to capture images at specific rates. The rate of image capture may be based in part on the speed with which the crop assessment system 100 is travelling through the crops. In some embodiments, the rate of image capture may be configured to prevent image capture artefacts such as motion blur.

[0038] In some embodiments, there may be a plurality of imaging sensors 104. In some embodiments, each of the plurality of imaging sensors 104 may generally be oriented to capture images in the same direction. In some embodiments, each of the plurality of imaging sensors 104 may be configured to capture images at different orientations. In some embodiments, each of the plurality of imaging sensors 104 may be sensitive to different spectral bands. For example, each of the imaging sensors 104 may be sensitive to the spectral band provided by one of the light emitters 102 or sensitive to a spectral band reflected or reemitted by fruit from one of the light emitters 102.

[0039] In some embodiments, the at least one imaging sensor 104 is configured to detect multispectral light. Such embodiments may include a standard sensor without any filter (it can detect any light compatible with its spectral response characteristics, i.e. , any visible and near- IR light for a conventional complementary metal-oxide-semiconductor (CMOS) sensor).

[0040] In some embodiments, the crop assessment system 100 may include three different imaging units comprised of imaging sensors 104 and optics including a wide-angle, colour (RGB Bayer filter array), a telephoto, colour (RGB Bayer filter array), and a wide-angle, monochrome (near-IR bandpass filter). The framerates of the imaging sensors 104 may be about 1 to about 160 fps.

[0041] The light emitters 102 and the imaging sensors 104 may be encased in the crop assessment system 100 behind protective glass 106. The protective glass 106 may be fabricated from any material (e.g., plastic, plexiglass, glass, etc.) suitable for field use. The protective glass 106 may generally be transmissible for the spectral bands for use with the crop assessment system 100 (e.g., intended to be emitted by the light emitters 102 and / or received by the imaging sensors 104).

[0042] The crop assessment system 100 may also include a positioning system antenna and receiver 108. The positioning system antenna and receiver 108 may provide, for example, position, velocity, and time for the crop assessment system 100. For example, the positioning system antenna and receiver 108 may provide the position of the crop assessment system 100 in an absolute context (e.g., as a global positioning system) or in a relative context (e.g., it may be configured to determine the location of the crop assessment system 100 within the crop itself). The positioning system antenna and receiver 108 may, for example, be part of a global navigation satellite system (GNSS) such as a global positioning system (GPS). This may include a multiband (and / or multi-constellation) GNSS antenna and receiver. It may also include RTK technology for cm-level positioning.

[0043] In some embodiments, the crop assessment system 100 may be configured to generate a crop metric indicative of crop yield. The crop assessment system 100 may further be configured to use the positioning system antenna and receiver 108 to determine the position of the plants that the crop assessment system 100 is currently investigating and associate the crop metric (or other metric) with the location of the plant. Advantages of this approach can provide localized information about the crops. Such information might be useful to identify regions of the crop which are more or less productive than expected (e.g., due to soil conditions, disease, etc.).

[0044] The visual indicators and antenna 110 can provide information about the status of the crop assessment system 100 and can communicate with external devices. The visual indicator and antenna 110 may include, for example a WiFi and / or Bluetooth antenna to allow for radio communication. The visual indicators and antenna 110 may include LEDs to serve as indicators that can signal the system status, such as “normal operation” or “recording”.

[0045] The heatsink 112 can optionally provide cooling and heat dissipation to the system 100. In some embodiments, the components of the crop assessment system 100 may produce heat and for proper and continued operation of the crop assessment system 100, it may be beneficial to remove the heat generated by these components. Further, the crop assessment system 100 may be configured to operate under a variety of different conditions which may include in hot weather and the heatsink 112 may be helpful to dissipate heat in such situations.

[0046] The enclosure 114 may encase some of the components of the crop assessment system 100. For example, it may encase the light emitters 102 along with the imaging sensors104. It may do so with the protective glass 106. The enclosure 114 may be waterproof and ruggedized for field use.

[0047] The display 118a may provide output information or status information to the user. For example, it may provide the user with error messages to help the user troubleshoot an issue with the crop assessment system 100.

[0048] The button 118b may provide the user with a means of inputting commands to the system 100.

[0049] The ethernet connector 125 may enable an ethernet connection for uploading and / or downloading information to and from the crop assessment system 100. This can help download information from the system 100 once it has passed through the crop. Other data transfer methods are also possible.

[0050] The power connector 121 can enable power to connect to the crop assessment system 100 to power the system.

[0051] The crop assessment system 100 may be modular. For example, the crop assessment system 100 illustrated has four separate imaging sensor 104 subsystems. Each of these subsystems can be individually modified, for example, with the same or different light emitters 102 and / or imaging sensors 104. Several different light emitters 102 can be placed inside the front panel and aligned with the imaging sensors for optimal illumination of the scene. In some embodiments, the light emitters 102, the imaging sensors 104, and other sensors can be reconfigured and swapped easily. For example, two identical near-IR imaging sensors 104 could be used in two slots for standard stereo vision applications. As a further example, arbitrary bandpass / lowpass / highpass filters with multiple monochrome sensors could be used to capture some arbitrary bands simultaneously. In some embodiments, different light emitters 102 of different wavelengths may simultaneously be controlled. In some embodiments, the imaging sensors 104 can, but do not need to be synchronized with each other and / or the light emitters 102.

[0052] In some embodiments, the crop assessment system 100 may have more or less input modalities than described above. For example, in some embodiments, the crop assessment system 100 can include an inertial measurement unit (I MU) to measure acceleration and / or rotation of the crop assessment system 100, a temperature sensor to measure the temperature,a radar to measure, for example, depth, light detections and ranging (LiDAR) sensor to measure, for example, depth, an ambient light sensor to measure ambient light, a humidity sensor to measure humidity. Crop assessment systems 100 with one or more differing input modalities may be capable of using the information from these modalities to further control the crop assessment system 100 or to provide additional information when generating the crop metric indicative of crop yield. For example, the ambient light sensor may be used to determine the number of light emitters 102 to activate and at what intensity.

[0053] In some embodiments, the crop assessment system 100 may include and / or control the operation of additional devices. For example, the crop assessment system 100 may include or control agricultural equipment (e.g., variable-rate spraying, robotic pruning, picking, etc.), industrial equipment (e.g., factory automation, mine monitoring, etc.), etc.

[0054] The crop assessment system 100 can provide a specific temporal resolution for the control signal (e.g., imaging sensor 104 control, light emitter 102 control, etc.). The resolution may be optimized based on the expected field use of the crop assessment system 100 (e.g., expected speed of movement through the crops). In some embodiments, the crop assessment system 100 can provide 83 ns resolution for the control signal.

[0055] In some embodiments, the crop assessment system 100 can quickly adjust the light source to capture certain properties of an imaged object (e.g. defects of a certain kind), based on the scene / object imaged. In some embodiments, the crop assessment system 100 may adjust imaging and lighting parameters based on the condition of the plant imaged, its surroundings, or location to capture specific properties or to improve the image quality. In some embodiments, the crop assessment system 100 can be used as a real-time control unit for agricultural equipment, including light-based equipment, such as laser-based weeding systems, UV-based pest control, etc. In some embodiments, the crop assessment system 100 may be mounted to a picking robot and can be used to dynamically illuminate the scene if an object of interest is present (turning off the light otherwise to minimize interference with other robots). In some embodiments, the crop assessment system 100 can be used for high-resolution spectral imaging of fast-moving objects with a single or small number of imaging sensors 104. For example, depending on the object, capturing an image using either spectral band A or band B could be desirable. In such cases, to maximize image quality, a monochrome sensor could be used with light sources of different wavelengths and depending on the shape (or another feature) of the object inferred in a first frame (1), light source A or B can be conditionallyswitched on for the next frame (2). This may be similar to a fast-moving camera with static scenes.

[0056] In operation, some embodiments of the crop assessment system 100 may travel through a crop illuminating the crops with the light emitters 102 using, for example, infrared light. The imaging sensors 104 may capture one or more images of the crops illuminated by the light emitters 102. The imaging sensors 104 may be configured to generate a stream of images (e.g., live feed). The imaging sensors 104 may make use of infrared image capture to obviate the hindrance caused by occlusions to the crop units. The images may be processed by a processing unit on the crop assessment system 100 or processed on an external device. The images may be analyzed using models trained with machine learning to identify individual plants (e.g., based on the trunks of trees) and to identify the crop units thereon. The models may be trained to assess the number of crop units, the volume of crop units, the crop load (e.g., the number of crop units per plant), the crop surface area (e.g., canopy leaf area), the crop diameter (e.g. fruit diameter), and the crop density (e.g., blossom cluster density). The models may also be configured to detect other features such as canopy density, the presence of pests and / or disease, nutrient deficiencies, blemishes (and percentage of crop units with blemishes), and colour grading. The system may take the information pulled from the images and associate it with the locations of the plant (e.g., as determined by the positioning system). This may all be combined to produce a 2D or 3D map of the crop with the crop metrics super imposed thereon.

[0057] In some embodiments the system may further be configured to assess the full crop for further insights (e.g., detection of pests via regions of underperforming crops).

[0058] The output may be used for farming strategies such as dormant pruning, pest / disease detection and thinning. The output may also be used for yield prediction and post-harvest cost optimization. It may further be used for program planning.

[0059] The outputs may give a prediction for the yield of the crop as it presently exists. The output may further be configured to forecast the yield of the crop when it is harvested.

[0060] According to an aspect, there is provided a computer-implemented system for assessing crops 100. The system 100 including at least one light emitter 102, at least one imaging sensor 104, including an infrared imaging sensor configured to detect infrared light, a geo-positioning subsystem, a processing subsystem 117 that includes one or more processors and one or more memories coupled with the one or more processors. The processingsubsystem 117 configured to cause the system to, for each given plant of a plurality of plants, assign a unique identifier to the given plant, emit light toward the given plant using the at least one light emitter 102, capture an image using the at least one imaging sensor 104, detect the given plant and a predictor of a crop output of the plant in the image, estimate a location of the given plant based on a signal from the geo-positioning subsystem, and generate at least one crop metric predictive of yield, crop load, the crop surface area, the crop diameter, and the crop density, or crop health of the given plant based on the crop output, and generate a crop output data structure with data defining the at least one crop metric and the location, in association with the unique identifier, for each of the plurality of plants.

[0061] In some embodiments, the crop output is estimates for each plant based on, for example, its overall appearance in the image and various features extracted from the image.

[0062] In some embodiments, multiple image frames taken from different angles (e.g., structure from motion) can be combined to estimate the position of the plant in combination with the geo-positioning subsystem.

[0063] While the foregoing description described an embodiment where the crop assessment system 100 includes light emitters 102 and imaging sensors 104 on the same device (as illustrated), in some embodiments, the light emitters 102 may be provided on a separate device than the imaging sensors 104. For example, the crop assessment system 100 may include the imaging sensors 104 on a device as illustrated in FIG. 1A - FIG. 1D while the light emitters 102 may be provided on a different device (not pictured) at a distance from the imaging sensors 104. This may be done to achieve better illumination, to use larger / different light emitters 102, or to render the lighting module optional to the system 100. This may also enable the light emitters 102 to illuminate plants from the rear (e.g., to analyze the plants using transmission spectra as opposed to or in combination with reflection spectra). In such embodiments, the system 100 may be configured to make use of multiple light emitters 102 that may be distributed (or synchronized) over different distances (e.g. to scan a large plant or wide row). In such embodiments, the device including the light emitters 102 can be synchronized with the device including the image sensors 104 through, for example, an external or built-in mechanism / signal, such as GPS (which both devices may receive independently of one another) or a local time pulse generator (which may trigger both devices).

[0064] FIG. 2 illustrates the crop assessment system 100 mounted on a vehicle 200, according to some embodiments.

[0065] In some embodiments, the crop assessment system 100 may be configured to be mounted on a vehicle 200. In such embodiments, the crop assessment system 100 may be interoperable with a mount 202 which can hold the crop assessment system 100 in an orientation to carry out crop assessment as the vehicle 200 travels through the crops. In some embodiments, the crop assessment system 100 may be built directly into a vehicle 200. In some embodiments, the crop assessment system 100 may be built into an autonomous vehicle. In such embodiments, the crop assessment system 100 may control or provide input into the control unit of the autonomous vehicle.

[0066] FIG. 3 is block schematic diagram of the system of the crop assessment system 100, according to some embodiments.

[0067] In some embodiments, the crop assessment system 100 may include a real-time controller 116, main processor 117, buttons, LEDs, display 118, LISB-C port 120, GB Ethernet 122, non-volatile memory express storage 124, WiFi / BT 126 and antenna 128, LTE modem 130 and antenna 132, IMU 138, and GNSS receiver 134 and active antenna 136. These internal components can be used by the crop assessment system 100 to control its operation and to output information to external devices / signal statuses to a user.

[0068] The real-time controller 116 may be configured to control the operation of the light emitters 102 and the imaging sensors 104. For example, the real-time controller 116 may activate or deactivate components based on present system needs. The real-time controller 116 may further be configured to modify the intensity and / or timing of the light emitters 102 based on current lighting or other considerations.

[0069] The main processor 117 may be configured to interact with and / or control other components of the crop assessment system 100. For example, the data from one or multiple imaging sensors 104 can be transferred to an image processor within the main processor 117, which, using neural networks or other computer vision techniques can extract features from the images. Based on inferred properties, the processor 117 can send control signals to a real-time control unit 116, which can precisely control the timing of the imaging sensors 104, light emitters 102, and other periphery such as external agricultural equipment, robotic systems, processinglines etc. Signals can be locked to time signals received from positioning systems, and additional sensor data might be used as inputs (e.g. accelerometer or position data).

[0070] In some embodiments, the real-time controller 116 and the main processor 117 can be implemented in the same component rather than being separated as illustrated in the Figures.

[0071] The buttons, LEDs, and display 118 (which may include, but may not be limited to, display 118a and button 118b) may form a part of the visual indicators and antenna 110. For example, the buttons, LEDs, and display 118 may serve as indicators that can signal the system status, such as “normal operation” or “recording” or enable the user to input things directly into the crop assessment system 110.

[0072] The USB-C port 120 may provide a means to power the crop assessment system 100. The USB-C port 120 may provide a means to upload / download data onto / off of the crop assessment system 100. The USB-C port 120 may provide a port to interoperate with further devices (e.g., a smartphone). The USB-C port 120 may be another type of port which may provide one or more of the foregoing functions.

[0073] The GB Ethernet 122 may provide the main processor 117 with the ability to transmit Ethernet frames, for example, at a rate of a gigabit per second. Other possible data link layer protocol data units are also usable.

[0074] The non-volatile memory express storage 124 can store instructions for carrying out one of more operations of the crop assessment system 100. The non-volatile memory express storage 124 may also be used to store information generated during operation of the system 100, for example, prior to download onto an external device. The non-volatile memory express storage 124 may alternatively be a different type of storage device.

[0075] The WiFi / BT 126 and antenna 128 may be configured to enable the crop assessment system 100 to use WiFi and / or Bluetooth. The WiFi / BT 126 and antenna 128 may be configured to transmit information to or from an external device. For example, the WiFi / BT 126 and antenna 128 may be configured to receive control instructions from, for example, a user’s smartphone. As another example, the WiFi / BT 126 and antenna 128 may be configured to transmit data to an external computing device to offer the user a real-time view of the data generated thus far during an assessment. The crop assessment system 100 may use other short-distance wireless communication modalities.

[0076] The LTE modem 130 and antenna 132 can provide the crop assessment system 100 with long-term evolution wireless broadband communication. This may enable the crop assessment system 100 to transmit data over longer distances (e.g., to an external device in a remote location). The crop assessment system 100 may use other long-distance wireless communication modalities.

[0077] The I MU 138 may provide the crop assessment system 100 with an inertial measurement unit. The IMU 138 may provide the crop assessment system 100 with information regarding its acceleration and orientation. Such information may be helpful to orient the images captured by the imaging sensor 104. The information may further assist the crop assessment system 100 to map crop metrics indicative of yield onto the appropriate plants or in the appropriate region in the crop on a readout.

[0078] The GNSS receiver 134 and active antenna 136 may be used by the positioning system to determine the location of the crop assessment system 100. This can be used by the crop assessment system 100 to correspond metrics indicative of crop yield to specific regions in the crop or to particular plants in the crop. This may be useful to provide particularized information about the crop. Other positioning system modalities are also possible.

[0079] In operation, the real-time controller 116 may control the light emitters 102 to emit active lighting (e.g., IR lighting) to illuminate the crops. The real-time controller 116 may image the crops using the imaging sensors 104. These images may be passed to the main processor 117 for analysis. The crop assessment system 100 may also track its location using the GNSS receiver 134 and active antenna 136 and may track the orientation of the imaging sensors 104 using the IMU 138. The images may be processed by the main processor 117 to identify plants in the image (for example, the system may make use of machine learning models to detect the trunks of trees to identify plants) and use the positional data from the GNSS receiver 134 and active antenna 136 and / or IMU 138 to associate the plant with a position. The main processor 117 may further process the images (for example using a model trained using machine learning) to determine a metric indicative of crop yield. Such metrics may optionally include, for example, number of crop units (e.g., number of fruits), size of crop units (e.g., size of fruits), crop load (e.g., number of fruits per tree), the crop surface area (e.g., canopy leaf area), the crop diameter (e.g. fruit diameter), the crop density (e.g., blossom cluster density), and / or any diseases or other abnormalities on the crop units. These metrics may be associated with the position data of the plant. For example, the metrics may be superimposed or otherwise mapped onto an outputto provide the user with an overview of the whole crop’s performance to date. In some embodiments, the crop assessment system 100 may further be configured to provide overview metrics for the whole crop based on the metrics collected for each of the plants (e.g., average number of crop units per plant, average size of crop units, diseased or abnormal regions, etc.). By using active lighting in the IR ranges, the crop assessment unit 100 may be better able to determine crop yields as the IR may not be occluded by leaves or other plant debris that is not the crop unit (e.g., the fruit). Further, as described above, IR (or near-IR) light absorption and emission of plant tissue can be linked to its photosynthetic activity and may be helpful to determine plant health-related criteria (e.g., tissue damage, photosynthetic activity, pest, disease, nutrient deficiency) and may help distinguish between objects that are hard to distinguish using just the visible range.

[0080] Once the metrics are computed, then the crop assessment system 100 may be configured to transmit the metrics to an external device or to the user. For example, the crop assessment system 100 may transmit the results to a user via the buttons, LEDs, and display 118. Alternatively, the crop assessment system 100 may store the information on the nonvolatile memory express storage 124 for later download using the LISB-C 120. Alternately, the crop assessment system 100 may transmit the information using the WiFi / BT 126 and antenna 128, the LTE modem 130 and antenna 132, and / or with GB Ethernet 122.

[0081] FIG. 4 is block schematic diagram of the interfaces of the crop assessment 100, according to some embodiments.

[0082] The data coming out of the crop assessment system 100 can be transferred to cloud infrastructure for further processing, visualization, aggregation, etc. The crop assessment system 100 may further be updated by this infrastructure as well.

[0083] For data, the information may be downloaded from the crop assessment system 100 via the data upload 406. This may be via any of the data transfer methods described above of via other data transfer methods. The data upload 406 may transmit the data to a data warehouse 416. The data warehouse 416 may transmit the data to an analytics backend 414 to conduct analytics on the data. Analysis of the data may be carried out, for example, in the data warehouse 416 and / or the analytics backend 414. Analysis of the data could include monitoring of trends (e.g., incorporating historic data), growth curves / rates, yield estimation, data aggregation across multiple locations, conversion of the data into action plans (e.g. for farmstaff) or input prescriptions (e.g., for sprayer control). The analytics backend 414 may push the data (and its analysis) to an analytics dashboard 412 for review by a user.

[0084] In some embodiments, the data in the data warehouse 416 may be fed into machine learning models 418. These machine learning models 418 may use the obtained data to further refine the models. In particular, if the data include error reports by users, these errors may be used for further training of any machine learning models used to identify the plants and / or the crop units (e.g., identify the trunks of trees and fruit growing thereon). These refined models may be transferred into the model management 420. The model management 420 may push new models to the crop assessment system 100 via model deployment 410. Model development may be particularly useful when an initial model was trained using limited training data or the model is being adapted to a new crop or spectral band used in active lighting.

[0085] In some embodiments, the crop assessment system 100 may further communicate with a mobile client / app 402 located, for example, on a user’s mobile device. This mobile client / app 402 may be configured using an app backend 404. In some embodiments, the data warehouse 416 may provide data to the mobile client / app 402 via the app backend 404. Such configurations may enable to user to see the progress of the crop assessment system 100 on their mobile device.

[0086] FIG. 5A shows an image captured using a sensor sensitive to visible light (the RGB image) (502) next to an image captured using a sensor sensitive to near-IR light (the near-IR image) (504), both captured under difficult lighting conditions (facing the sun) using the crop assessment system 100, according to some embodiments.

[0087] While some detail is missing in the RGB image (502), e.g. due to overexposure, the near-IR image (504) still shows a clear depiction of the object.

[0088] FIG. 5B shows tissue damage in a near-IR / IR image using the crop assessment system 100, according to some embodiments.

[0089] As seen in FIG. 5B, tissue damage can potentially be more easily be identified in as dark regions 506 in near-IR / IR images.

[0090] FIG. 6 illustrates different information outputs provided by the crop assessment system 100, according to some embodiments.

[0091] As described above, the crop assessment system 100 may be configured to generate metrics indicative crop yields associated with specific plants and / or locations in the crops. As such, the crop assessment system 100 may be configured to produce, for example, 2D maps of the crops with the metrics placed thereon. In some embodiments, the metrics may be illustrated by means of colour, shade, or other visually identifiable feature (see, e.g., image 602). For example, the crop unit count may be represented on a scale between two colours (or shades of a colour) wherein purely the first colour (or shade) means no or few crop units while purely the second colour (or shade) means the a maximum or threshold crop unit count. In such configurations, the colour of the location of the plant may indicate the count for that plant thereby providing a intuitively digestible output for a user (e.g., regions of poor or good production will all illuminate with the same colour or shade; see for example images 602 and 604). In some embodiments, the metrics may be illustrated by means of size of a shape such as a circle (see, e.g., image 604). In some embodiments, the division may be associated with a unit of ground rather than individual plants (to control for densely packed plants). In some embodiments, the user may hover over specific locations to retrieve a precise count (see, for example image 602).

[0092] FIG. 7 illustrates a dashboard output 700 provided by the crop assessment system 100, according to some embodiments.

[0093] The dashboard output 700 may include a control panel 701, global statistics 702, a crop map 704, plant distribution of crop unit count 706, and a crop unit volume distribution 708. The dashboard 700 may further provide more information based on the information pulled using the crop assessment system 100. For example, the dashboard output 700 may compare information pulled from different crops to provide comparative insights.

[0094] The control panel 701 may enable the user to select which crop they want to view and at what time. For example, the user may be able to select a particular crop (e.g., fruit) season, month, scan date, farm name, farm block, section name, variety, stage name, and specific parameters (e.g., size units).

[0095] The global statistics 702 may provide statistics by type of crop unit (e.g., type of fruit or subtype of fruit (e.g., variety of apple)). The statistics provided may include the average crop unit per plant, the rows scanned, the plants scanned, the crop units counted, and the average crop unit size (e.g., volume or diameter).

[0096] The crop map 704 may be a 2D geographic output which maps the metrics predictive of crop yield onto a map of the crop to give geographic insights. The crop map 704 may further color code (or otherwise visually indicate) different types of crops. Such displays are described above with reference to FIG. 6.

[0097] The plant distribution of crop unit count 706 may provide a information relating to the distribution of crop unit counts over plants. This may be helpful to establish a visual means of comparing different crops to each other (e.g., using the shape of the graph to ascertain further insights about different crops).

[0098] The crop unit volume distribution 708 may provide information relating to the distribution of crop unit volumes or sizes (or any spatial measurement of the crop unit). This may be helpful to establish a visual means of comparing different crops to each other (e.g., using the shape of the graph to ascertain further insights about different crops).

[0099] Other possible statistics which the crop assessment system 100 may be configured to provide include one or more of canopy volume / density, trunk or branch cross-sectional area, (fruit) color, quality grading, ripeness, glucose content (brix), dry matter, water content, disease statistics, pest statistics, nutrient deficiencies.

[0100] In some embodiments, it may further be possible to select subset regions of the crop map and obtain the above statistical analyses for the subregion. Such implementations may be helpful when comparing different subregions of the crops to one another (e.g., to determine whether a region may be diseased, nutrient deficient, etc.).

[0101] FIG. 8 is a process diagram of a method of assessing crops 800, according to some embodiments.

[0102] According to a further aspect, there is provided a method of assessing crops 800. The method 800 includes, for each given plant of a plurality of plants, assigning a unique identifier to the given plant (block 802), emitting light toward the given plant using at least one light emitter (block 804), capturing an image using at least one imaging sensor (block 806), detecting the given plant and a predictor of a crop output of the plant in the image (block 808), estimating a location of the given plant based on a signal from a geo-positioning subsystem (block 810), and generating at least one crop metric predictive of yield of the given plant based on the crop output (block 812), and generating a crop output data structure with data defining the at leastone crop metric and the location, in association with the unique identifier, for each of the plurality of plants (block 814).

[0103] In some embodiments, the method 800 may update crop outputs (block 814) as each plant is measured. In some embodiments, the method 800 may wait until all (or some subset of) plants are analyzed before generating the crop output (block 814).

[0104] FIG. 9 is a schematic diagram of computing device 900, according to some embodiments.

[0105] The technical solution of embodiments may be in the form of a software product. The software product may be stored in a non-volatile or non-transitory storage medium, which can be a compact disk read-only memory (CD-ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the methods provided by the embodiments.

[0106] The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and particularly configured computer hardware arrangements. The embodiments described herein are directed to electronic machines and methods implemented by electronic machines adapted for processing and transforming electromagnetic signals which represent various types of information. The embodiments described herein pervasively and integrally relate to machines, and their uses; and the embodiments described herein have no meaning or practical applicability outside their use with computer hardware, machines, and various hardware components. Substituting the physical hardware particularly configured to implement various acts for non-physical hardware, using mental steps for example, may substantially affect the way the embodiments work. Such computer hardware limitations are clearly essential elements of the embodiments described herein, and they cannot be omitted or substituted for mental means without having a material effect on the operation and structure of the embodiments described herein. The computer hardware is essential to implement the various embodiments described herein and is not merely used to perform steps expeditiously and in an efficient manner.

[0107] For simplicity only one computing device 900 is shown but the crop assessment system 100 (or greater system) may include more computing devices 900 operable by users toaccess remote network resources and exchange data. The computing devices 900 may be the same or different types of devices. The computing device 900 at least one processor, a data storage device (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface. The computing device components may be connected in various ways including directly coupled, indirectly coupled via a network, and distributed over a wide geographic area and connected via a network (which may be referred to as “cloud computing”).

[0108] For example, and without limitation, the computing device may be a server, network appliance, set-top box, embedded device, computer expansion module, personal computer, laptop, personal data assistant, cellular telephone, smartphone device, LIMPC tablets, video display terminal, gaming console, electronic reading device, and wireless hypermedia device or any other computing device capable of being configured to carry out the methods described herein.

[0109] As depicted, computing device 900 includes at least one processor 902, memory 904, at least one I / O interface 906, and at least one network interface 908.

[0110] Each processor 902 may be, for example, any type of general-purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an integrated circuit, a field programmable gate array (FPGA), a reconfigurable processor, a programmable read-only memory (PROM), or any combination thereof.

[0111] Memory 904 may include a suitable combination of any type of computer memory that is located either internally or externally such as, for example, random-access memory (RAM), read-only memory (ROM), compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, erasable programmable read-only memory (EPROM), and electrically-erasable programmable read-only memory (EEPROM), Ferroelectric RAM (FRAM) or the like.

[0112] Each I / O interface 906 enables computing device 900 to interconnect with one or more input devices, such as a keyboard, mouse, camera, touch screen and a microphone, or with one or more output devices such as a display screen and a speaker.

[0113] Each network interface 908 enables computing device 900 to communicate with other components, to exchange data with other components, to access and connect to networkresources, to serve applications, and perform other computing applications by connecting to a network (or multiple networks) capable of carrying data including the Internet, Ethernet, plain old telephone service (POTS) line, public switch telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g. Wi-Fi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network, and others, including any combination of these.

[0114] Computing device 900 is operable to register and authenticate users (using a login, unique identifier, and password for example) prior to providing access to applications, a local network, network resources, other networks and network security devices. Computing devices 900 may serve one user or multiple users.

[0115] Applicant notes that the described embodiments and examples are illustrative and non-limiting. Practical implementation of the features may incorporate a combination of some or all of the aspects, and features described herein should not be taken as indications of future or existing product plans. Applicant partakes in both foundational and applied research, and in some cases, the features described are developed on an exploratory basis.

[0116] In the foregoing, numerous references were made to fruit and the trees that bare them. The systems and methods described herein could be used on other crops. For example, the systems described herein may make use of light when can reflect off of tubers underground and assess them using similar strategies.

[0117] In some embodiments, the systems and methods described herein can be used to analyze produce post-harvest, for example, in a warehouse, on a truck, in a processing plant, etc. In some embodiments, the systems and methods described herein can be used to analyze produce crop-growth, for example, as seeds or pre-seedings.

[0118] According to an aspect, there is provided a computer-implemented system for assessing crops 100. The system 100 including at least one light emitter 102, at least one imaging sensor 104, including an infrared imaging sensor 104 configured to detect infrared light, a geo-positioning subsystem, a processing subsystem 117 that includes one or more processors and one or more memories coupled with the one or more processors. The processing subsystem 117 configured to cause the system to emit light using the at least one light emitter 102, capture an image using the at least one imaging sensor 104, predict a crop output from the image, estimate a location of the image based on a signal from the geo-positioning subsystem, generate at least one crop metric predictive of yield based on the crop output, and generate a crop output data structure with data defining the at least one crop metric and the location.

[0119] Such systems may be configured to survey crops for example to assess seeds, preseedlings, or seedlings. Such systems may be configured to be mounted onto vehicles that survey warehouses and / or processing plants and may provide, for example, assessment of a harvest once in the warehouse or productivity of a processing plant. In some embodiments, such systems may be stationary and assess crops as they pass by (e.g., counting crops as they are loaded onto a vehicle).

[0120] The term “connected” or "coupled to" may include both direct coupling (in which two elements that are coupled to each other contact each other) and indirect coupling (in which at least one additional element is located between the two elements).

[0121] Although the embodiments have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the scope. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification.

[0122] Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed, that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0123] As can be understood, the examples described above and illustrated are intended to be exemplary only.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented system for assessing crops, the system comprising: at least one light emitter; at least one imaging sensor, including an infrared imaging sensor configured to detect infrared light; a geo-positioning subsystem; a processing subsystem that includes one or more processors and one or more memories coupled with the one or more processors, the processing subsystem configured to cause the system to: for each given plant of a plurality of plants: assign a unique identifier to the given plant; emit light toward the given plant using the at least one light emitter; capture an image using the at least one imaging sensor; detect the given plant and a predictor of a crop output of the plant in the image; estimate a location of the given plant based on a signal from the geopositioning subsystem; and generate at least one crop metric predictive of yield of the given plant based on the crop output; and generate a crop output data structure with data defining the at least one crop metric and the location, in association with the unique identifier, for each of the plurality of plants.

2. The computer-implemented system of claim 1, wherein the at least one imaging sensor is configured to detect multispectral light.

3. The computer-implemented system of claim 1 , wherein the at least one light emitter emits infrared light.

4. The computer-implemented system of claim 1 , wherein the at least one light emitter includes an infrared light emitter that emits light with a wavelength in the range of 800 nm to 900 nm.

5. The computer-implemented system of claim 4, wherein the wavelength is approximately 850 nm.

6. The computer-implemented system of claim 1 , wherein the infrared imaging sensor detects light with a wavelength in the range of 800 nm to 900 nm.

7. The computer-implemented system of claim 6, wherein the wavelength is approximately 850 nm.

8. The computer-implemented system of claim 1, wherein the geo-positioning subsystem is configured to receive signals from a global navigation satellite system.

9. The computer-implemented system of claim 1 , wherein the at least one crop metric measures at least one of a crop unit number, a crop unit volume, a crop load, crop surface area, crop diameter, and crop density.

10. The computer-implemented system of claim 1, wherein the at least one crop metric comprises at least one of a disease metric, a pest metric, a blemish metric, a canopy density / volume metric, a nutrient deficiency metric, a colour grading metric, a quality grading, a ripeness grading, a trunk or branch cross-sectional area metric, a glucose (brix) metric, a dry matter metric, a water content metric.

11. The computer-implemented system of claim 1 , wherein the infrared light is selectively activatable based on a type of crop.

12. The computer-implemented system of claim 1 , wherein the at least one light emitter is selectively activatable based on the type of crop.

13. The computer-implemented system of claim 1 , wherein the at least one light emitter is disposed above and / or below the imaging sensor.

14. A method of assessing crops, the method comprising:for each given plant of a plurality of plants: assigning a unique identifier to the given plant; emitting light toward the given plant using at least one light emitter; capturing an image using at least one imaging sensor; detecting the given plant and a predictor of a crop output of the plant in the image; estimating a location of the given plant based on a signal from a geo-positioning subsystem; and generating at least one crop metric predictive of yield of the given plant based on the crop output; and generating a crop output data structure with data defining the at least one crop metric and the location, in association with the unique identifier, for each of the plurality of plants.

15. The method of claim 14, wherein the at least one imaging sensor is configured to detect multispectral light.

16. The method of claim 14, wherein the at least one light emitter emits infrared light.

17. The method of claim 14, wherein the at least one light emitter includes an infrared light emitter that emits light with a wavelength in the range of 800 nm to 900 nm.

18. The method of claim 17, wherein the wavelength is approximately 850 nm.

19. The method of claim 14, wherein the infrared imaging sensor detects light with a wavelength in the range of 800 nm to 900 nm, and more preferably.

20. The method of claim 19, wherein the wavelength is approximately 850 nm.

21. The method of claim 14, wherein the geo-positioning subsystem is configured to receive signals from a global navigation satellite system.

22. The method of claim 14, wherein the at least one crop metric measures at least one of a crop unit number, a crop unit volume, and a crop load.

23. The method of claim 14, wherein the at least one crop metric comprises at least one of a disease metric, a pest metric, a blemish metric, a canopy density / volume metric, a nutrient deficiency metric, a colour grading metric, a quality grading, a ripeness grading, a trunk or branch cross-sectional area metric, a glucose (brix) metric, a dry matter metric, a water content metric.

24. The method of claim 14, further comprising selectively activating the infrared light based on a type of crop.

25. The method of claim 14, further comprising selectively activating the at least one light emitter based on the type of crop.

26. The method of claim 14, wherein the at least one light emitter is disposed above and / or below the imaging sensor.

27. A computer-implemented system for assessing crops, the system comprising: at least one light emitter; at least one imaging sensor, including an infrared imaging sensor configured to detect infrared light; a geo-positioning subsystem; a processing subsystem that includes one or more processors and one or more memories coupled with the one or more processors, the processing subsystem configured to cause the system to: emit light using the at least one light emitter; capture an image using the at least one imaging sensor; predict a crop output from the image; estimate a location of the image based on a signal from the geo-positioning subsystem;generate at least one crop metric predictive of yield based on the crop output; and generate a crop output data structure with data defining the at least one crop metric and the location.

Images