Monitoring plant material

Abstract

A mechanism for reducing light artifact noise in an image dataset containing a representation of plant material. A plurality of image datasets, each representing a monitoring region through which plant material passes, is obtained. The lowest pixel value for each pixel element of the plurality of image datasets is obtained and used to define a reference image dataset. An image dataset to be calibrated is processed with the reference image dataset to produce a calibrated image dataset.

Description

TITLEMONITORING PLANT MATERIALFIELD[OOO1] Embodiments of the present disclosure relate generally to monitoring plant material in an agricultural machine.BACKGROUND

[0002] There is an increasing use and reliance upon agricultural machines to perform agricultural tasks within an agricultural environment. One capability shared by many agricultural machines is the movement of plant material, e.g., during harvesting, collecting or cutting of plant material.

[0003] By way of example, a combine harvester is designed to cut crop material before separating the grain from any other material (also known as material-other-than-grain (MOG)). The grain is transported to a grain bin or other reservoir, and the MOG is expelled or dispersed out of the combine harvester. The dispersed MOG can be subsequently regathered (e.g., for use as fodder, compost or fertilizer) or be left where it lies (e.g., to function as fertilizer for future harvests).

[0004] As another example, forage harvester comprises a cutting mechanism configured to cut plant material from a field. The forage harvester may further include a feeding system to transport the cut plant material into a processing unit that chops or otherwise conditions the cut plant material. The forage harvester may also comprise a blower or conveyor system to transfer or move the processed plant material into a storage container or transport vehicle.

[0005] Thus, different forms of agricultural machine include mechanisms for moving or manipulating the position of plant material.

[0006] It is common to perform monitoring of plant material in an agricultural machine, for quality and / or quantity assessments. One approach to performing monitoring comprises capturing images of the plant material as it is moved through a monitoring region, i.e., capturingimages of plant material flow. For instance, in a combine harvester, a grain quality sensor may comprise an image sensing arrangement or camera positioned to capture images of grain as it transported to the grain bin or other reservoir. As another example, in a cotton harvester, a cotton quality sensor may be positioned to monitor picked cotton as it is conveyed to a storage element.

[0007] There is an ongoing desire to improve the accuracy and reliability of plant material monitoring in the agricultural machine.BRIEF SUMMARY

[0008] The invention is defined by the claims.

[0009] In accordance with a proposed approach, there is provided a plant monitoring system for monitoring plant material in a monitoring region of an agricultural machine.

[0010] The plant monitoring system comprises an image sensing arrangement configured to iteratively produce image datasets representing a region of interest in the monitoring region, wherein each image dataset comprises a respective pixel value for a plurality of pixel elements.

[0011] The plant monitoring system also comprises a processing system configured to, during movement of plant through the monitoring region: receive, from the image sensing arrangement, first image datasets of plant material produced by the image sensing arrangement as the plant material is moved through the monitoring region; for each of the plurality of pixel elements, identify the lowest pixel value for said pixel element amongst the first image datasets; define a reference image dataset using each identified lowest pixel value; receive, from the image sensing arrangement, one or more second image datasets of plant material moved through the monitoring region; and modify each second image dataset using the reference image dataset to produce one or more calibrated second image datasets.

[0012] The present disclosure proposes a plant monitoring system that generates image datasets representing a monitoring region through which plant material is moved (e.g., by a plant material transport system). First image datasets are processed to produce a referenceimage dataset. The reference image dataset is constructed from, for each pixel element of each first image datasets, a lowest pixel value. The reference image dataset is used to process and calibrate one or more (second) image datasets.

[0013] The present disclosure recognizes that lighting artifacts in an image sensing system will create noise in image datasets produced by the image monitoring system. These lighting artifacts may, for instance, result from reflections from a protective element positioned between the image sensor (producing the image datasets) and the monitoring region. More particularly, any such lighting artifact(s) will define a baseline value for any pixel value within an image dataset. By defining a reference image dataset using the lowest pixel values of a plurality of image datasets, it is highly probably that the reference image dataset will only represent the lighting artifacts, which can thereby be attenuated from the image dataset(s).

[0014] In some embodiments, the first image datasets comprise no fewer than 100 first image datasets. In particular, the first image datasets may comprise no fewer than 500 first image datasets and preferably no fewer than 1,000 first image datasets. By using a large number of first image datasets, there is an increased probability that at least one first image dataset will contain, for each pixel element, a pixel value representing background information rather than plant material. Thus, these embodiments improve a robustness and reliability of the proposed approach.

[0015] In some embodiments, the processing system is configured to modify each second image dataset by subtracting the reference image dataset from each second image dataset. The subtraction of the reference image dataset from each second image dataset provides an efficient mechanism for removal of static lighting artifacts. This process enhances the visibility of relevant plant characteristics, and reduces a risk of artifacts being misinterpreted or misprocessed as representing plant material.

[0016] In some embodiments, the image sensing arrangement comprises an image sensor for capturing raw image data from which each image dataset is derivable, and a protective element positioned between the image sensor and the monitoring region, wherein the protective element is at least partially optically transmissive.

[0017] The inclusion of a protective element safeguards the sensitive image sensor from potential damage caused by dust, debris, or plant material in the monitoring region. Moreover, it is herein recognized that the use of a protective element increases the risk of lighting artefacts, due to at least reflections from imperfections or scratches on the protective element impacting the ability of the image sensor. As such, the proposed approach is particularly advantageous in these circumstances.

[0018] In some embodiments, a side of the protective element facing the image sensor comprises an anti-reflective coating or surface. The anti-reflective coating or surface on the protective element reduces unwanted light reflections.

[0019] In some embodiments, a side of the protective element facing away from the image sensor is exposed to the monitoring region. The proposed approach is particularly advantageous in such examples, as the exposed side of the protective element will be subject to wear and tear by the plant material, which can introduce (more) imperfections to the protective element leading to a greater amount of lighting artifacts. The proposed approach provides a technique that is able to mitigate at least some of these lighting artifacts.

[0020] In some embodiments, each image dataset comprises a portion of raw image data captured by the image sensor.

[0021] In some embodiments, each image dataset comprises a digital image derived from raw image data captured by the image sensor.

[0022] In some embodiments, each image dataset represents the monitoring region at a different point of or period in time. By capturing image datasets at different time points or periods, there is an increased probability that at least one first image dataset will contain, for each pixel element, a pixel value representing background information rather than plant material.

[0023] In some embodiments, the plant monitoring system further comprises a light source configured to illuminate the monitoring region.

[0024] In some embodiments, when the system comprises a protective element, the protective element is positioned between the light source and the monitoring region.

[0025] In accordance with another proposed approach, there is provided an agricultural machine comprising the plant monitoring system as described above and a plant materialtransport device configured to transport plant material, wherein the plant material transport device comprises the monitoring region. The integration of the plant monitoring system into an agricultural machine with a plant material transport device enables continuous, in-situ monitoring of plant material during harvesting or processing.

[0026] In some embodiments, the agricultural machine is a combine harvester.Implementing the plant monitoring system in a combine harvester allows for immediate analysis of harvested crops, providing valuable data on crop quality, yield, and potential issues during the harvesting process. This real-time information can guide operational decisions and improve overall harvesting efficiency.

[0027] In some embodiments, the agricultural machine may comprise or be any harvesting machine, such as a forage harvester, mower, cotton harvester, fruit picker, root vegetable picker, or windrower. Other forms of agricultural machine(s) that move plant material and in which embodiments may be employed are also widely known, such as balers and so on.

[0028] In some embodiments, the plant material transport device is a grain conveyer system. Integrating the monitoring system with a grain conveyer system enables continuous assessment of grain quality and characteristics as it moves through the harvesting equipment. This setup can help detect variations in grain properties, foreign objects, or other quality factors, allowing for prompt adjustments to harvesting parameters or sorting processes.

[0029] The plant material may be any form of plant material, such as a forage crop (e.g., grass / cereal crop such as corn) or a grain (e.g., wheat or similar). Of course, the specific type of plant material monitored by the plant monitoring system will depend upon the specific use case scenario for the plant monitoring system and / or the agricultural machine in which it is employed (where relevant).

[0030] In accordance with yet another proposed approach, there is provided a method for monitoring plant material in a monitoring region of an agricultural machine. The method comprises iteratively producing, using an image sensing arrangement, image datasets representing a region of interest in the monitoring region, wherein each image dataset comprises a respective pixel value for a plurality of pixel elements. During movement of plant material through the monitoring region, the method comprises: receiving, from the image sensingarrangement, first image datasets of plant material moved through the monitoring region; for each of the plurality of pixel elements, identifying the lowest pixel value for said pixel element amongst the first image datasets; defining a reference image dataset using each identified lowest pixel value; receiving, from the image sensing arrangement, one or more second image datasets of plant material moved through the monitoring region; and modifying each second image dataset using the reference image dataset to produce one or more calibrated second image datasets.BRIEF DESCRIPTION OF THE DRAWINGS

[0031] One or more embodiments of the invention / disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0032] FIG. 1 illustrates an agricultural machine;

[0033] FIG. 2 illustrates a proposed plant monitoring system;

[0034] FIG. 3 is a flowchart illustrating a proposed method;

[0035] FIG. 4 illustrates image datasets produced using the proposed method;

[0036] FIG. 5 illustrates a proposed plant monitoring system; and

[0037] FIG. 6 illustrates a processing system for use in a proposed plant monitoring system.DETAILED DESCRIPTION

[0038] The invention will be described with reference to the Figures.

[0039] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the Figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the Figures to indicate the same or similar parts.

[0040] This disclosure relates to a mechanism for reducing light artifact noise in an image dataset containing a representation of plant material. A plurality of image datasets, each representing a monitoring region through which plant material passes, is obtained. The lowest pixel value for each pixel element of the plurality of image datasets is obtained, and used to define a reference image dataset. An image dataset to be calibrated is processed with the reference image dataset to produce a calibrated image dataset.

[0041] FIG. 1 illustrates an agricultural machine 100 in which proposed embodiments may be employed, for the sake of improved contextual understanding.

[0042] The agricultural machine 100 is here embodied as a combine harvester, although embodiments are not restricted thereto.

[0043] The combine harvester 100 is coupled to a header 191 which is operable, in use, to cut and gather a strip of crop material as the combine harvester 100 is driven across a field / area to be harvested during a harvesting operation. A conveyor section 192 conveys the cut crop material from the header 192 into a crop processing apparatus 193 operable to separate grain and non-grain (i.e., material other than grain (MOG) or residue material (used interchangeably herein)) as will be appreciated. It is noted here that apparatus for separating grain and non-grain material are well-known in the art and the present invention is not limited in this sense. The skilled person will appreciate that numerous different configurations for the crop processing apparatus may be used as appropriate. Clean grain separated from the cut crop material is moved by a grain elevator into a grain bin, which may be periodically emptied, e.g., into a collection vehicle, storage container, etc. utilizing an unloading auger 195. The remaining non-grain material (MOG) / residue material is separately moved to a spreader tool 115 which is operable in use to eject the non-grain material or MOG from the rear of the combine harvester 100 and onto the ground. It will be appreciated that in some embodiments the combine harvester 100 may also include a chopper tool positioned, for example, between the crop processing apparatus 193 and the spreader tool 115 and operable, in use, to cut the residue material before it is spread by the spreader tool 115.

[0044] It will be appreciated that the agricultural machine 100 comprises one or more plant material transport device configured to transport plant material, e.g., between differentelements of the agricultural machine. The conveyer section 192 is one example of a plant material transport device, as it transports cut crop from the header to the crop processing apparatus. The grain elevator is another example of a plant material transport device, as it transports grain from the crop processing apparatus 192 to the grain bin. The unloading augur 195 is yet another example of a plant material transport device, as it transports grain from the grain bin to an external container.

[0045] It will be apparent that the plant material may comprise grain, harvested crop, tailings, MOG, straw, chaff, stems, leaves, pods, husks, cobs, or any other plant-derived material that may be processed or transported by an agricultural machine. Other suitable examples of plant materials (e.g., for use with different forms of agricultural machine) will be readily apparent to the skilled person, such as cotton, root crops, fruits, vegetables, berries, seeds and so on.

[0046] There is a desire to facilitate monitoring of plant material, e.g., whilst it is being transported. One example plant monitoring system comprises an image sensing arrangement (e.g., a camera) that captures one or more image datasets (e.g., image(s) and / or video(s)) of plant material. More particularly, the image sensing arrangement will capture image datasets representing a monitoring region through which plant material moves or is moved.

[0047] FIG. 2 provides an illustrative example of a plant monitoring system 200 in which proposed embodiments may be employed.

[0048] The plant monitoring system 200 comprises an image sensing arrangement 210 that captures one or more image datasets 215 of a monitoring region 220 (e.g., through which plant material 290 travels). The monitoring region 220 may, for instance, be a region of a plant material transport device 295 that moves or transports material (e.g., a conveyor or elevator as previously exemplified).

[0049] In particular, the image sensing arrangement 210 may comprise an image sensor211 for capturing raw image data from which each image dataset is derivable.

[0050] Each image dataset comprises a respective pixel value for a plurality of pixel elements. It will be appreciated that each pixel element (or each of a subset of pixel elements) may represent a different sub-region of the monitoring region 220, e.g., defining a pixel of theimage dataset. In general, a larger magnitude for a pixel value indicates a greater amount of light represented by the pixel element.

[0051] For instance, each image dataset may comprise a raw image dataset representing the raw data captured by an image sensor. Thus, each image dataset produced by the image sensing arrangement may comprise a portion of raw image data captured by the image sensor. One example of a portion of raw image data is a Bayer pattern image. In a Bayer pattern image, each pixel typically has only one color value (red, green, or blue) arranged in a specific mosaic pattern. Another example of a portion of raw image data is a raw image dataset in the form of a grayscale image. In a grayscale raw image dataset, each pixel element represents a different sub-region and typically has a single intensity value representing the brightness of that pixel, e.g., ranging from 0 (black) to 255 (white) for an 8-bit image. This type of raw image data may be captured by a monochrome image sensor.

[0052] As another example, each image dataset may comprise a processed image dataset representing a processed version of the raw data captured by the image sensor. Thus, each image dataset produced by the image sensing arrangement may comprise a digital image derived from raw image data captured by the image sensor. Such image datasets comprise a plurality of pixels, each representing a specific sub-region of the monitoring region, where each pixel has multiple pixel elements and therefore pixel values. By way of example, each image dataset may be an RGB image dataset, in which each pixel comprises three pixel elements providing separate red, green, and blue values. As another example, each image dataset may be a (e.g., synthesized) grayscale image data, in which each pixel value is produced by combining the pixel value(s) of a raw image dataset together.

[0053] Where each image dataset comprises a processed image dataset, the image sensing arrangement 220 may comprise an imaging processor 214 for processing raw image data generated by the image sensor 211 to produce the image datasets.

[0054] Other forms and structures for image datasets are well known to the skilled person. Each image dataset has a common structure of comprising a pixel value for each of a plurality of pixel elements. Different sets of one or more pixel elements may form or define a pixel representing a different sub-region of the monitoring region.

[0055] The image dataset(s) captured by the image sensing arrangement may be output from the image sensing arrangement 210 and further processed or used for one or more further tasks. In some examples, the image dataset(s) captured by the image sensing arrangement may be stored in a memory or storage unit 280, e.g., for later consultation and / or retrieval.

[0056] By way of example, the captured image(s) may be displayed at a user interface230 for an operator of the agricultural vehicle to monitor the plant material. The operator may, for instance, monitor the image dataset(s) to monitor a presence / absence of plant material to identify any blockages and / or monitor a color or other characteristic of the plant material (which may indicate a quality and / or harvesting success of the plant material).

[0057] Thus, in some examples, the plant monitoring system 200 comprises a user interface 230 configured to provide a visual representation of one or more image dataset(s) captured by the image sensing arrangement 210.

[0058] In some examples, the plant monitoring system 200 may comprise an analytical system 240 (e.g., a computer) that processes any captured image to monitor or analyze the plant material, i.e., to produce plant information 245 representing a predicted or measured property or characteristics of the plant material.

[0059] By way of example, the analytical system may use one or more image processing techniques such as computer vision algorithms, machine learning models, or other image processing methods to produce the plant information.

[0060] For instance, the analytical system may process the image(s) to perform one or more of the following: a quality assessment task (e.g., foreign object detection, damaged material detection, MOG detection and so on); quantity / yield estimation (e.g., counting or quantifying an amount of plant material); a color assessment task (e.g., to assess the health or harvest readiness of plant material); a disparity / uniformity estimation task; a material flow quantification task and so on. The plant information may comprise any one or more pieces of information produced by performing any one or more of these tasks. These example analytical tasks are merely exemplary, and the skilled person will readily appreciate how a wide variety of other image analysis tasks may be performed by an analytical system.

[0061] As a simple working example, an image of plant material carried by a grain elevator of the combine harvester may be processed to predict, as part of the plant information, an amount or rate of MOG in the grain elevator. This information is useful for determining a threshing success of the crop processing apparatus.

[0062] In some examples, the plant monitoring system 200 comprises a user interface230 configured to provide a visual representation of the plant information 245 produced by the analytical system 240. In some examples, the plant monitoring system 200 comprises a storage or memory configured to store the plant information 245 therein. In some examples, the plant monitoring system comprises one or more further processing units (e.g., to further process the plant information) and / or control unit (e.g., to control an operation of the agricultural machine responsive to the plant information).

[0063] The above example of an agricultural machine and plant monitoring system is merely exemplary, and the skilled person will appreciate that there are a wide variety of other agricultural machines that may employ a plant monitoring system for monitoring plant material.

[0064] The present disclosure recognizes it would be advantageous to reduce or minimize an amount of noise in any image dataset(s) produced by an image sensing arrangement. This reduces a risk of an operator or viewer of the image dataset(s) misinterpreting or missing potentially relevant elements, as well as improving an accuracy of any (automated) analysis of the image dataset(s).

[0065] In particular, the present disclosure recognizes that an image sensing arrangement used for a plant monitoring system will commonly and undesirably capture light artifacts, such as (near-)fixed or (near-)static reflections, that are present in every or a majority of the image datasets. Examples of light artifacts include reflections resulting from illumination of fully / partially reflective elements present in the monitoring region or between the image sensor and the monitoring region. Other examples of light artifacts include refraction or scattering from an element present in the imaged region or positioned between the image sensor and the monitoring region.

[0066] It is further recognized that light artifacts (e.g., reflections) are particularly prevalent in the context of a plant monitoring system 200, as it is common for an image sensingarrangement to comprise a protective element 212 that is positioned between the image sensor 211 and the monitoring region 220. The protective element 212 aims to protect the image sensor 211 from the plant material 290 and / or any dust / dirt carried alongside the plant material. However, this inevitably results in the protective element being subjected to significant wear and tear, e.g., causing dust deposits, imperfections and / or scratches in the side of the protective element exposed to the plant material. This causes further undesirable reflections / refraction / scattering of light (from the protective element) to the image sensor 211.

[0067] Moreover, it is common for a plant monitoring system 200 to also comprise a light source 260 configured to illuminate the monitoring region 220. Typically, this light source is placed in close proximity to the image sensor 211, e.g., forming part of the image sensing arrangement 210, to illuminate the monitoring region 220 effectively. However, this proximity is not essential.

[0068] If present, the protective element 212 may also cover the light source 260, to provide protection to the light source. Reflections of light emitted by the light source 260 and reflected by the protective element can provide a significant impact on the image dataset(s) captured by the image sensing arrangement. However, this is not essential, and in some examples, the light source may comprise its own dedicated protective element.

[0069] The present disclosure provides a technique for processing the image dataset(s) to mitigate or reduce the effect of lighting artifacts. In particular, it is proposed to compensate for such artifacts using a background estimation algorithm that estimates the light artifact part of the image dataset(s) by observing pixel values over time. As the image dataset produced from the desired portion of the monitoring region (e.g., a reflection from a grain kernel or other plant material) will be a contribution that is added on top of the (near- static) light artifact, statistically speaking the reflection will be the smallest value for a pixel over a sufficiently large set of image datasets. In particular, with a sufficiently large number of image datasets, it can be assumed (with a high likelihood or probability) that, for at least one image dataset, there will not be plant material present in the sub-region represented by a particular pixel. This allows for measurement of the specific value of the reflection for that specific pixel.

[0070] The identified value(s) can effectively define a reference image dataset, which can be subtracted from each image dataset and / or future image dataset to produce a respective set of one or more denoised image datasets.

[0071] Thus, the present disclosure proposes a plant monitoring system 200 comprises the image sensing arrangement 210 and a processing system 219 that processes image dataset(s) produced during movement of plant material through the monitoring region

[0072] FIG. 3 is a flowchart illustrating a proposed computer-implemented method 300.The method 300 may be performed by the plant monitoring system 200 (FIG. 2).

[0073] The method 300 comprises a step 310 of iteratively producing image datasets representing a region of interest in the monitoring region. Each image dataset comprises a respective pixel value for a plurality of pixel elements. Step 310 is performed by the image sensing arrangement 210 (FIG. 2).

[0074] In particular, step 310 may comprise at least a step 311 of producing first image datasets of plant material produced by the image sensing arrangement as the plant material is moved through the monitoring region.

[0075] In some examples, as later elucidated, step 310 may also comprise a separate step 312 of producing second image datasets of plant material produced by the image sensing arrangement as the plant material is moved through the monitoring region.

[0076] It will be appreciated that each image dataset may represent the monitoring region at a different point or period in time. Thus, as plant material is moving through the monitoring region during capture of each image dataset, each image dataset will represent a different portion or section of plant material (which portions or sections may overlap, depending upon the frame rate of the image sensing arrangement). In this way, each image dataset represents a different sample of the plant material.

[0077] The method 300 also comprises a step 320 of receiving, from the image sensing arrangement, first image datasets of plant material produced by the image sensing arrangement as the plant material is moved through the monitoring region. Thus, the first image datasets of plant material are produced by the image sensing arrangement during movement of the plant material through the monitoring region (e.g., by the plant material transport device).

[0078] Step 320 is performed by the processing system.

[0079] The method also comprises a step 330 of for each of the plurality of pixel elements, identifying the lowest pixel value for said pixel element amongst the first image datasets.

[0080] By identifying the lowest pixel value for each pixel element across multiple first image datasets, step 330 is effectively able to estimates or identify the reflected part of the first image dataset that is consistently present.

[0081] As previously explained, it can be assumed that over a sufficiently large number of image datasets, there will be at least one instance where no plant material is present in the sub-region represented by a particular pixel (element). In such cases, where no plant material is present, the pixel value will represent only the background value including any (near-)static reflection, which is expected to be the lowest value observed for that pixel element.

[0082] The method 300 also comprises a step 340 of defining a reference image dataset using each identified lowest pixel value. The reference image dataset effectively represents an estimate of the background reflections (and possibly other noise) present in the image datasets.

[0083] Step 340 can be trivially performed by creating a new image data set, comprising a same number of pixel elements as the plurality of pixel elements of the first image dataset(s), and setting the pixel value of each pixel element with the lowest identified value for said pixel element.

[0084] The method 300 also comprises a step 350 of receiving, from the image sensing arrangement, one or more second image datasets of plant material moved through the monitoring region.

[0085] The one or more second image dataset may comprise some or all of the first image datasets. In such circumstances, sub-step 312 may be omitted in some examples.

[0086] In some examples, the one or more second image dataset(s) may comprise one or more other image datasets captured / produced by the image sensing arrangement that do not form any of the first image datasets. In such circumstances, sub-step 311 is performed.

[0087] Of course, in some examples, the one or more second image datasets may comprise some or all of the first image datasets and one or more other image datasets (captured / produced by the image sensing arrangement).

[0088] The method 300 also comprises a step 360 of modifying each second image dataset using the reference image dataset to produce one or more calibrated second image datasets.

[0089] Step 360 may be performed, for instance, by subtracting the reference image dataset from each second image dataset. This subtraction process effectively removes the estimated background reflections (and possibly other noise) from each second image dataset. By doing so, the resulting (calibrated) second image datasets will have reduced interference from reflections and (possibly) other static artifacts.

[0090] The subtraction performed in step 360 can be performed on a pixel element by pixel element basis, where the pixel value of each pixel element in the reference image dataset is subtracted from the corresponding pixel value (of the corresponding pixel element) in each second image dataset.

[0091] Where each image dataset comprises a two-dimensional array or grid of pixel values (defining pixel elements), this operation may be represented mathematically as: r[x, y] = I[x,y] - ?[x,y] (1) where l*[x,y] represents the calibrated second image dataset, I [x,y] represents the (unmodified) second image dataset, R[x,y] represents the reference image dataset and [x,y] represents the co-ordinates of each pixel element.

[0092] In some uncontrolled circumstances, a subtraction performed in step 360 may result in one or more negative pixel values in the calibrated second image dataset(s). Accordingly, step 360 may be configured or designed to handle the possibility of any negative number(s) appropriately, e.g., by clipping a result of a subtraction to be no less than zero or another predetermined value. For instance, where a pixel value of a reference image dataset is greater than the corresponding pixel value of a second image dataset, the resultant corresponding pixelvalue for the calibrated second image dataset may take a value of zero or another predetermined value.

[0093] In some examples of step 360, the reference image dataset may be weighted(e.g., each pixel value multiplied by a predetermined scalar value) before being subtracted from each second image dataset.

[0094] By applying a scalar value to the reference image dataset, step 360 is able to tune the extent of (near-)static reflection removal. For instance, if the predetermined scalar value is less than 1, this would result in a more conservative removal of (near-)static reflections. This may be useful, for instance, in preserving detail in low-contrast areas. Conversely, a scalar value greater than 1 would lead to more aggressive removal of (near-)static reflections. This may be advantageous in circumstances in which the reflection(s) are particularly strong or persistent.

[0095] In some examples of step 360 the reference image dataset may be filtered, e.g., using one or more spatial filters, before being subtracted from each second image dataset.

[0096] By way of example, a Gaussian filter may be applied to the reference image dataset to smooth out high-frequency noise. As another example, a bilateral filer may be applied to the reference image dataset to perform smoothing whilst preserving edges in the reference image dataset, which may be advantageous to handle reflections with distinguishable boundaries (e.g., caused by imperfections in the protective element, if present).

[0097] Other approaches for performing step 360 will be apparent to the appropriately skilled person.

[0098] By way of example, another approach could involve using adaptive thresholding techniques. Instead of subtracting the entire reference image dataset, step 360 may comprise comprising compare each pixel value in the second image dataset to its corresponding pixel value in the reference image dataset. If the difference exceeds a certain threshold, only then would the subtraction be applied. This approach could help preserve details in areas where the reflection is less prominent.

[0099] Steps 320, 330, 340, 350 and 360 are performed by the processing system 219(FIG. 2). In some examples, steps 320, 330, 340, 350 and 350 are performed during movement of plant material through the monitoring region

[0100] The proposed approach makes use of first image datasets representing the monitoring region as plant material is moved through the monitoring region, e.g., as compared to any image dataset when no plant material is present.

[0101] This is advantageous as it allows for a more accurate representation of lighting artifacts that are present during use and / or in real-world conditions. For instance, the action of moving plant material through the monitoring region may introduce dynamic lighting effects, shadows, and reflections that may not be present during a static calibration. Such lighting effects may, for instance, result from the plant material transport device being operational.

[0102] Additionally, using image datasets captured during plant material movement allows for calibration and adaptation to the specific condition(s). For instance, as the agricultural machine (and specifically the plant material transport device) operates, factors such as dust accumulation, temperature changes, or mechanical vibrations may affect the optical properties of the plant monitoring system. As such, capturing image datasets during ongoing operation enables the plant monitoring system to continuously update its reference image dataset and maintain accuracy over time.

[0103] It is also noted that the proposed approach avoids the need for performing a separate, dedicated calibration step.

[0104] In some examples, the first image datasets (received in step 320 and processed in step 330) comprises no fewer than 100 first image datasets. Preferably, the first image datasets comprises no fewer than 500 first image datasets. More preferably, the first image datasets comprises no fewer than 1,000 first image datasets. A larger number of image datasets increases the likelihood of capturing instances where no plant material is present in the sub-region represented by a pixel element, allowing for a more accurate estimation of the background reflections.

[0105] The method 300 may be iteratively repeated to update the reference image dataset and produce one or more calibrated image dataset(s). For instance, the method may be repeated at a fixed, periodic basis (e.g., once a minute, once every 5 minutes). It is not expected that the fixed / static light artifacts will significantly change within a time period of less than 1minute, such that processing resource can be saved by only repeating the method 300 no more frequently than once a minute.

[0106] This iterative approach may be particularly advantageous because, as the plant monitoring system warms up, its components may experience thermal changes or other gradual shifts that affect the lighting artifacts captured in the image datasets.

[0107] For instance, the image sensor or protective element may undergo slight thermal expansion, potentially altering the optical path and resulting in subtle changes to reflections or other artifacts. By way of example, reflections may appear in different locations or with altered intensities, and the overall light distribution across the image sensor may change.

[0108] As another example, if present, one or more properties of light (e.g., the intensity of light) output by the light source will change as the light source warms up. This is particularly noticeable if the light source comprises an LED, whose properties will change upon warming up. This change in the property / ies of the light source will influence the near-static or fixed lighting artifacts captured by the image sensor.

[0109] By continuously updating the reference image dataset through the iterative approach, the plant material monitoring system is able to adapt to these thermal and mechanical changes, ensuring that the calibration process remains effective throughout the operation of the agricultural machine.

[0110] This iterative approach may comprise using a windowed set of first image datasets, e.g., comprising the most recently available set of X image datasets produced by the image processing system. X may be a predetermined number, e.g., no less than 100, no less than 500 or no less than 1,000.

[0111] In this approach, the method 300 may comprise maintaining a buffer or queue of the X most recent image datasets. As new image datasets are produced by the image sensing arrangement, they may be added to the buffer while the oldest datasets are removed, maintaining a constant window size of X datasets.

[0112] In this way, method 300 may be adapted to use the X most recent image datasets in the buffer as the first image datasets for steps 320 and 330. Similarly, in each iteration of method 300, steps 330 and 340 may be performed to define an updated reference imagedataset based on the windowed set of first image datasets. The updated reference image dataset may then be used in step 360 to modify subsequent second image datasets.

[0113] This window-based approach functions to configure the method 300 to adapt to gradual changes in the background reflections or other static artifacts over time. For example, if the protective element accumulates dust or wear over time, the reference image dataset is able to adapt to these changes, ensuring that the calibration of the second image dataset(s) remains effective.

[0114] In some implementations, the processing system may apply a weighting factor to the first image datasets within the window, e.g., giving more importance to more recent datasets in the estimation of the background reflections. For example, the weighting factor could be implemented as a decay function, where the weight decreases exponentially with the age of the dataset. This could be represented mathematically as:

[0115] In some examples, new reference image datasets (produced by an iteration of method 300) may be combined with one or more historic reference image datasets (produced by one or more previous iterations of method 300).

[0116] For instance, the new reference image dataset may be averaged with the one or more historic reference image datasets produced in one or more previous iterations of the method. In some cases, a weighted average may be used, giving more importance to more recent reference image datasets.

[0117] This combination of new and historic datasets may allow for more robust and adaptive reflection removal over time. In particular, the combined reference image dataset may account for gradual changes in reflections and possibly other static artifacts.

[0118] The method 300 may thereby comprise maintaining a buffer of recent reference image datasets and update this buffer as new datasets are produced.

[0119] FIG. 4 provides a visual representation of the process performed by the proposed approach.

[0120] In particular, FIG. 4 provides a first image 410, providing a visual representation of a second image dataset, a second image 420, providing a visual representation of a referenceimage dataset and a third image 430 providing a visual representation of a calibrated second image dataset.

[0121] More particularly, the third image 430 provides a visual representation of the second image dataset (visually represented by the first image 410) after calibration using the reference image dataset (visually represented by the second image 420).

[0122] Conceptually, the third image 430 is produced by subtracting the second image from the first image. It can be clearly identified that reflection artifacts (visible in the first image 310 and represented in the second image 320) have been attenuated or removed.

[0123] Turning back to FIG. 2, another approach to attenuating or reducing reflection(s) in the image dataset(s) is to configure a side of the protective element 212 facing the image sensor 211 to further comprise an anti-reflective coating 213 or surface.

[0124] An anti-reflective coating 213 or surface may comprise one or more: multi-layer interference coatings; nano-structured surfaces; and / or moth-eye structures. Suitable examples of anti-reflective coatings are provided by, inter alia, Raut, Hemant Kumar, et al. "Anti-reflective coatings: A critical, in-depth review." Energy & Environmental Science 4.10 (2011): 3779-3804.

[0125] Implementing an anti-reflective coating or surface on the protective element complements the software-based reflection removal technique described earlier, providing a comprehensive approach to mitigating reflections and improving the overall performance of the plant monitoring system.

[0126] Although possible, provision of an anti-reflective coating or surface on the side of the protective element facing the monitoring region (i.e., away from the image sensor) is less advantageous, as it is recognized that such a coating will be exposed to plant material and thereby, over time, rubbed or eroded away.

[0127] Thus, to reduce manufacturing costs and / or complexity, in some examples the protective element does not comprise an anti-reflective coating or surface on the side of the protective element facing the monitoring region (i.e., away from the image sensor).

[0128] More particularly, in some examples, the protective element does not comprise an anti-reflective coating or surface on the side of the protective element facing the monitoring region during at least a time period before the image sensing arrangement captures any imagedatasets of the monitoring region and / or (during manufacture) before the protective element is moved to cover the image sensor.

[0129] The skilled person would be readily capable of developing a plant monitoring system for carrying out any herein described method. Thus, each step of the flow chart may represent a different action performed by a plant monitoring system, and may be performed by a respective module of the plant monitoring system.

[0130] FIG. 5 illustrates a proposed plant monitoring system 500 comprising an image sensing arrangement 510 and a processing system 520 according to an embodiment. The plant monitoring system is configured to perform any herein disclosed method 300, for monitoring plant material in a monitoring region of an agricultural machine.

[0131] The image sensing arrangement 500 is configured to iteratively produce image datasets representing the monitoring region, wherein each image dataset comprises a respective pixel value for a plurality of pixel elements.

[0132] In the illustrated example, the image sensing arrangement 510 comprises an image sensor 511 (e.g., a camera) that captures raw image data. The raw image data may itself function as the image datasets. In some embodiments, the image sensing arrangement 500 further comprises an imaging processer 519 configured to process the raw image data to produce the image datasets, e.g., convert raw image data (e.g., Bayer pattern images) into image datasets (e.g., RGB or grayscale images).

[0133] The image sensing arrangement 510 may comprise any other feature or element described for the image sensing arrangement 210 (FIG. 2), such as the protective element 212. Although not illustrated, the plant monitoring system 500 may further comprise a light source configured to illuminate the monitoring region. If present, the protective element may (also) be positioned between the light source and the monitoring region.

[0134] The processing system 520 is configured to receive, from the image sensing arrangement, first image datasets of plant material produced by the image sensing arrangement as the plant material is moved through the monitoring region. More particularly, the processing system 520 may comprise an input interface 521 configured to receive the first image datasets.

[0135] The processing system 520 is further configured to for each of the plurality of pixel elements, identify the lowest pixel value for said pixel element amongst the first image datasets; and define a reference image dataset using each identified lowest pixel value. This may be carried out by a processing unit 522 of the processing system 520.

[0136] The processing system 520 is further configured to receive, from the image sensing arrangement 510, one or more second image datasets of plant material moved through the monitoring region. As previously explained, the second image datasets may comprise a subset or all of the first image dataset and / or one or more other image datasets. The input interface 521 may be configured to receive the second image dataset(s).

[0137] The processing system 520 is further configured to modify each second image dataset using the reference image dataset to produce one or more calibrated second image datasets. This may be carried out by a processing unit 522 of the processing system 520.

[0138] Example approaches that the processing unit 522 may use to perform its functions have been previously disclosed.

[0139] In some examples, the processing system 520 may be configured to output the calibrated second image dataset(s). Any output of the processing system may be controlled via an output interface 523. In particular, the output of the processing system may be defined by the processing unit 522 of the processing system via the processing unit.

[0140] The processing system 520 may, for instance, store the calibrated second image dataset(s) in a memory or storage unit 530 and / or control a user interface 540 to provide a user perceptible output of the calibrated second image dataset(s)and / or pass the calibrated second image dataset(s) to a further processing system 550 (of the plant monitoring system 500) for further processing.

[0141] FIG. 6 illustrates an embodiment of the processing system 520 described with reference to FIG. 5. The processing system is able to carry out or perform one or more embodiments of an invention, e.g., for producing the calibrated second image dataset(s).

[0142] The processing system 520 comprises an input interface 521 that receives communications from one or more inputting devices. Examples of suitable inputting devicesinclude external memories, sensors, user interfaces (such as mice, keyboards, microphones, and so on).

[0143] The processing system 520 also comprises a processing unit 522.

[0144] In one example, the processing unit 522 may comprise an appropriately programmed or configured single-purpose processing device. Examples may include appropriately programmed field-programmable gate arrays or complex programmable logic devices.

[0145] As another example, the processing unit may comprise a general purpose processing system (e.g., a general purpose processor or microprocessor) that executes a computer program 615 comprising code (e.g., instructions and / or software) carried by a memory 610 of the processing system 520.

[0146] The memory 610 may be formed from any suitable volatile or non-volatile computer storage element, e.g., FLASH memory, RAM, DRAM, SRAM, EPROM, PROM, CD-ROM and so on. Suitable memory architectures and types are well known to the person skilled in the art.

[0147] The computer program 615, e.g., the software, carried by the memory 610 may include comprise a sequence or set of instructions that are executable by the processing unit for implementing logical functions to carry out the desired method or procedure. Each instruction may represent a different logical function, step or sub-step used in performing a method or process according to an embodiment. The computer program may be formed from a set of subprograms, as would be known to the skilled person. The computer program 615 may be written in any suitable programming language that can be interpreted by the processing unit 522 for executing the instructions. Suitable programming languages are well known to the skilled person.

[0148] The processing system 520 may also comprise an output interface 523. The processing system may be configured to provide information, such as the calibrated second image dataset(s), via the output interface.

[0149] In some examples, the processing system may be configured to control one or more other devices connected to the output interface 523 by providing appropriate controlsignals to the one or more other devices. Suitable control examples include controlling a visual representation (e.g., of the calibrated second image dataset(s)) at a user interface.

[0150] Different components of the processing system 520 may interact or communicate with one another via one or more intra-system communication systems (not shown), which may include communication buses, wired interconnects, analogue electronics, wireless communication channels (e.g., the internet) and so on. Such intra-system communication systems would be well known to the skilled person.

[0151] It is not essential for the processing system 520 to be formed on a single device, e.g., a single computer. Rather, any of the system blocks (or parts of system blocks) of the illustrated processing system may be distributed across one or more computers.

[0152] There is also proposed an agricultural machine comprising the plant monitoring system and a plant material transport device configured to transport plant material, wherein the plant material transport device comprises the monitoring region.

[0153] The agricultural machine may be a combine harvester. In particular, the plant material transport device may be a grain conveyer system for the combine harvester, e.g., a grain elevator or unloading augur.

[0154] It will be understood that disclosed methods are preferably computer- implemented methods. As such, there is also proposed the concept of a computer program comprising code means for implementing any described method when said program is run on a processing system, such as a computer. Thus, different portions, lines or blocks of code of a computer program according to an embodiment may be executed by a processing system or computer to perform any herein described method.

[0155] A computer program may be stored on a computer-readable medium, itself an embodiment of the invention. A "computer-readable medium" is any suitable mechanism or format that can store a program for later processing by a processing system. The computer readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device. The computer- readable medium is preferably non-transitory.

[0156] In some alternative implementations, the functions noted in the block diagram(s) or flow chart(s) may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

[0157] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. If a computer program is discussed above, it may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. If the term "adapted to" is used in the claims or description, it is noted the term "adapted to" is intended to be equivalent to the term "configured to". If the term "arrangement" is used in the claims or description, it is noted the term "arrangement" is intended to be equivalent to the term "system", and vice versa. Any reference signs in the claims should not be construed as limiting the scope.

[0158] All references cited herein are incorporated herein in their entireties. If there is a conflict between definitions herein and in an incorporated reference, the definition herein shall control.

Claims

CLAIMSWhat is claimed is:

1. A plant monitoring system (200, 500) for monitoring plant material (290) in a monitoring region (220) of an agricultural machine (100), the plant monitoring system comprising: an image sensing arrangement (210, 510) configured to iteratively produce (310) image datasets (215) representing the monitoring region, wherein each image dataset comprises a respective pixel value for a plurality of pixel elements; and a processing system (219, 520) configured to: receive (320), from the image sensing arrangement, first image datasets of plant material produced by the image sensing arrangement as the plant material is moved through the monitoring region; for each of the plurality of pixel elements, identify (330) the lowest pixel value for said pixel element amongst the first image datasets; define (340) a reference image dataset (420) using each identified lowest pixel value; receive (350), from the image sensing arrangement, one or more second image datasets (410) of plant material moved through the monitoring region; and modify (360) each second image dataset using the reference image dataset to produce one or more calibrated second image datasets (430).

2. The plant monitoring system of claim 1, wherein the first image datasets comprises no fewer than 100 first image datasets and preferably no fewer than 1,000 first image datasets.

3. The plant monitoring system of claim 1 or 2, wherein the processing system is configured to modify each second image dataset by subtracting the reference image dataset from each second image dataset.

4. The plant monitoring system of any one of claims 1 to 3, wherein the image sensing arrangement comprises: an image sensor (211) for capturing raw image data from which each image dataset is derivable; and a protective element (212) positioned between the image sensor and the monitoring region, wherein the protective element is at least partially optically transmissive.

5. The plant monitoring system of claim 4, wherein a side of the protective element facing the image sensor comprises an anti-reflective coating (213) or surface.

6. The plant monitoring system of claim 4 or 5, wherein a side of the protective element facing away from the image sensor is exposed to the monitoring region.

7. The plant monitoring system of any one of claims 4 to 6, wherein each image dataset comprises a portion of raw image data captured by the image sensor.

8. The plant monitoring system of any one of claims 4 to 6, wherein each image dataset comprises a digital image derived from raw image data captured by the image sensor.

9. The plant monitoring system of any one of claims 1 to 8, wherein each image dataset represents the monitoring region at a different point or period in time.

10. The plant monitoring system of any one of claims 1 to 9, further comprising a light source (250) configured to illuminate the monitoring region.

11. The plant monitoring system of claim 10, when dependent upon claim 4, wherein the protective element is positioned between the light source and the monitoring region.

12. An agricultural machine comprising: the plant monitoring system of any one of claims 1 to 11; and a plant material transport device configured to transport plant material, wherein the plant material transport device comprises the monitoring region.

13. The agricultural machine of claim 12, wherein the agricultural machine is a combine harvester.

14. The agricultural machine of claim 13, wherein the plant material transport device is a grain conveyer system.

15. A method (300) for monitoring plant material (290) in a monitoring region (220) of an agricultural machine (100), the method comprising: iteratively producing (310), using an image sensing arrangement (210, 510), image datasets representing the monitoring region, wherein each image dataset comprises a respective pixel value for a plurality of pixel elements; andreceiving (320), from the image sensing arrangement, first image datasets of plant material produced by the image sensing arrangement as the plant material is moved through the monitoring region; for each of the plurality of pixel elements, identifying (330) the lowest pixel value for said pixel element amongst the first image datasets; defining (340) a reference image dataset (420) using each identified lowest pixel value; receiving (350), from the image sensing arrangement, one or more second image datasets (410) of plant material moved through the monitoring region; and modifying (360) each second image dataset using the reference image dataset to produce one or more calibrated second image datasets (430).

Images