Vegetation coverage estimation

Abstract

This disclosure relates to a method for estimating coverage of a plant in an image. The method comprises dividing the image into patches, applying a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of a first class or a second class, wherein the first class indicates presence of the plant, determining a sub-set of patches containing patches that are classified as the first class, determining a segmentation of each of the sub-set of patches, to determine areas of the sub-set of patches that contain the plant; and calculating the plant coverage of the plant in the image based on the segmentation.

Description

"Vegetation coverage estimation"Cross-Reference to Related Applications

[0001] The present application claims priority from Australian Provisional Patent Application No 2024900574 filed on 5 March 2024, Australian Provisional Patent Application No 2024903844 filed on 22 November 2024, the contents of which are incorporated herein by reference in their entirety.Technical Field

[0002] This disclosure relates to estimating coverage of a plant in an image.Background

[0003] Coverage estimation can be particularly useful in many applications including areas of ecology and agriculture. For example, coverage estimation such as a percentage cover metric is a key ecological parameter. In a more specific example, the coverage estimation of seagrass in the ocean can provide an estimation of the amount of “blue carbon” i.e., carbon dioxide that has been absorbed from the atmosphere and stored in the ocean. Disruption of such stores of blue carbon can cause the emission of carbon dioxide back into the atmosphere. So there is a need to understand where large amounts of blue carbon is stored, which can be achieved using coverage estimation of seagrass.

[0004] By capturing an image of vegetation, image processing techniques can be used to estimate the coverage of the vegetation in the image. In particular, machine learning techniques have emerged as a powerful imaging processing technique for coverage estimation. For example, to estimate the coverage of seagrass on the ocean floor, a machine learning model (such as a segmentation model) can be applied to an image of the ocean floor and will provide an output that is indicative of the coverage estimation. More particularly, the machine learning model may determine the areas of the imagethat are occupied by the seagrass and areas that are unoccupied by the seagrass to thereby provide an estimate of the seagrass coverage.

[0005] One common machine learning approach is to label each pixel into specific class so as to train a segmentation-based model. However, training such a segmentation model requires many thousands (if not more) images which have been manually segmented (i.e., annotated) by a human. This requires a human to go through many training images and manually label which parts of each image belong to a particular class. This is very labour intensive and expensive work.

[0006] Besides the labour intensive and expensive work to delineating vegetation from the background it is also complex and subject to human error. For example, vegetation often exhibits irregularities that can include small and intricate features that are difficult to distinguish, which further complicates the annotation process.

[0007] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each of the appended claims.

[0008] Throughout this specification the word “comprise”, or variations such as “comprises” or “comprising”, will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps.Summary

[0009] Disclosed herein are methods for estimating coverage of a plant using image processing techniques. More specifically, the methods disclosed herein uses machine learning techniques to estimate the plant coverage. The methods disclosed herein also address the difficulties in manually labelling each pixel in training image data.

[0010] According to the present disclosure, there is provided a method for estimating coverage of a plant in an image, the method comprising: dividing the image into patches; applying a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of a first class or a second class, wherein the first class indicates presence of the plant; determining a sub-set of patches containing patches that are classified as the first class; determining a segmentation of each of the sub-set of patches, to determine areas of the sub-set of patches that contain the plant; and calculating the plant coverage of the plant in the image based on the segmentation.

[0011] It is an advantage to divide the image into patches and apply a trained machine learning classifier to each of the multiple patches as this provides a way to determine the segmentation without having to label each pixel in the training data. Labelling each pixel into specific class to train a segmentation-based model is labour intensive and expensive work.

[0012] In some embodiments, the trained machine learning classifier is a multi-class classifier, wherein the first class and the second class are each one of multiple classes.

[0013] In some embodiments, the plant is of genus seagrass and plant coverage is for species of seagrass.

[0014] In some embodiments, the plant is a first species of seagrass and plant coverage is for the first species of seagrass, wherein the second class is a second species of seagrass different to the first species.

[0015] In some embodiments, the multiple class comprises one or more of: sand; algae;biotic; human-made; coral; and substrate.

[0016] In some embodiments, the multiple classes comprise a mixed class, wherein the sub-set of patches further contains patches that are classified as the mixed class.

[0017] In some embodiments, the trained machine learning classifier is trained using random point annotation.

[0018] In some embodiments, the trained machine learning classifier is trained using patch-based annotation.

[0019] In some embodiments, the trained machine learning classifier is based on ConvNext.

[0020] In some embodiments, determining the segmentation comprises applying a trained machine learning segmentation model to the patches classified with the first class.

[0021] In some embodiments, the trained machine learning segmentation model is a binary classifier, wherein applying the trained machine learning segmentation model to the patches classified with the first class comprises classifying each pixel in the respective patches into one of the first class or the second class.

[0022] In some embodiments, the binary classifier is trained on images of plant genus.

[0023] In some embodiments, the binary classifier is trained on images of a plant species.

[0024] In some embodiments, the trained machine learning segmentation model is based on OCRNet.

[0025] In some embodiments, the method comprises re-training the trained machine learning classifier for a current batch of images from a first environmental context and re-using the trained machine learning segmentation model from previous image batches from previous environmental contexts.

[0026] In some embodiments, determining the segmentation of each of the sub-set of patches comprising creating a binary patch for each of the sub-set of patches based on RGB thresholding.

[0027] According to the present disclosure, there is provided software that, when executed by a computer, causes the computer to perform the previously described method.

[0028] According to the present disclosure, there is provided a system for estimating coverage of a plant in an image, the system comprising: a processor configured to: divide the image into patches; apply a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of a first class or a second class, wherein the first class indicates presence of the plant; determine a sub-set of patches containing patches that are classified as the first class; determine a segmentation of each of the sub-set of patches, to determine areas of the sub-set of patches that contain the plant; and calculate the plant coverage of the plant in the image based on the segmentation.

[0029] Optional features provided in relation to the method, equally apply as optional features to the software and the system.Brief Description of Drawings

[0030] An example will be described with reference to the following drawings:

[0031] Fig. 1 illustrates a system for estimating coverage of a plant in an image.

[0032] Fig. 2 illustrates a method for estimating coverage of a plant in an image.

[0033] Fig. 3 shows an image with random point annotation for training of a machine learning classifier.

[0034] Fig. 4 illustrates the method of coverage estimation based on patch classification applied to an image.

[0035] Fig. 5 shows two sample images with 20-patch grid for annotation.

[0036] Fig. 6 shows the patch-based classification step overview as per Experimental example 1.

[0037] Fig. 7 shows sample images of the public dataset used in Experimental example 1.

[0038] Fig. 8a shows a sample image in the “background” class from the public database of images.

[0039] Fig. 8b shows a sample image in the “mixed” class from the public database of images.

[0040] Fig. 8c shows a sample image in the “seagrass” class from the public database of images.

[0041] Fig. 9 shows a confusion matrix for the seagrass classification in Experimental example 1.

[0042] Fig. 10 shows the results of the patch-based approach for seagrass coverage estimation for Experimental example 1.

[0043] Fig. 11 shows some example patches from the Fiji 2020 dataset.

[0044] Fig. 12 shows the results of the patch-based approach that has been finetuned on the Fiji 2020 dataset, for seagrass coverage estimation in Experimental example 1.

[0045] Fig. 13 shows further results of the patch-based approach that has been finetuned on the Fiji 2020 dataset, for seagrass coverage estimation in Experimental example 1.

[0046] Fig. 14 shows the grid-based segmentation approach overview as per Experimental example 2.

[0047] Fig. 15 shows a selected sample of images of patch labels before and after mask generation with the Fiji 2022 dataset as per Experimental example 2.

[0048] Fig. 16 shows a visualisation of testing results for Experiment A and B on two selected images, with the corresponding coverage estimation.

[0049] Fig. 17 shows an original image, the “ground truth” segmentation, the patchbased approach to coverage estimation and the grid-based segmentation approach to coverage estimation for sample validation images.

[0050] Figure 18 illustrates an example underwater scene, camera image, 3D model and orthoimage.

[0051] Figure 19 illustrates a camera image where segmented seagrass is shown in solid black and unsegmented seagrass that is visible in the camera image but not yet segmented as seagrass is shown in white with a black outline.

[0052] Figure 20 illustrates an example method for seagrass segmentation.

[0053] Figure 21 illustrates a point- supervised segmentation framework.

[0054] Figure 22 illustrates 2D-3D mapping and orthomosaic projection.

[0055] Figure 23 is a diagram of a patch-based orthomosaic segmentation approach.Description of Embodiments

[0056] The methods disclosed herein are directed towards a patch level approach to estimating plant coverage in an image. More specifically, the methods disclosed herein are directed towards a hybrid approach to estimating plant coverage at pixel level in an image with patch-level annotation efforts. For instance, rather than simply determining a segmentation on an image directly, by applying a trained segmentation model to the image for example, the disclosed methods involve dividing the image that may contain the plant into patches and classifying in a class indicative of the plant being present. Some methods then involve determining a segmentation of the patches. More specifically, fine grained segmentation then determines the plant coverage in plant occupied patches.

[0057] The disclosed methods are applicable to different types of vegetation including, but not limited to, trees and plants. For example, the disclosed methods are applicable to, but not limited to, estimating coverage of redwood forests, coastal mangrove stands, sphagnum bogs, desert soil crusts, roadside weed patches, wheat fields, cultivated gardens and lawns. A hybrid approach to coverage estimation can be applied to vegetation coverage estimation as vegetation tends to grow in groups. As such, there is a high chance that patches of the original image will contain large segments of the vegetation, as well as patches that contain no vegetation. Therefore, the patches that contain no vegetation can be disregarded during the segmentation process. As a result, the amount of computer processing power and time is reduced.

[0058] As the amount of computer processing power and time is reduced as a result of disregarding some of the patches, this means that a pixel- wise segmentation approachcan be applied to the patches that contain the vegetation. This provides the ability to perform a finer segmentation that may account for more intricate parts or small parts of the vegetation (such as the leaves and small branches). Such fine segmentation is difficult to achieve for a whole image, due to the requirement of large amounts of manually annotated trained data. However, such fine segmentation is achievable using the disclosed methods as only some of the original image is segmented. For example, such fine segmentation is achievable using the disclosed methods as only binary segmentation is required with identified patch from the original image.

[0059] Moreover, although vegetation may have small objects (such as the leaves and small branches), small mistakes in the segmentation can be accounted for as they are averaged out in the disclosed methods, leading to an accurate coverage estimation despite the difficultly in segmenting these small features. The disclosed methods also provide an accurate coverage estimation for different density of vegetation, such as different density of grass patches. The disclosed methods are also scalable. In particular, this disclosure shows that there is scalable performance, which can achieve full-supervision performance.

[0060] The disclosed methods are particularly applicable to estimating coverage of seagrass species. A primary challenge for seagrass coverage estimation is the need for pixel-level multi-class annotation for large variations of seagrass, which requires heavy repetitive human manual process for model to be adapted. In particular, it is difficult to label seagrass leaves, especially considering the significant variations in appearance that seagrasses exhibit across different geographic locations. Traditionally, pixel level annotation has demanded extensive and repetitive manual effort, making it a labour- intensive and time-consuming process.

[0061] The disclosed methods address this issue by providing a hybrid model. The method disclosed herein takes advantage of both patch-level coarse annotation (which greatly reduces the amount of labour work required during manual annotation) and enables pixel level seagrass coverage estimation by further integrating a binary segmentation process (seagrass vs. background), which only requires small amount oftraining dataset without the need for multi-class annotation. Such a fine-grained segmentation can account for the leaves of the seagrass.

[0062] It is noted that, while this disclosure discusses coverage estimation of seagrass and presents experimental results for seagrass coverage estimation, the disclosed methods are not limited to seagrass coverage estimation. More specifically, while in some embodiments, the plant may be of seagrass genus or the plant may be a species of seagrass, the disclosed methods are not restricted to this limitation. The plant may be any type of vegetation. For example, the plant (for which a plant coverage is calculated) may be of genus (or species of) triticum (i.e., wheat) or may be of genus (or species of) quercus (i.e., oak).System

[0063] Fig. 1 illustrates an example system 100 for estimating coverage of plant 101 in an image. Fig. 1 is one example of a configuration of system 100. However, system 100 is not strictly limited to this configuration and this may be one possible embodiment of system 100. It is noted that system 100 of Fig. 1 is only meant to illustrate an example and a preferred system which is capable of performing the disclosed method.

[0064] The reference to “plant” in this disclosure may represent any or all vegetation present in the image. However, in some embodiments, “plant” may represent a particular type of plant or more specifically, a plant species. In some embodiments, plant 101 is a seagrass genus. For example, plant 101 may represent all species of seagrass. In other embodiments, plant 101 is a species of seagrass. More specifically, plant 101 represents a single species of seagrass. However, it is noted that plant 101 is not limited to a seagrass genus or species of seagrass.

[0065] System 100 comprises a device 102, which may be smartphone, computer, tablet, a server device, or any other similar device. Device 102 comprises processor 103. Device 102 comprises memory 104, which comprises non-volatile memory 105and / or volatile memory 106. Processor 103 may communicate with memory 104 by communicating with non-volatile memory 105, such as an optical disk drive, hard disk drive, solid-state drive, flash memory, storage server or cloud storage, or volatile memory 106, such as cache or RAM. Non-volatile memory 105 is a non-transitory computer readable medium, such as a magnetic hard drive, a solid-state disk or DVD or CD-ROM.

[0066] Software, that is, an executable program stored on non-volatile memory 105 causes processor 103 to perform methods for estimating coverage of a plant in an image. While the singular of “processor” is used herein, it is meant to also encompass multiple processors that are individually or together configured (e.g., programmed) to perform the methods disclosed herein. As such, processor 103 may refers to multiple computer processing units (CPUs) and / or graphical processing units (GPUs) that are configured to collectively perform the methods disclosed herein.

[0067] Once executed, the software may cause processor 103 to divide the image into patches, apply a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of a first class or a second class, determine a sub-set of patches containing patches that are classified as the first class, determine a segmentation of each of the sub-set of patches, to determine areas of the sub-set of patches that contain the plant and calculate the plant coverage of the plant in the image based on the segmentation.

[0068] Memory 104 may store the image and retrieve the image for later use. The image may be indicative of a two-dimensional image, which may be stored on memory 104 as Joint Photographic Experts Group (JPEG) format, RAW image format or a similar / equivalent image format, for example. In some examples, the image may be an RGB image, a multispectral image, a hyperspectral image or an infrared image.Similarly, after processor 103 divides the image into patches, the patches may be stored on memory 104 as Joint Photographic Experts Group (JPEG) format, RAW image format or a similar / equivalent image format.

[0069] The methods described herein may comprise applying one or more trained machine learning models to the image. These machine learning models may be stored on memory 104 by storing the weights that from the respective models, for example. Memory 104 may also store any output values calculated by processor 103 applying the one or more trained machine learning models, or any other variable or data necessary to perform such methods described herein.

[0070] Device 102 may further comprise camera 107 that is in communication with processor 103 via the input / output (I / O) port 108. In some embodiments, camera 107 is integrally contained in the device 102, despite being drawn as disjointed entities in Fig. 1. In other embodiments, camera 107 may be in communication with device 102 by way of a wired or wireless connection, such as through the Internet. Camera 107 captures an image of plant 101, which is then communicated to processor 103 via the I / O port 108. After capturing plant 101, the image may be stored on memory 104 or stored on a database or other storage means external to device 102.

[0071] Although VO port 108 is shown as single entity, it is to be understood that any kind of data port may be used to receive data, such as a network connection, a memory interface, a pin of the chip package of processor 103, or logical ports, such as IP sockets or parameters of functions stored on non-volatile memory 105 and executed by processor 103. The parameters of functions may be stored on memory 104 and may be handled by-value or by-reference, that is, as a pointer, in the source code.

[0072] Software may provide a user interface (such as a graphical user interface) presented to the user on device 102. The user interface is configured to accept input (via buttons or text fields etc.) from the user, via a touch screen or a device attached to device 102 such as a keyboard or computer mouse. These devices may also include a touchpad, an externally connected touchscreen, a joystick, a button, and a dial. In an example, device 102 may display several images which capture a plant. The user may select an image by interacting the touch screen or keyboard, which causes processor 103 to perform the method disclosed herein. The user interface may also display the estimated coverage as a numerical value after performing the method described herein.Method

[0073] Fig. 2 illustrates method 200 for estimating coverage of plant 101 in an image. Fig. 2 is to be understood as a blueprint for a software program and may be implemented step-by-step, such that each step in Fig. 2 is represented by a function in a programming language, such as, but not limited to, Python, C++ or Java. The resulting source code is then compiled and stored as computer-executable instructions on nonvolatile memory 105, which causes processor 103 (or multiple processors or a distributed computing architecture) to perform method 200.

[0074] Processor 103 first divides 201 the image into patches. For example, the image may be of size 256 x 256 pixels and processor 103 may divide 201 the images into 32 patches, each of size 8 x 8 pixels. Each patch may be considered as an image similar to the original image and may be referred to as a patch image. In some embodiments, before dividing 201 the image into patches, processor 103 may receive or retrieve the image from memory 104, camera 107 or another similar source, such as an external database.

[0075] Processor 103 then applies 202 a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of a first class or a second class. The first class indicates presence of the plant. For example, the first class may be indicative of “plant present” in the respective patch and the second class may be indicative of “plant absent”. The first and second class may be more specific. For example, the first class may be indicative of the presence of a plant genus and the second class may be indicative of the absence of the plant genus. In other examples, the first class may be indicative of the presence of a plant species and the second class may be indicative of the absence of the plant species. In further examples, the first class may be indicative of the presence of a first plant species and the second class may be indicative of a second plant species that is different to the first class. As such, the coverage may by based on the first plant species.

[0076] In some embodiments, the trained machine learning classifier is a binary classifier. Hence, the first class and the second class may be the only two classes. In other embodiments, the first class and the second class may each be one of multiple classes. In such an embodiment, the trained machine learning classifier may be a multiclass classifier. For example, the multiple classes may be seagrass species, sand, algae, biotic, human-made, coral and substrate. The first class may correspond to the multiple seagrass species classes and the second class may correspond to any of the other classes. Processor 103 then applies 202 a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of the classes. In other examples, the first class and the second class may correspond to more than one of the multiple classes e.g., the first class may correspond to different seagrass species.

[0077] In some embodiments, the multiple classes comprise a mixed class. The mixed class may be indicative of a class that partially contains the plant. As such, the mixed class may be indicative of a class that is between a “plant present” class and a “plant absent” class. As, at least, some of the plant is present in the mixed class, including the mixed class in the sub-set of patches may increase the accuracy of the plant coverage accuracy. However, as only part of the patch classified in the mixed class contains the plant, processor 103 may account for this when calculating the plant coverage e.g., using a factor.

[0078] Processor 103 then determines 203 a sub-set of patches containing patches that are classified as the first class. In other words, processor 103 determines 203 a sub-set of patches containing patches that contain the plant. In some embodiments, the sub-set of patches may further contain patches that are classified as the mixed class. Processor 103 then determines 204 a segmentation of each of the sub-set of patches, to determine areas of the sub-set of patches that contain the plant. As processor 103 only determines a segmentation for the patches classified as the first class (or the mixed class, in some embodiments), processor 103 does not need to process each of the patches of the original images. As such, this reduces the computational resources (processing power and memory requirements, for example) required to calculate the plant coverage.

[0079] Processor 103 may determine 204 the segmentation pixel-wise i.e., by analysing each pixel in the patches and to determine whether the pixel contains the plant. In some embodiments, processor 103 determines 204 a segmentation by applying a segmentation algorithm. For example, processor 103 may apply a binary segmentation algorithm which determines areas of the sub-set of patches that contain the plant and areas of the sub-set of patches that do not contain the plant.

[0080] In some embodiments, processor 103 determines 203 the segmentation of each of the sub-set of patches by creating a binary patch for each of the sub-set of patches based on RGB thresholding. For example, processor 103 may analysis each patch pixel- wise and determine whether each pixel value is above or below a threshold pixel value. If below, processor 103 changes the corresponding pixel to ‘black’ and if above, processor 103 changes the corresponding pixel to ‘white’, thereby obtaining a binary patch. The threshold value may be based on the particular plant being analysed. For example, the threshold value may be based on training images that a human annotator has manually annotated the plant. In other examples, processor 103 may apply other image processing techniques such as adaptive thresholding to determine the segmentation.

[0081] In some embodiments, processor 103 determines 203 the segmentation by applying a trained machine learning segmentation model to the patches classified with the first class. In some embodiments, the trained machine learning segmentation model is a binary classifier. As such, processor 103 applies the trained machine learning segmentation model to the patches classified with the first class comprises classifying each pixel in the respective patches into one of the first class or the second class. As a result, instead of classifying each pixel in the original image into one of the first class or the second class, classifying each pixel in the respective patches reduces the amount of computer processing power and time.

[0082] Finally, processor 103 calculates 205 the plant coverage of the plant in the image based on the segmentation. In other words, processor 103 calculates 205 the plant coverage of the plant in the image based on the determined areas of the sub-set ofpatches that contain the plant. For example, processor 103 may calculate 205 the plant coverage based on the number of pixels in the (original) image determined to contain the plant. The plant coverage may be calculated using a ratio based on the number of pixels determined to contain the plant.

[0083] As, in some embodiments, the plant is of genus seagrass, the binary classifier may be trained on images of the plant genus. More specifically, the trained machine learning segmentation model (i.e., the binary classifier) may have been trained to distinguish pixels in the patches that contain the plant genus and pixels that do not contain the plant genus. It is noted that the plant genus may contain all plant species. In other embodiments, the binary classifier may be trained on images of a plant species. More specifically, the trained machine learning segmentation model (i.e., the binary classifier) may have been trained to distinguish pixels in the patches that contain the plant species and pixels that do not contain the plant species. In other words, the trained machine learning segmentation model may be trained to distinguish different species of the plant from one another.

[0084] In some examples, plants grow differently, and as a result, image patches may be classified differently when classified by the trained machine learning classifier. So there is a need to be able to retrain the trained machine learning classifier to adapt to the change in the plant growth. However, the trained machine learning segmentation model does not necessarily need to be re-trained. As such, in some embodiments, processor 103 may re-train the trained machine learning classifier for a current batch of images from a first environmental context and re-use the trained machine learning segmentation model from previous image batches from previous environmental contexts.

[0085] As discussed above, processor 103 applies 202 a trained machine learning classifier to the image to each of the multiple patches and, in some embodiments, processor 103 applies a trained machine learning segmentation model to the patches classified with the first class. The machine learning model is understood to be a model, such as a mathematical model, that receives input and generates an output based on theinput. The machine learning model may be of an architecture, such as, but not limited to, a neural network, for example. In general, machine learning models are ‘trained’ to learn and recognise patterns in an input and provide an output that is a prediction based on the training it has undergone. Training involves updating weights or parameters (as referred to as hyperparameters) of the machine learning model, which define the machine learning model, to minimise a loss value, thereby creating a trained machine learning model (in other words, a machine learning model trained to generate an output). This may involve a gradient descent and backpropagation method.

[0086] In some embodiments, the trained machine learning classifier is trained using patch-based annotation. More specifically, processor 103 may train a machine learning classifier using training images generated by patch-based annotation to generate the trained machine learning classifier. In patch-based annotation, the training images may be divided into patches and a human annotator may annotate each patch in the training images into one of the first class or the second class. Annotating the patches, rather than annotating each pixel in the training images, reduces the labour of the human annotator.

[0087] In some embodiments, the trained machine learning classifier is trained using random point annotation. Fig. 3 shows an image with random point annotation. In random point annotation, random points in each training image are determined and a human annotator labels the patch around each point based on the dominant class in the patch area as one of the first class or the second class (or, in some embodiments, one of multiple classes). Processor 103 creates patches from each training image based on the random points. For example, processor 103 creates a patch from a respective training image that contains one random point at the centre of the patch. Annotating the random points, rather than annotating each pixel in the training images, reduces the labour of the human annotator. The random point annotation also provides a wide distribution of training patches, enabling the trained machine learning classifier to be more robust. In other embodiments, a human annotator may choose and label the points in the training images, rather than being a random point annotation.

[0088] The machine learning models recited in this disclosure may be stored on data memory 104 or on a server by storing the weights that define the model. As such, the machine learning model may be referred to as a “memory model”, given that it is defined by parameters (i.e., the weights) which can be stored on computer memory. In some embodiments, the machine learning model may be programmed on an integrated circuit, such as a field-programmable gate array (FPGA) or an NVIDIA processing unit. In such an embodiment, processor 103 may not retrieve the parameters from data memory 104. Instead, an input may be communicated from processor 103 to the integrated circuit and the integrated circuit may apply the machine learning model to the input and generate an output, which is then communicated to processor 103.

[0089] Integrated circuits, such as FPGAs, can be used where flexibility, speed, and parallel processing capabilities are desired. In such an embodiment, the integrated circuit may be part of device 102 of system 100 and may be considered as a “processor” or “processing unit”, similar to processor 103. Other implementations, such as application specific integrated circuits (ASIC) or neuromorphic architectures are equally useable.

[0090] There may be a number of different ways to invoke the machine learning model, which all fall within the meaning of “applying the training machine learning model” (e.g., applying the trained machine learning classifier or applying the trained machine learning segmentation model) as recited throughout this disclosure. For example, applying the model may involve calling an API routine to send the image to a server and the server then performs the calculations according to the trained machine learning model and returns the results. In other examples, applying may involve issuing a command to local hardware, such as a local chip, device, machine learning accelerator (e.g., a USB device design to efficiently perform machine learning tasks or NVIDIA’s Deep Learning Accelerator (DLA)), etc., that has the trained machine learning model stored thereon and provides a command interface to interact with the model. It is also possible to have a local copy of the machine learning model available so that the calculations are performed by the main processor of the local machine.Other local, remote or distributed implementations (such as cloud computing environments) are equally useable.

[0091] In some examples, the trained machine learning models described herein may be a neural network. In further examples, these trained machine learning models may be a neural network comprising one or more convolutional layers. As such, the trained machine learning models may perform the methods described herein by creating feature maps using convolutional filters. Such a machine learning model is known as a convolutional neural network (CNN). A CNN is ideal for applications involving images and the image as it accounts for the positioning and shape of objects captures in the image. However, other types of machine learning models are equally applicable here, such as K nearest neighbour, decision tree and support vector machines.

[0092] Mathematically, a convolution is an integration function that expresses the amount of overlap of one function g as it is shifted over another function f. Intuitively, a convolution acts as a blender that mixes one function with another to give reduced data space while preserving the information. In terms of neural networks and deep learning, convolutions are filter (matrix / vectors) with learnable parameters that are used to extract low-dimensional features from an input data. They have the property to preserve the spatial or positional relationships between input data points. CNNs exploit the spatially-local correlation by enforcing a local connectivity pattern between neurons of adjacent layers. In this example, the CNN architecture may comprise an optimal number of convolutional layers, filter size and stride length.

[0093] Intuitively, a convolution is the step of applying the concept of sliding window (a filter with learnable weights) over the input and producing a weighted sum (of weights and input) as the output. The weighted sum is the feature space which is used as the input for the next layers. More specifically, each convolutional layer comprises a filter, which calculates a weighted sum of pixel values. For example, a 2 x 2 filter comprises 4 weights, which are the coefficients of the filter. The filter starts at an initial position in the image structure, multiplies each pixel value in the image structure with the respective filter coefficient and adds the results. Finally, the filter stores theresulting number in an output pixel. In this sense, the output pixel values of each of the one or more convolutional layers comprises a weighted sum of input pixel values. The weights in the weighted sum correspond to the coefficients of the filter. Then, the filter moves by one pixel along one direction in the data structure and repeats the calculation for the next voxel of the output image. For example, if the stride is 2, the filter will move by two pixels along one direction. That direction may be an x-dimension or a y- dimension.

[0094] A single convolutional layer may use multiple filters, where each filter corresponds to a different feature. The corresponding feature of each filter is determined during training of the CNN. More specifically, each filter may contain weights, which are adjusted during a training process through a backpropagation process, for example. The filters are used to quantitatively determine the contribution of a particular feature on the output of the CNN. For a convolutional layer that occurs at the start of a CNN architecture, the filters of this convolutional filter may correspond to simple features. However, convolutional layers that occur later in the CNN architecture may exhibit more complex or abstract features. As an example, the complex or abstract features may be a combination of the simple features from the previous convolutional layer.

[0095] The CNN architecture may also comprise a batch normalization layer and rectifier linear unit (ReLU) activation function. Further, the output of the last convolution layer may be subjected to Global Average Pooling (GAP). Even further, a fully connected layer with Sigmoid activation may also be used on the output layer. However, other activation functions may be used. These activation functions include, but are not limited to, a binary step function, a tanh function, a ReLU function or a softmax function.

[0096] In some embodiments, the trained machine learning classifier and / or the trained machine learning segmentation model may be based on a transformer model. As such, these trained machine learning models may comprise encoder and / or decoder blocks. In essence, machine learning models based on transformers generatepredictions by ‘tokenising’ the input (such as an image). For example, an image or the image patches described herein may be further divided into smaller patches and each of these smaller patches may be converted into a token. These tokens can also incorporate other information such as the position of the smaller patch in the original image or the patch image and the information about the adjust smaller patches.

[0097] The process of creating these tokens is also referred to as ‘encoding’ or ‘semantic encoding’, as information regarding each smaller patch is essential encoded into a vector of numbers. The process of creating these token that incorporate the position of each smaller patch in the original image or respective patch image is referred to as “positioning embedding”. The encoding process also includes embedding at least one learnable parameter, which is determined through training. The opposite process then occurs where the tokens are converted from a vector of numbers into another numerical value that may be indicative of a classification class, which is referred to as ‘decoding’.

[0098] The machine learning model may also comprise an attention mechanism that applies weights to the tokens and may be characterized by its self-attention layers. The attention mechanism may also include a scaled dot-product between different matrices generated by the model to calculate the weights. In simple terms, the attention mechanism allows a model to focus on different parts of the input when generating each element of the output. This dynamic focusing capability results in a more contextually aware model, producing better results. Other ways of achieving attention within the model would be equally possible.

[0099] In some embodiments, the trained machine learning classifier is based on ConvNext (https: / / arxiv.org / abs / 2201.03545). In some embodiments, the trained machine learning segmentation model is based on OCRNet (https: / / arxiv.org / abs / 1909.11065).

[0100] There is presented another method for estimating coverage of a plant in an image. Rather than determining a segmentation of each of the sub-set of patches thatcontain patches classified as the first class, processor 103 may calculate the plant coverage in the image based directly on the sub-set of patches. This method may be considered as a coarse coverage estimation based on patch classification.

[0101] In this method, processor 103 divides the image into patches, applies a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of a first class or a second class, wherein the first class indicates presence of the plant; and determines a sub-set of patches containing patches that are classified as the first class. These steps are similar to steps 201, 202 and 203 of method 200. Processor 103 then calculates the plant coverage based on the sub-set of patches. For example, rather than determining a segmentation, processor 103 may calculate the plant coverage based on the number of patches in the sub-set of patches. In a further example, processor 103 may calculate the plant coverage using a ratio based on the number of patches in the sub-set of patches (e.g., number of patches in the sub-set / total number of patches in the image).

[0102] Fig. 4 shows the method of coverage estimation based on patch classification applied to an image. Fig. 4 represents an example when the trained machine learning classifier is a multi-class classifier, as the trained machine learning classifier has classified each patch into one of multiple classes (i.e., biotic, coral, human made etc). In this example, the coverage estimation is based on the patches that were classified in any of the “seagrass” classes, as well as the patches that were classified in the “mixed” class.

[0103] Use of the disclosed methods will now be described by way of experiments, as well as the performance evaluation of the disclosed methods and systems.

[0104] The goal in these experiments was to use the disclosed methods to detect and classify morphologic classes of seagrass and algae and provide a percentage coverage estimation for seagrass. A hybrid two-step machine learning based approach was investigated in these experiments: (1) patch-based classification model and (2) binary segmentation model, while the first step provides coarse patch-level coverageestimation, and the integration of a second binary segmentation as second step to provide fine-grained coverage estimation of seagrass.Image pre-processing and annotation

[0105] Due to various sensor resolutions used in the data collection process, data cleaning was performed to remove blur / un-useful images. These was followed by a preprocessing step to ensure images are uniformly labelled. The detailed pre-processing steps are:(1) Crop every image to the largest 4 (height):5 (width) aspect ratio rectangle;(2) Resize each image to 2048 (h=256 p*8)* 2560(w=256 p* 10); and(3) Segment the image into either 4 or 16 sections for annotation, with each one containing 4*5 grid patches.

[0106] Fig. 5 shows a sample image with 20 patches grid for annotation.Experimental example 1: patch-based classification approach

[0107] Fig. 6 shows the patch-based classification approach overview. As shown in Fig. 6, the main idea is to train a classifier that works with image patches and use the coverage of the patches which contain target class, seagrass, to aggregate into a coverage estimation for the overall image. In this experimental example, the trained machine learning classifier is based on ConvNext. More specifically, the backbone of the trained machine learning classifier is ConvNext. Further, the trained machine learning classifier is constructed entirely from standard ConvNext modules, which means it maintains the efficiency and desirable property for computer vision tasks, and yet compete favourably with transformers in terms of accuracy and scalability.Public Seagrass Dataset

[0108] The base model of this approach was developed using a public dataset of seagrass images. A total 3,066 images were used (resolution: 1920 x 1080) with data split at 80% for training (80%) and 20% for validation / test. Fig. 7 shows samplesimages of the public dataset used in this experimental example. Fig. 7a shows samples images of the public dataset used in Experimental example 1.

[0109] Although the public dataset provides full annotation (i.e., pixel-wise annotation), these annotations were converted into patch -based annotation for our base model development. Each image was split into 10 (columns) x 8 (rows) = 80 patches (resolution 192 x 135). As such, a total of 3,066 x 80 = 245,280 image patches were used for training and validation as shown in Table 1. Three classes were created using ground-truth masks: (1) background class: more than 75% background; (2) seagrass class: more than 75% seagrass; and (3) mixed class: otherwise.

[0110] Fig. 8 shows images from the public database belonging to the three classes mentioned above. Fig. 8a shows a sample image in the “background” class. Fig. 8b shows a sample image in the “mixed” class. Fig. 8c shows a sample image in the “seagrass” class.

[0111] The model’s classification performance was evaluated using testing dataset: testing time: 435 patches / sec, total 113s for 49,045 patches, as detailed below in Table2 and Fig. 9. Fig. 9 shows a confusion matrix for the seagrass classification.

[0112] Using this model, coverage of seagrass was estimated based on patches classified in the “seagrass” class. Coverage of seagrass was also estimated based on patches classified in the “mixed” class. More specifically, coverage was estimated using a ratio of the number of patches classified as “seagrass” plus the number of patches classified as “mixed” and the total number of patches, according to the following equation:Ns+ 0.4Nmcoverage = - — - where Nsis the number of patches in the “seagrass” class, Nmis the number of patches in the “mixed” class and N is the total number of patches.

[0113] The seagrass coverage estimation on sample test images is shown in Fig. 10. In other words, Fig. 10 shows the results of the patch-based approach for seagrass coverage estimation. Compared with “ground truth” (GT) coverage, it can be observed that the patch-based approach was able to accurately calculate the coverage of seagrass.Fiji 2022 Dataset

[0114] The base model described above was then finetuned on the dataset collected from a Fiji trip in 2022 (as referred to as the “Fiji 2020 datset”). A total of 15 full RAW images were annotated. After cropping and resizing, a splitting of 4 sections was performed, resulting in 60 training images for finetuning operation. Each of 60 images was split into 5 (columns) x 4 (rows) = 20 patches (resolution 256 x 256). Five classes were created by merging the subclasses using ground-truth masks: Algae, Mixed, Other, Seagrass, and Substrate_Benthos.

[0115] A summary of the annotation and number of patches been annotated are shown below:

[0116] Fig. 11 shows some example patches from the Fiji 2020 dataset. As there are more than two classes, coverage was estimated differently as compared to the previous example. Here, coverage was estimate according to the following equation:Ns+ Na+ O.SNmcoverage = - — - where Nsis the number of patches in the “seagrass” class, Nais the number of patches in the “algae” class, Nmis the number of patches in the “mixed” class and N is the total number of patches.

[0117] Two examples of the seagrass coverage estimation on sample test images are shown in Fig. 12 and Fig. 13. In other words, Fig. 12 and Fig. 13 shows the results of the patch-based approach that has been finetuned on the Fiji 2020 dataset, for seagrass coverage estimation.Experimental example 2: Grid-based segmentation approach

[0118] Fig. 14 shows a grid-based segmentation approach overview. The grid-based approach consists of two parts: (1) super-pixel mask generation and segmentation model trained using the generated masks. The first step of super-pixel mask generation consists of the following steps:1. Super-pixels are generated for each image for multiple number of seed points;2. Super-pixels are mapped to the patch and the label is projected onto that super-pixel; and3. Iterative refinement of super-pixels and the propagation of labels to generate a final mask.

[0119] Fig. 15 shows a selected sample of images of patch labels before and after mask generation with Fiji 2022 dataset. In the second step, a segmentation model is trained based on the masks generated by the super-pixel step. More specifically, an object-contextual representation approach for semantic segmentation (OCRNet) was exploited and customised in this experimental example. The OCRNet model provides an optimal trade-off between accuracy and speed (15 fps) with fewer training parameters. In this experimental example, binary segmentation was performed to segment seagrass pixels from rest of the image for coverage estimation.

[0120] The public dataset and the Fiji 2022 dataset were combined for the training and validation of the model. The segmentation model was trained from scratch with the following implementation details:• The image dataset was split into 80% for training and 20% for validation.• Standard data augmentation: random horizontal flip, blur, distort.• Target size training: 512 x 512 x 3.• Batch size: 4.• Iterations: 3,000.

[0121] Two experiments were conducted with the grid-based segmentation approach, with algae being included and excluded into foreground:• Experiment A: Seagrass + Algae o class 0: background [substrate + others] o class 1: foreground [Seagrass types (5 types) + Algae]• Experiment B: Only Seagrass (Algae as background) o class 0: background [substrate + others + Algae] o class 1: foreground [Seagrass types (5 types)]

[0122] The segmentation model was trained with a combination of the publicly available dataset (containing 2 classes and 3,066 images) and Fiji 2020 dataset (containing 8 class and 60 images). Fig. 16 shows a visualisation of testing results for Experiment A and B on two selected images, with the corresponding coverage estimation. It can be seen that the best result was obtained when the model was trainedusing the combined dataset Fiji only seagrass and the publicly available dataset, given that segmentation model successfully segments the square guide shown in each image.Performance evaluation

[0123] Both the patch-based classification approach and the grid-based segmentation approach were compared by comparing their respective seagrass coverage estimates. A set of 9 images are selected from the Fiji 2022 dataset with non-expert pixel level full annotation to provide ground truth (such ground truth should be validated by domain experts). The coverage estimation from these two approaches is validated on the test dataset, as shown in Table 3.Table 3. Seagrass coverage estimation results compared from two approaches with nonvalidated “ground truth”.

[0124] Fig. 17 shows some sample results. More specifically, Fig. 17 shows an original image, the “ground truth” segmentation, the patch-based approach to coverage estimation and the grid-based segmentation approach to coverage estimation for sample validation images.Point-based annotations

[0125] Figure 18 illustrates an underwater scene 1800 and a camera 1801 taking camera images of the underwater scene 1800 according to a camera view 1802. Scene 1800 comprises seagrass including a central seagrass 1803 and a peripheral seagrass 1804. There is also a fish 1805.

[0126] Figure 18 further illustrates a camera image 1810 including a depiction of central seagrass 1813 and a depiction of peripheral seagrass 1814. It is noted that camera image 1810 has already been segmented and only seagrass segments are shown (the fish 1805 is segmented as non-seagrass so has been removed from the image). It can be seen that, as a result of the perspective of camera 1801, the central seagrass 1803 appears smaller than the peripheral seagrass 1804 although they are about the same size in scene 1800. This illustrates how an estimate of total seagrass in image 1810 would be inaccurate.

[0127] However, as camera 1801 moves through the scene, it captures further images from different perspectives. As a result, the depictions of the seagrass will also change. This can be used in a three-dimensional reconstruction algorithm, such as structure from motion (SfM), Neural Radiance Field (NeRF) or Gaussian splatting to generate a three-dimensional representation 1820 of scene 1800. Three-dimensional representation 1820 may be in the format of a point cloud or other format, such as a parameterised format including radiance fields or geometrically transformed Gaussians. The three- dimensional representation 1820 may be segmented in the sense that it only contains three-dimensional elements that are classified as seagrass, such as only voxels, mesh nodes, Gaussians, etc. that are classified as seagrass. This can be achieved by mapping the classification of each camera image onto the three-dimensional scene. More specifically, the method may estimate the camera location and orientation for each camera image, to determine a transformation from a three-dimensional point in the three-dimensional representation to a pixel in the camera image. The method can then apply that transformation (its inverse) to the classification to map the classification of that pixel to the three-dimensional point of the three-dimensional representation.

[0128] As the next step, to obtain a more accurate estimate of seagrass coverage, the method can generate an orthoimage, which is a synthetic image that shows the scene without perspective distortions from a camera. An orthoimage can be considered as a synthetic top view for each pixel in the image. Figure 18 shows an orthoimage 1830 where it can be seen that peripheral seagrass 1824 is more accurate and now has the same size as central seagrass 1823.

[0129] The images are then mosaicked together, meaning they are stitched seamlessly to form a single, continuous image. This involves matching the edges of adjacent images and blending them to ensure uniformity in colour and brightness. In the context of underwater seagrass assessment, an orthoimage can be generated to provide an accurate estimate of seagrass coverage. The process involves capturing multiple images of the seagrass from different angles using underwater cameras. These images are then orthorectified to remove perspective distortions, resulting in a top-down view of the seagrass bed and mosaicking the multiple images into a single orthomosaic image. This accurate representation allows for precise measurement and monitoring of seagrass health and distribution, essential for environmental studies and conservation efforts.

[0130] It is also possible to perform segmentation on the orthomosaic image as an alternative or in addition to the mapping of the segmentation of the images to the three- dimensional representation described above.

[0131] Figure 19 illustrates a camera image 1900 where segmented seagrass is shown in solid black and unsegmented seagrass that is visible in the camera image 1900 but not yet segmented as seagrass is shown in white with a black outline. It can be seen that central seagrass 1901 has not been segmented correctly. Therefore, a user can provide a point annotation to indicate that this object is seagrass. The system takes this point annotation as part of the input for the embedding modules to incorporate it into the classification / segmentation output. Similarly, there is a further seagrass 1903 that is thin and long. In this case, the user may provide multiple point annotations 1904 along the edge of further seagrass 1903. Again, this is incorporated into the embedding layer for the semantic segmentation output.

[0132] It is noted that camera 1801 may capture thousands of images of the same scene from slightly different angles. It would not be practical to have a human user annotate each of the thousands of images. Therefore, the proposed method maps the three-dimensional points of the three-dimensional representation to all of the images that include that point and therefore has a segmentation result from each of the images. The annotations from the user can now be either geometrically transformed to apply tothe other image, or the classification can be aggregated across the multiple images to create a single classification for that three-dimensional point of the 3D model.

[0133] The following description provides further details on the method outline above and an overview of the approach is depicted in Figure 20. The proposed seagrass segmentation method may begin by employing a supervised segmentation model, Segformer, as the baseline to generate coarse segmentation masks, which serve as pseudo masks. These training images and pseudo masks are then utilised to train the segment anything model (SAM)-adapter. Additionally, point labels are used as prompts to supervise the model training, ensuring a precise segmentation of the region of interest (leaf-level estimation). Sequential images captured underwater are used to create 3D mesh models and orthoimages of the recorded seagrass habitat. These outputs are then leveraged to map the semantic information back into the 3D space, resulting in a 3D fine-grained semantic model of seagrass.Seagrass SegmentationCoarse Semantic Segmentation

[0134] The baseline network may be based on Segformer . This framework consists of a hierarchical Transformer encoder and a lightweight multi-layer perceptron (MLP) decoder head making it faster and more efficient achieving great results in semantic segmentation datasets such as AED20K and Cityscapes. The framework features a hierarchical Transformer encoder that does not rely on positional encodings, avoiding the need for interpolation of positional codes, which can degrade performance when the testing resolution differs from the training resolution. This encoder generates multiscale features, providing both high-resolution fine features and low-resolution coarse features, contrasting with the single-scale feature output of Vision Transformers (ViT). To handle the quadratic complexity of self-attention with high-resolution inputs, SegFormer employs a sequence reduction process, significantly reducing the computational burden. Additionally, the framework uses overlapped patch merging to preserve local continuity, enhancing the feature representation.

[0135] The lightweight All-MLP decoder aggregates information from different layers of the encoder using MLPs, combining local and global attention. This design avoids the complexity and computational demands of traditional decoders. The MLP decoder benefits from the large effective receptive field (ERF) of the Transformer encoder, which captures both local and global context effectively.

[0136] SegFormer is scaled into different models (SegFormer-BO to SegFormer-B5) to balance performance and efficiency. For example, SegFormer-B4 achieves 50.3% mloU on ADE20K with 64M parameters, outperforming previous methods in both accuracy and efficiency. The framework demonstrates strong robustness to common corruptions and perturbations, making it suitable for safety-critical applications.

[0137] In one example, the model variant SegFormer-B5 (MiT-b5) is adapted to learn both the fine and coarse features of an image. The backbone is pretrained only on ImageNet-lK to decrease computational requirements. SegFormer outperformed other semantic segmentation models that have been used for seagrass segmentation such as Deeplabv3+.Point-supervised Semantic Segmentation

[0138] Seagrass leaves may exhibit irregularities in shape, making them less uniform and more challenging to segment using existing methods. The disclosed method may employ a segment anything model (SAM)-adapter as the backbone, which takes advantage of the powerful generalisation ability of the foundation model SAM. An overview of the proposed point- supervised segmentation framework is shown in Figure 21.

[0139] The model used in this method may be comprise three interconnected components: a promptable segmentation task, a segmentation model, and a data engine for collecting a large dataset called SA- IB. The promptable segmentation task is designed to return a valid segmentation mask given any segmentation prompt, which can include spatial or text information identifying an object. The model may support flexible prompts and can output segmentation masks in real-time, allowing forinteractive use. It consists of an image encoder, a prompt encoder, and a lightweight mask decoder. The image encoder computes an image embedding, the prompt encoder embeds prompts, and the mask decoder predicts segmentation masks. The data engine iterates between using the model to assist in data collection and using the newly collected data to improve the model. The model architecture is designed to support flexible prompts, compute masks in real-time, and handle ambiguity by predicting multiple masks for a single prompt. The data engine is developed to collect a large and diverse set of masks, iterating through assisted-manual, semi-automatic, and fully automatic stages.

[0140] SAM-adapter leverages the knowledge learned from the SAM for image segmentation. It adopts the ViT-H / 16 model as the image encoder, keeping the pretrained weight frozen during training. The mask decoder in SAM-adapter consists of a modified transformer decoder block followed by a dynamic mask prediction head. The pretrained SAM weights initialise and fine-tune the mask decoder during training.

[0141] The SAM-Adapter is designed to enhance the performance of the Segment Anything Model (SAM) in specific segmentation tasks. Instead of fine-tuning the entire SAM network, the SAM- Adapter incorporates domain- specific information or visual prompts into the segmentation network using simple yet effective adapters. This approach leverages the general knowledge learned by the large model and integrates it with task- specific knowledge to significantly improve performance in challenging tasks.

[0142] The SAM-Adapter is a lightweight model that can learn alignment with a relatively small amount of data. It serves as an additional network to inject task-specific guidance information from the samples of that task. Information is conveyed to the network using visual prompts, which has been demonstrated to be efficient and effective in adapting a frozen large foundation model to many downstream tasks with minimal additional trainable parameters and less fine-tuning time and resource consumption.

[0143] The architecture of the SAM-Adapter includes two MLPs and an activation function within the MLPs. The adapter takes the task- specific information and generates prompts that are attached to each transformer layer of the SAM model. The task-specific information can be in various forms, such as texture or frequency information, or hand-crafted rules, and can be flexibly designed to suit the specific task.

[0144] Point Prompt: Although SAM-adapter achieved state-of-the-art performance using pixel-level dense annotations, such detailed annotation may not always be available for practical scenarios. To address this, this disclosure introduces point prompts as additional supervision for training SAM-adapter. Unlike SAM-adapter, which relies on pixel-level annotations, point prompts can be placed along the length and contours of seagrass leaves, matching their irregular shapes more effectively than bounding boxes, and thus allowing the segmentation model to focus precisely on the regions of interest. More importantly, point labels can be sparsely generated, significantly reducing annotation costs. This can be critical in domains requiring professional expertise, which can be expensive or scarce.

[0145] In the proposed framework, given an input image I and point prompts Ppoints, the method first extracts image embeddings E;using SAM-adapter. Sparse prompt embeddings Psare generated from the point prompts, while dense prompt embeddings P, are derived as positional encodings. These embeddings are fused into the integrated feature map:F = / (E;-i- P^P , (1) where f concatenates the combined image and dense prompt embeddings with sparse prompt embeddings. The fused feature map F is then passed to the mask decoder and prediction module, which refines the learned features and predicts masks and the quality scores using F , P, , and Ps. This process leverages the spatial and contextual information from the point prompts. After obtaining predicted masks from the proposedpoint- supervised model, the total loss is calculated by the sum of binary cross-entropy (BCE) and Intersection over Union (loU) loss between the coarse segmentation mask from Segformer and the predicted masks:where Mcoaraeis the coarse segmentation mask from Segformer and Mredis the predicted masks from the the proposed method in Fig. 3.Digital Surface Models and Orthomosaic Generation

[0146] Digital surface models and orthomosaic images are produced using a photogrammetric projection based on OpenDroneMap (ODM) software. ODM employs a Structure from Motion (SfM) 3D reconstruction method to accurately estimate the positions and orientations of cameras, producing a sparse 3D point cloud and texture 3D mesh.3D Seagrass Mapping

[0147] ODM is a software toolkit dedicated to deriving 3D representations from survey imagery. Among its outputs, orthomosaics are produced by projecting the reconstructed 3D model onto a 2D plane, correcting distortions and perspective differences. Consequently, it makes sense to first segment the reconstructed 3D mesh model before projecting its segmentation onto a 2D orthophoto for orthomosaic segmentation (Approach 1). This method uses a precise 2D-to-3D mapping process using estimated poses from input images, which may introduce errors and inconsistencies.

[0148] In other examples, the method directly segments the orthomosaic image (Approach 2). 2D-3D Mapping and Orthomosaic Projection

[0149] The disclosed 2D-3D mapping method aims to assign the most likely label to each vertex of the reconstructed 3D model by projecting it onto each of the segmentation masks obtained from the proposed seagrass segmentation approach. Thisprocess aggregates vertex label candidates from each mask to generate a final vertex segmentation label. The resulting 3D segmentation is subsequently utilised by ODM to produce orthomosaic segmentation. This approach is illustrated in Figure 22, which shows a diagram of the patch-based orthomosaic segmentation approach. To achieve this, a rasterization algorithm was implemented to project the 3D model onto a 2D plane using estimated camera poses. This method tests each vertex of the 3D mesh for visibility in the given plane and determines whether it falls inside or outside a segmented seagrass area. Vertex candidate binary labels from each projection are weighted by the squared norm of the mesh face to which the vertex belongs, aligned most closely with the camera direction, and then averaged across visible projections. The label with the highest score is assigned to the vertex in the 3D segmentation output. Finally, the ODM toolkit is employed to generate a 2D orthophoto using the 3D segmentation as input to obtain the orthomosaic segmentation.Patch-based Orthomosaic Segmentation

[0150] In other examples, the method directly segments orthomosaic images. However, resizing the image to fit a neural network segmentation model may yield poor results due to significant visual discrepancies between training images and orthorectified orthomosaic patches.

[0151] To overcome these difficulties, the method uses an augmented patch-based inference algorithm for orthomosaic segmentation, exclusively trained on acquired images. Fig. 23 shows this inference pipeline in detail. Initially, consider an input orthomosaic image of height h and width w as a tensor of h x w pixels. Given the absence of a canonical orientation in orthomosaic images, the method augments the input by applying geometric transformations. The method performs four rotations (each by 90° ) and scale each rotated version by factors of 1.0 , 2.0 , and 3.0 , resulting in a set of 12 augmented orthomosaics, forming a tensor of 12x / zx w pixels. Subsequently, patches of 512 pixels (the receptive field size of the neural network) are cropped from this augmented orthomosaic pixel tensor. To maintain contextual information and ensure smooth segmentation transitions between patches, the method enforces a 25%overlap between neighbouring patches. This patch extraction employs a standard sliding window algorithm, resulting in a tensor of size r H -512 nrVK-512 n , 12 x | - 1 + 1 x | - | + 1 pixels.512(1-0.25) ) 512(1-0.25) )

[0152] Furthermore, model segmentation may tend to be more accurate in patches centres compared to the comers, attributable to challenging underwater illumination conditions. To mitigate this issue, the method may use a pixel weighting scheme that prioritises the central patch area, giving greater importance to predictions in these regions. The method may employ a 2D Gaussian kernel with 95% of its density concentrated within one-third of the patch’s diagonal from the center.

[0153] Finally, orthomosaic segmentation is achieved by reassembling the segmentations produced by the model for each augmented patch, weighted by the described pixel weighting scheme. This method enhances prediction accuracy by leveraging multiple geometrically transformed and scaled patches, ensuring comprehensive and precise segmentation of the orthomosaic image. Notably, the developed inference procedure has minimal time overhead because the augmented patches can be processed in parallel as batches of images.Table 1: Performance comparison with the public available CHAMELEON, CAMO, and COD 1 OK datasets.

[0154] This disclosure provides a practical solution to the challenges of seagrass monitoring by introducing a label-efficient, point-supervised segmentation method that reduces the need for labour-intensive annotations. By combining this with advanced photogrammetric image projection technology, the method can achieve 3D mapping and detailed habitat analysis.

[0155] It is noted that the method disclosed above relating to 3D orthomosaic image generation and segmentation can equally be performed by the computer system described with reference to Figure 1.

[0156] The method for detecting seagrass in multiple overlapping camera images of an underwater scene is carried out by processor 103, as shown in Figure 1. This involves the processor 103 executing steps to determine a three-dimensional representation of the underwater scene from the multiple camera images, create an orthomosaic image of the scene based on this three-dimensional representation, determine an initial segmentation of the seagrass in the orthomosaic image, receive user input indicating points in the orthomosaic image that contain seagrass, and refine the initial segmentation based on the user input.

[0157] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The presentembodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

CLAIMS:

1. A method for estimating coverage of a plant in an image, the method comprising: dividing the image into multiple patches; applying a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of a first class or a second class, wherein the first class indicates presence of the plant; determining a sub-set of patches containing patches that are classified as the first class; determining a segmentation of each of the sub-set of patches, to determine areas of the sub-set of patches that contain the plant; and calculating the plant coverage of the plant in the image based on the segmentation.

2. The method of claim 1, wherein the trained machine learning classifier is a multi-class classifier, wherein the first class and the second class are each one of multiple classes.

3. The method of claim 1 or 2, wherein the plant is of genus seagrass and plant coverage is for species of seagrass.

4. The method of claim 3, wherein the plant is a first species of seagrass and plant coverage is for the first species of seagrass, wherein the second class is a second species of seagrass different to the first species.

5. The method of any one of claims 2 to 4, wherein the multiple class comprises one or more of: sand; algae; biotic; human-made;coral; and substrate.

6. The method of claim 5, wherein the multiple classes comprise a mixed class, wherein the sub-set of patches further contains patches that are classified as the mixed class.

7. The method of any one of the preceding claims, wherein the trained machine learning classifier is trained using random point annotation.

8. The method of any one of the preceding claims, wherein the trained machine learning classifier is trained using patch-based annotation.

9. The method of any one of the preceding claims, wherein the trained machine learning classifier is based on ConvNext.

10. The method of any one of the preceding claims, wherein determining the segmentation comprises applying a trained machine learning segmentation model to the patches classified with the first class.

11. The method of claim 10, wherein the trained machine learning segmentation model is a binary classifier, wherein applying the trained machine learning segmentation model to the patches classified with the first class comprises classifying each pixel in the respective patches into one of the first class or the second class.

12. The method of claim 11, wherein the binary classifier is trained on images of plant genus.

13. The method of claim 11, wherein the binary classifier is trained on images of a plant species.

14. The method of any one of claims 10 to 13, wherein the trained machine learning segmentation model is based on OCRNet.

15. The method of any one of claims 10 to 14, wherein the method comprises retraining the trained machine learning classifier for a current batch of images from a first environmental context and re-using the trained machine learning segmentation model from previous image batches from previous environmental contexts.

16. The method of any one of claims 1 to 9, wherein determining the segmentation of each of the sub-set of patches comprising creating a binary patch for each of the subset of patches based on RGB thresholding.

17. Software that, when executed by a computer, causes the computer to perform the method of any one of the preceding claims.

18. A system for estimating coverage of a plant in an image, the system comprising: a processor configured to: divide the image into multiple patches; apply a trained machine learning classifier to each of the multiple patches to classify each of the multiple patches into one of a first class or a second class, wherein the first class indicates presence of the plant; determine a sub-set of patches containing patches that are classified as the first class; determine a segmentation of each of the sub-set of patches, to determine areas of the sub-set of patches that contain the plant; and calculate the plant coverage of the plant in the image based on the segmentation.

19. A method for detecting seagrass in multiple overlapping camera images of an underwater scene, the method comprising: determining a three-dimensional representation of the underwater scene from the multiple camera images; creating an orthomosaic image of the scene based on the three-dimensional representation; determining an initial segmentation of the seagrass in the orthomosaic image;receiving user input indicative of points in the orthomosaic image that contain seagrass; and refining the initial segmentation based on the user input.

20. The method of claim 19, wherein the initial segmentation comprises a semantic segmentation.

21. The method of claim 19 or 20, wherein determining the three-dimensional representation comprises performing structure from motion.

22. The method of any one of claims 19 to 21, wherein the determining the initial segmentation comprises determining the initial segmentation for each of the multiple overlapping camera images and projecting the initial segmentation onto the orthomosaic image.

23. The method of claim 22, wherein the mapping uses the three-dimensional representation as a transformation from the multiple overlapping camera images to the orthomosaic image.

24. The method of any one of claims 19 to 23, wherein the method further comprises determining, for each of the multiple overlapping camera images, a camera position to map the segmentation to the orthomosaic image.

25. The method of any one of claims 19 to 24, wherein the segmentation is performed by applying a trained machine learning model.

26. The method of claims 25, wherein the trained machine learning model comprises a visual transformer model comprising an embedding based on the user input.

27. A computer system comprising one or more processors configured to perform the method in any one of claims 19 to 26.

Images