System for monitoring palm tree and oil palm fruitlet thereof in lightweight and efficient manner and method thereof

Abstract

The present invention discloses a system for monitoring a palm tree and an oil palm fruitlet in a plantation in a lightweight and efficient manner and a method thereof. The system comprises a data acquisition module (100), a tree detection module (200), a fruitlet detection module (300), and an environment representation module (400). The data acquisition module (100) comprises a remote sensing device (101) and an image-capturing device (102) mountable to a mobility platform (103) for generating an occupancy grid map associated with the palm tree in the plantation and an image associated with the oil palm fruitlet scattered within a vicinity of the palm tree, respectively. The tree detection module (200) generates a submap from the occupancy grid map to detect the palm tree therein and determine a tree position of the palm tree detected thereof, which is transformed to a global tree position on a semantic or global map. The fruitlet detection module (300) detects the oil palm fruitlet from the image, a fruitlet position, and a quantity of oil palm fruitlets using a grid analysis and transforms the fruitlet position to a global fruitlet position. The environment representation module (400) compares the tree position against a tree tracking list using a chain-based tree-tracking and re-mapping algorithm and assigns a harvest-ready status to the palm tree based on the quantity of oil palm fruitlets detected thereof.

Description

[0001] SYSTEM FOR MONITORING PALM TREE AND OIL PALM FRUITLET THEREOF IN LIGHTWEIGHT AND EFFICIENT MANNER AND METHOD THEREOF

[0002] FIELD OF THE INVENTION

[0003] The present invention generally relates to the field of object identification. More particularly, the present invention relates to a system for monitoring, including detecting, classifying, counting, localising, tracking and mapping a palm tree and an oil palm fruitlet thereof in a plantation in a lightweight and efficient manner, and a method thereof.

[0004] BACKGROUND OF THE INVENTION

[0005] The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognised in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also correspond to implementations of the claimed technology.

[0006] Oil palm trees produce a continuous supply of fruit bunches in the axils of their fronds from which a high-quality oil used chiefly for cooking can be extracted. Palm oil can also be used in food products, cosmetics, detergents, and biofuels. The fruit bunches are typically harvested manually using a variety of tools and methods. The farmers must first examine the maturing bunches on the palm tree and select only those deemed to be ripe for harvesting. The fruit colour typically determines the ripeness of oil palm fruits. The degree of ripeness of those oil palm fruits directly affects the capacity of palm oil production and the oil content extracted therefrom. An optimal maturity stage is when the bunch is optimally ripe as it contains a high oil content that contributes to a high oil extraction rate. The number of fruits that become detached from the fruit bunch can also indicate a certain degree of ripeness that cues for harvesting. The detached or “loose” fruits eventually fall to the ground. Overripe fruits, obviously, contribute to a lower oil extraction rate than a properly ripe bunch. Fruit bunches must be harvested at the right time and ripeness and in the right way. The frequency or interval of harvesting also has a large effect on oil recovery. The suggested frequency of harvesting is once every seven to ten days. The farmers, however, harvest oil palm fruits once every two weeks. Some bunches will get overripe during the long waiting period or interval. Harvesting at too high of intervals can result in fruit bunches becoming over-ripe with a higher percentage of loose fruits. In this case, the collected fruits may have a high free fatty acid content due to lipase activity following detachment. Those loose fruitlets also might be missed as they are covered amongst the weeds. At an excessive frequency, the harvesting may result in a higher percentage of unripe bunches with a lower oil content. Hence, the farmers need to know the best time and frequency or interval to harvest fruit bunches in order to maximise the production yields and avoid losses.

[0007] The utilisation of technology, for example, remote sensing, has become increasingly popular for many activities, including the detection and harvesting of palm trees. Oil palm tree counting and detection not only benefit oil palm industries and smallholders in yield prediction, irrigation and fertilisation planning, and smart management but also contribute to the protection of native species and the ecological environment due to more reasonable oil palm plantation planning. With the rapid development of remote sensing techniques, it has the potential to automatically and accurately detect oil palm trees from high-resolution images instead of time-consuming, labour-exhausting and costly manual work. By way of background, United States Patent No. US 10,223,807 B1 (hereinafter “the ‘807 patent”) discloses a method of localisation for a non-transitory computer-readable storage medium storing one or more programs comprising instructions, which when executed by a computing device, causes the computing device to perform utilising one or more autonomous vehicle driving modules that execute processing of images from a camera and data from a LiDAR the following steps comprising aligning a 3D submap with a global map; extracting features from the 3D submap and the global map; classifying the extracted features in classes; and establishing correspondence of features in a same class between the 3D submap and the global map.

[0008] Conventional technologies, however, suffer from the inability to detect palm trees with different appearances on the plantation, to track the position of palm trees during detection, to know whether or not the same palm tree has been detected previously and to update the tree position upon changes of the map, for instance, a closed loop. Existing technology further suffers from the problem of detecting and counting oil palm fruitlets scattered around palm trees from different angles.

[0009] The fruitlet detection component is one of the most important components in the harvest-ready tree detection system, as it depends on many other components like fruitlet counting, tree harvest-readiness classification, tree harvestreadiness visualisation, etc. As the solution depends on counting the loose fruitlets to determine the harvest readiness of the tree, the fruitlet detection component is the first developed component of the whole system. Next, after detecting the fruitlets, the system needs to know the tree to which those loose fruitlets belong to update the tree’s harvest-ready status. Furthermore, since the harvest-readiness of a tree was determined by the number of loose fruitlets scattered around the tree, there is also a need to ensure that the loose fruitlets scattered around the tree must be able to capture by the camera for counting. Thus, the system needs to have a method to detect and count loose fruitlets from different angles to ensure full coverage of the loose fruitlets scattered around the tree, as some of the loose fruitlets might not be visible by the camera at certain angles, as shown in Figure 7.

[0010] The abovementioned issues may be solved by integrating the fruitlet and tree detection components. The fruitlet detection process will be triggered only when a ‘tree-in’ event happens (the detected tree enters the Rol in the camera view as defined by the user.) The fruitlets counting component will then count the loose fruitlets until the ‘tree-out’ event happens (the detected tree exits the Rol in the camera view as defined by the user.), as shown in Figure 8. In this way, the system is not only able to count the number of loose fruitlets from different angles of view but also can straightaway know the tree that those loose fruitlets belong to. However, since the prior art needs to be integrated with the camera-based tree detection component, there is a need to propose a new method to associate the detected fruitlets with the tree.

[0011] The tree detection component is also an important component in the harvest-ready tree detection system as it is the entry point of the system and the dependencies of many other components like tree harvest-readiness classification, tree position tracking, tree position mapping, tree position visualisation, etc. Traditionally, tree detection can be done in two ways: from images captured by the camera or from 3D point cloud data captured by a 3D-UDAR or stereo camera. The prior art performed tree detection by analysing the images streamed from the cameras. However, streaming data from both LiDAR and cameras can cause performance degradation due to insufficient bandwidth. This issue often happens when dealing with a resource-constrained platform or having multiple processes utilising the device’s bandwidth simultaneously (for example, the high-resolution image streaming for fruitlet detection). Furthermore, as different palm tree varieties have unique characteristics and appearances, as shown in Figure 1 , the detection method based on cameras required a lot of training images that covered all the different appearances of the tree. Therefore, the detection model might be unable to detect the trees with appearances not included in the training data, which causes false negative (FN) detection.

[0012] The tree tracking and clustering component was added alongside the tree detection component in the prior art. Detecting and tracking the tree position from the different regions of the camera view can prevent multiple detections for the same tree and provide a more accurate position estimation as the tree position was computed from different angles. However, it could still be challenging to fuse this information generated by the cameras to the map generated by the LiDAR. The position of the cameras and the LiDAR need to be calibrated precisely to get a high- accuracy result. Thus, this approach is not friendly for the portable and plug-and- play platform, as the user must do calibration every time they assemble the devices.

[0013] Detecting and tracking the tree position from the different regions of the camera view can provide a more accurate position estimation as the tree position was computed from different angles of view. However, it could still be challenging to fuse this information generated by the cameras to the map generated by the LiDAR. Alongside the challenges of fusing and calibrating multiple sensors (cameras and LiDAR), there is also a challenge of realising the fused position data with the generated tree position data. For example, the SLAM algorithm generated the final map with the fusion of both the visual odometry data from the stereo camera and the laser-scanning data from the 2D LiDAR to achieve a robust and high-accuracy result. Since the tree position was computed on the spot with the raw laser-scanning reading, realising the fused position data with the generated tree position data is necessary to map the tree position accurately. A software tracker for tree position mapping might be needed to solve the above challenge. The tracker will track and map the initial tree position obtained from the raw laser-scanning data to the corresponding position on the map obtained from the fused visual odometry laser-scanning data. In this way, the system can update the tree position to a new position on the map whenever the corresponding position on the map has changed during an event like loop closure. Furthermore, tracking the global position of a tree on the map can also prevent multiple tree detections by merging the repeated detection when the robot revisits a scanned area, as shown in Figure 6. However, the software tracker tracked and updated the position of all the trees on the map whenever a change on the map happened, which is usually not necessary most of the time. The changes on the map only affect a small area. This issue became critical with the increase in the size of the map and the number of trees, which slowed down the whole position mapping / position updating process and caused a lot of unnecessary waste of computing resources.

[0014] Hence, there is a continuing need in the art to improve and remedy such systems. More particularly, it would be desirable to provide a system for detecting a palm tree in a given plantation and harvesting oil palm fruit bunches therefrom that is lightweight, efficient, effective, reliable, inexpensive and easy to use, and yet estimates the harvest-readiness status of palm trees, provides the plantation environment with a virtual map.

[0015] For the reasons stated above and for other reasons which will become apparent to those skilled in the art upon reading and understanding the specification, there is a need in the art for an improved system for monitoring a palm tree and an oil palm fruitlet thereof in a plantation in a lightweight and efficient manner. Although there may be similar approaches to the same in prior art, there is still considerable room for improvement for many practical purposes.

[0016] SUMMARY OF THE INVENTION

[0017] The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.

[0018] Accordingly, the present invention provides a system for monitoring a palm tree and an oil palm fruitlet thereof in a plantation in a lightweight and efficient manner. The system of the present invention is characterised by a data acquisition module comprising a remote sensing device and an image-capturing device mountable to a mobility platform configured for generating an occupancy grid map associated with the palm tree in the plantation and an image associated with the oil palm fruitlet scattered within a vicinity of the palm tree, respectively; a tree detection module configured for generating a submap from the occupancy grid map to detect the palm tree therein and determine a tree position of the palm tree detected thereof which is transformed to a global tree position on a semantic or global map prepared from the occupancy grid map; a fruitlet detection module configured for detecting the oil palm fruitlet from the image, a fruitlet position, and a quantity of oil palm fruitlets using a grid analysis, wherein the fruitlet detection module transforms the fruitlet position to a global fruitlet position to associate the same with the global tree position of the palm tree detected thereof; and an environment representation module configured for comparing the tree position against a tree tracking list using a chain-based tree-tracking and re-mapping algorithm, and for assigning a harvestready status to the palm tree based on the quantity of oil palm fruitlets detected thereof, wherein the environment representation module clusters the tree position using a density-based spatial clustering of applications with noise (DBSCAN) algorithm and initiates noise filtering for processing the palm tree detected thereof with a tree marker list.

[0019] Preferably, the tree detection module, upon detection of the palm tree, computes a tree score for the palm tree detected thereof and compares the same against a threshold score.

[0020] Preferably, the tree detection module determines the tree position with respect to the mobility platform prior to the transformation to the global tree position.

[0021] Preferably, the fruitlet detection module, with respect to the grid analysis, employs a grid with a region of interest, through which the quantity of oil palm fruitlets is determined.

[0022] Preferably, the fruitlet detection module determines the fruitlet position with respect to the mobility platform prior to the transformation to the global fruitlet position.

[0023] Preferably, the image includes a still image, a set of still images, a moving image, or any combinations thereof. Preferably, the image-capturing device includes a set of image-capturing devices mountable to opposing sides of the mobility platform thereof.

[0024] Preferably, the remote sensing device includes a non-image-based depth sensing device mountable to an elevated position relative to the image-capturing device.

[0025] Preferably, the system further comprises a visualisation module configured for displaying data including the occupancy grid map, the submap, the semantic or global map, the image, the palm tree with the tree position including the global tree position, the oil palm fruitlet with the fruitlet position including the global fruitlet position, the tree marker list, the tree tracking list, the harvest-ready status, and camera views from the image-capturing device thereof.

[0026] In accordance with another aspect of the invention, a method of monitoring a palm tree and an oil palm fruitlet thereof in a plantation in a lightweight and efficient manner is provided. The method of the present invention is characterised by the steps of providing a data acquisition module comprising a remote sensing device and an image-capturing device mountable to a mobility platform; generating, by the remote sensing device, an occupancy grid map associated with the palm tree in the plantation; generating, by the image-capturing device, an image associated with the oil palm fruitlet scattered within a vicinity of the palm tree; generating, by a tree detection module, a submap from the occupancy grid map to detect the palm tree therein; determining, by the tree detection module, a tree position of the palm tree detected thereof and transforming the same to a global tree position on a semantic or global map prepared from the occupancy grid map; detecting, by a fruitlet detection module, the oil palm fruitlet from the image, a fruitlet position, and a quantity of oil palm fruitlets using a grid analysis; transforming, by the fruitlet detection module, the fruitlet position to a global fruitlet position to associate the same with the global tree position of the palm tree detected thereof; comparing, by an environment representation module, the tree position against a tree tracking list using a chain-based tree-tracking and re-mapping algorithm; assigning, by the environment representation module, a harvest-ready status to the palm tree based on the quantity of oil palm fruitlets detected thereof; clustering, by the environment representation module, the tree position using a density-based spatial clustering of applications with noise (DBSCAN) algorithm; and initiating, by the environment representation module, noise filtering for processing the palm tree detected thereof with a tree marker list.

[0027] Advantageously, the system of the present invention can be deployed to a resource-constrained device as it is designed to be lightweight and requires only a single 2D LiDAR to perform tree detection and SLAM. No specific skills are needed to modify the system’s hardware as the system is designed to be highly portable and easily integrated into any mobility platform. No particular skills are required to operate the system as the system will automatically perform SLAM, tree detection, fruitlet detection, etc., to determine the harvest-readiness of a tree and generate the environment representation. Since the self-localisation process for SLAM and tree detection depends on the same 2D LiDAR, no other steps are required to calibrate / fuse multiple sensors. The system of the present invention determines the harvest-readiness of a tree based on the number of loose fruitlets scattered around the tree, which is more accessible compared to other methods that try to assess the appearance of the fruit on the trees to evaluate its maturity level. The system of the present invention can generate a map and indicate the harvest-readiness of the trees on the map, which provides more efficient agricultural resource management for harvesting activities. In current practice, the harvester must search for the tree to be harvested before they cut down the fruit. With the assistance of the map, they can go directly to the identified ready-to-harvest tree. Thus, it will help to eliminate missed trees during harvesting.

[0028] The foregoing and other objects, features, aspects and advantages of the present invention will become better understood from a careful reading of a detailed description provided herein below with appropriate reference to the accompanying drawings.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS

[0030] A more complete appreciation of the invention and many of the attendant advantages thereof will be readily as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein: Figure 1 shows examples of different palm tree varieties’ appearances, wherein 1 (a) trunk with no fronds, 1 (b) trunk with remaining fronds and less weed, and 1 (c) trunk full of fronds according to one embodiment of the present invention;

[0031] Figure 2(a) shows the map generated by the laser-scanning data from a 2D LiDAR according to one embodiment of the present invention;

[0032] Figure 2(b) shows the position of the detected trees on the generated map with markers according to one embodiment of the present invention;

[0033] Figure 3(a) shows the generated 2D submap (i.e., the extracted occupancy grid map) according to one embodiment of the present invention;

[0034] Figure 3(b) shows the detection of circle / semicircle shapes in the submap according to one embodiment of the present invention;

[0035] Figure 4 shows the steps employed for tracking a tree position with the “T ree In” and “Tree Out” events according to one embodiment of the present invention;

[0036] Figure 5 shows the clustering of the tree position using the DBSCAN algorithm according to one embodiment of the present invention;

[0037] Figure 6 shows the merging of the repeated detection of a tree according to one embodiment of the present invention;

[0038] Figure 7 shows the counting of loose fruitlets with different angles of view according to one embodiment of the present invention;

[0039] Figure 8 shows the counting of loose fruitlets with multiple frames according to one embodiment of the present invention;

[0040] Figure 9 indicates that a tree is harvest-ready by updating the marker that represents the tree on the map to green according to one embodiment of the present invention;

[0041] Figure 10 shows the objectives according to one embodiment of the present invention; Figure 11 shows the high-level architecture of the system according to one embodiment of the present invention;

[0042] Figure 12 shows the remote sensing device and the image-capturing device (in pairs) mounted on a mobility platform according to one embodiment of the present invention;

[0043] Figure 13 shows the steps involved in the tree detection module according to one embodiment of the present invention;

[0044] Figure 14 shows the steps involved in the fruitlet detection module according to one embodiment of the present invention;

[0045] Figures 15A-15F show the steps involved in the environment representation module according to one embodiment of the present invention;

[0046] Figure 16 shows the user interface with RViz according to one embodiment of the present invention;

[0047] Figure 17 shows the satellite view of the test area from the Google Map platform according to one embodiment of the present invention;

[0048] Figure 18 shows the simplified 2D view of the test area according to one embodiment of the present invention;

[0049] Figure 19 shows the generated semantic or global map according to one embodiment of the present invention; and

[0050] Figure 20 is a flow diagram depicting an overall method of monitoring a palm tree and an oil palm fruitlet on a plantation in a lightweight and efficient manner.

[0051] It is noted that the drawings may not be to scale. The drawings are intended to depict only typical aspects of the invention, and therefore should not be considered as limiting the scope of the invention. DETAILED DESCRIPTION OF THE INVENTION

[0052] The present invention discloses a system for monitoring a palm tree and an oil palm fruitlet thereof in a plantation in a lightweight and efficient manner and a method thereof, that can be made, used, and maintained in a highly specific and compact, cost-effective, quick, and simple manner, without the use of complicated and sophisticated steps, components, or parts. The present invention aims to estimate a harvest readiness status of a palm tree, represent the plantation environment with a virtual map, and tag palm trees on the map with an indication of the harvest readiness status of the same. It is also an objective of the present invention to develop a lightweight and efficient system and method for tree detection, classification, localisation, and mapping using only a laser scanning data from a single two-dimensional (2D) laser detection and ranging or a light detection and ranging (LADAR / LiDAR) device. A further objective of the present invention is to develop a reliable method for tree position clustering detection, noise filtering, position tracking, and position re-mapping.

[0053] The term “lightweight”, as used herein, at least includes a module that does not depend on file system access to a computing device at which it is configured to gather inventory data. For example, the lightweight executable module may not have a write access to the file system of a monitored computing device and may only write to temporary non-persistent memory. Further, the term may also include a feature of an executable module that is configured to throttle operation in order to reduce operation performance degradation of components of the computing device such as software applications, hardware components, and the like. The term may also refer to a feature of the executable module wherein an output data object is a human-readable text after machine-readable decompression and / or decryption is performed. In other words, the lightweight executable module may be configured to generate relatively simple string data objects compared to heavier rich data objects other than string data objects. In an embodiment, the lightweight executable module may be referred to herein as an enumeration module. The enumeration module may generate a data object in-memory. The data object may be compressed and encrypted for durable transport across networks, including non-secure network paths. The lightweight executable module may be useful in any networked device configuration. For example, the lightweight executable module may be configured to perform operations in an Internet of Things (loT) environment wherein networked devices may have relatively limited resources for device enumeration, including status reporting and the like.

[0054] The term “plantation”, as used herein, is intended to include natural and man-made estate or a large piece of land on which a group of palm trees are planted, grown and cared for. The term may encompass a systematic plantation system at a uniform spacing between palm trees, accommodating a number of palm trees in a given area.

[0055] In accordance with one preferred embodiment of the present invention, the system comprises a data acquisition module 100, a tree detection module 200, a fruitlet detection module 300, an environment representation module 400, and a visualisation module 500, which are interconnected with each other in a manner described herein below in conjunction with the accompanying figures. In an embodiment, the tree detection module 200, the fruitlet detection module 300, the environment representation module 400, and / or the visualisation module 500 are deployed as a single unitary device that is in communication and can be used with the data acquisition module 100.

[0056] The data acquisition module 100 preferably comprises a remote sensing device 101 and an image-capturing device 102. The remote sensing device 101 and the image-capturing device 102 are preferably mountable to a mobility platform 103. The mobility platform 103 may be a controlled mobile robot, an autonomous vehicle, or any vehicle with mobility through wheels, drive tracks, cogs, or other similar mechanisms. It is preferred that the remote sensing device 101 and the image-capturing device 102 are configured for generating an occupancy grid map associated with the palm tree in the plantation and an image associated with the oil palm fruitlet scattered on a ground (also known as loose fruitlet) within a vicinity of the palm tree (e.g., a predefined radius from the palm tree), respectively.

[0057] The remote sensing device 101 preferably includes a non-image-based depth sensing device mountable to an elevated position relative to the imagecapturing device 102, preferably between image-capturing devices 102. The elevated position may be associated with the top-most surface of the mobility platform 103, such as the roof or top of the mobility platform 103. In one embodiment, the remote sensing device 101 is deployed on the roof of the mobility platform 103, which is the highest elevated position along a point (e.g., a midpoint) between image-capturing devices 102 deployed on the opposing sides of the mobility platform 103. The non-image-based depth sensing device, which is the remote sensing device 101 , can be selected from a group comprising a LADAR / LiDAR device that uses a laser light source to probe the characteristics of a surface target (i.e., the palm tree), a photo wave device, a time-of-flight measurement device, and the like.

[0058] The occupancy grid map is preferably a spatial representation of the environments of the mobility platform 103, which primarily contains palm trees. It represents environments by fine-grained metric grids of variables that reflect the occupancy of the environment. Once acquired, the occupancy grid map can also be used for mobility platform navigation, such as localisation, path planning, obstacle detection and avoidance, and object (i.e., palm tree) finding.

[0059] The image-capturing device 102 preferably acquires one or more images of a scene external to the mobility platform 100. In one embodiment, the imagecapturing device 102 preferably includes a set of image-capturing devices 102 mountable to opposing sides of the mobility platform 103 thereof. Preferably, the set of image-capturing devices includes a pair of image-capturing devices 102 that is positioned to face forward in a direction that matches a travelling direction of the mobility platform 100, for instance, a pose of the mobility platform 100. In one embodiment, the said image-capturing device 102 may include an RGB camera (that delivers coloured images of objects by capturing light in red, green, and blue wavelengths (RGB)), an RGB-D camera (that captures RGB images along with per- pixel depth information), a stereo camera including a passive stereo camera and an active stereo camera, and the like. The RGB camera is preferred in the present invention. In one exemplary embodiment, the RGB camera is a three-colour camera providing RGB colour signals, in the form of analogue or digital electrical signals, regarding the object or sample viewed with the camera. Examples of three-colour cameras include, for example, CCD video cameras, CID video cameras, vidicon tube video cameras, CCD still cameras, CID still cameras, CCD line scan cameras, and CID line scan cameras. The term “image”, as used herein, may refer to a stereo pair of left and right images captured by the image-capturing device 102. It is preferred that the image generated by the image-capturing device 102 includes a still image, a set of still images, a moving image, or any combination thereof. The term “still image” refers to a photograph taken at a single point in time. The term “set of still images” refers to two or more still images, with each image in the set taken at a separate point in time during the same photography or video session. The term “moving image” refers to an image formed from a sequence of frames in which objects within regions of the image change substantially over time. One skilled in the art would realise that the use of these terms is for clarity, and the use of these terms in the following description does not limit the present invention.

[0060] The oil palm fruitlet captured in the image by the image-capturing device

[0061] 102 is preferably those from the vicinity of the palm tree, which must be within the predefined radius from the palm tree. The predefined radius may or may not overlap with the other radiuses of other neighbouring palm trees. In one embodiment, the predefined radius is determined relative to a trunk diameter of the palm tree. For instance, the predefined radius on which the oil palm fruitlet will only be counted as belonging to the said palm tree is at least twice the trunk diameter of the palm tree.

[0062] The tree detection module 200 is preferably configured to generate a submap from the occupancy grid map to detect the palm tree therein. The tree detection module 200 preferably determines a tree position (e.g., a centroid coordinate such as (x, y)) of the palm tree detected thereof and transforms the tree position (with or without the information on the pose of the mobility platform 103) to a global tree position (e.g., a global reference coordinate such as (x, y, z)) on a semantic or global map prepared from the occupancy grid map. The tree detection module 200 preferably determines the tree position with respect to the mobility platform 103 prior to the transformation of the tree position to the global tree position. It is preferred that the tree detection module 200, upon detection of the palm tree, computes a tree score for the palm tree detected thereof and compares the same against a threshold score.

[0063] The fruitlet detection module 300 is preferably configured for detecting the oil palm fruitlet from the image, a fruitlet position, and a quantity of oil palm fruitlets using a grid analysis. Assuming the presence of oil palm fruitlet within the predefined radius thereof, the determination of the quantity of oil palm fruitlets will be made by the fruitlet detection module 300. In an embodiment, the fruitlet detection module 300, with respect to the grid analysis, employs a grid with a region of interest, through which the quantity of oil palm fruitlets is determined. The fruitlet detection module 300 preferably determines the fruitlet position with respect to the mobility platform 103. Once the fruitlet position with respect to the mobility platform

[0064] 103 is determined, the fruitlet detection module 300 transforms the fruitlet position (with or without the information on the pose of the mobility platform 103) to a global fruitlet position to associate the same with the global tree position of the palm tree detected by the tree detection module 200.

[0065] The environment representation module 400 is preferably configured to compare the tree position obtained by the tree detection module 200 against a tree tracking list using a chain-based tree-tracking and re-mapping algorithm. The environment representation module 400 is also configured to assign a harvestready status to the palm tree based on the quantity of oil palm fruitlets detected by the fruitlet detection module 300. According to one embodiment, in relation to the assignment of the harvest-ready status, the environment representation module 400 compares the quantity of oil palm fruitlets against a threshold quantity to conclude a harvesting readiness of the said palm tree. The threshold quantity, according to one representative embodiment, is about 1 -10 loose fruitlets. Once the harvesting readiness is concluded for the palm tree, a harvest-ready status will be assigned to the palm tree. Records associated with them will be correspondingly updated.

[0066] It is preferred that the environment representation module 400, after the chain-based tree-tracking and re-mapping algorithm but before the assignment of the harvest-ready status, clusters the tree position using a density-based spatial clustering of applications with noise (DBSCAN) algorithm and initiates noise filtering for processing (e.g., entering a new entry and / or updating information of) the palm tree detected thereof with a tree marker list.

[0067] The visualisation module 500 is preferably configured for displaying data to a user. The data preferably includes, but is not limited to, the occupancy grid map, the submap, the semantic or global map, the image, the palm tree with the tree position including the global tree position, the oil palm fruitlet with the fruitlet position including the global fruitlet position, the tree marker list, the tree tracking list, the harvest-ready status, and camera views from the image-capturing device 102 thereof.

[0068] For clarity, the method of monitoring a palm tree and an oil palm fruitlet thereof in a plantation in a lightweight and efficient manner described in conjunction with the system in the preceding paragraphs will be summarised as follows: S100: providing a data acquisition module 100 comprising a remote sensing device 101 and an image-capturing device 102 mountable to a mobility platform 103;

[0069] S101 : generating, by the remote sensing device 101 , an occupancy grid map associated with the palm tree in the plantation;

[0070] S102: generating, by the image-capturing device 102, an image associated with the oil palm fruitlet scattered on a ground within a vicinity of the palm tree;

[0071] S103: generating, by the tree detection module 200, a submap from the occupancy grid map to detect the palm tree therein;

[0072] S104: determining, by the tree detection module 200, a tree position of the palm tree detected thereof and transforming the same to a global tree position on a semantic or global map prepared from the occupancy grid map;

[0073] S105: detecting, by the fruitlet detection module 300, the oil palm fruitlet from the image, a fruitlet position, and a quantity of oil palm fruitlets using a grid analysis;

[0074] S106: transforming, by the fruitlet detection module 300, the fruitlet position to a global fruitlet position to associate the same with the global tree position of the palm tree detected thereof;

[0075] S107: comparing, by the environment representation module 400, the tree position against a tree tracking list using a chain-based tree-tracking and re-mapping algorithm;

[0076] S108: assigning, by the environment representation module 400, a harvest-ready status to the palm tree based on the quantity of oil palm fruitlets detected thereof; S109: clustering, by the environment representation module 400, the tree position using a density-based spatial clustering of applications with noise (DBSCAN) algorithm; and

[0077] S110: initiating, by the environment representation module 400, noise filtering for processing the palm tree detected thereof with a tree marker list.

[0078] Although the method is depicted as a sequence of numbered steps for clarity, the numbering does not necessarily dictate the order of the steps. It should be understood that some of these steps may be skipped, performed in parallel, or performed without the requirement of maintaining a strict order of sequence.

[0079] The present invention will be specifically described by the following examples, but it should be understood that the present invention is not limited in any way to these examples.

[0080] The alternative approach proposed in this example solved the problems with a more lightweight methodology by performing SLAM and tree detection through 2D submap (i.e., the occupancy grid map constructed by the laser-scanning data from a 2D LiDAR for a defined period) analysis. Since the newly developed system mainly deals with laser scanning data from a 2D LiDAR, it is more lightweight and cost-effective than the prior art, which performs SLAM and tree detection with data from three stereo cameras and one LiDAR. The preliminary evaluation results show that the newly developed system of the present invention can achieve comparable performance compared to the prior art, especially when dealing with resource- constrained platforms.

[0081] Tree Detection

[0082] This example proposes a new approach for tree detection with a 2D submap (i.e., the occupancy grid map constructed by the laser-scanning data from a 2D LiDAR for a defined period). The laser-scanning data used to construct the 2D submap is much smaller than the images and 3D point cloud data, thus offering better efficient use of the computational resources. When installed horizontally at a certain height, the 2D LiDAR can capture obstacles at the same height as the sensors. One study shows that the height of a tree can grow from 2m to 10m from 2 to 25 years old. Thus, in this example, the 2D LiDAR is proposed to be installed horizontally at 1.5 m above the ground. In this way, the 2D laser-scanning can be higher than any on-the-ground obstacles and lower than the trees’ leaves, leaving only the tree trunks to be scanned as obstacles. The LiDAR generates different sets of laser-scanning data when a mobility platform, i.e., a robot, is moving. The system can construct an occupancy-grid map based on the laser-scanning data, as shown in Figure 2(a), when performing SLAM. An occupancy grid map is a map that shows the occupancy of spaces, whether there is any object in a particular area or free space. The pixels, commonly known as the cells in the occupancy grid map, can be divided into one of three groups: occupied, not occupied, and unknown. Occupied cells are the cells that contain obstacles and may not be traversed. These cells are indicated with a different level of intensity and alpha of grey textures. On the other hand, the white cells that are not filled with grey textures represent the non-occupied cells, indicating the free space that the robot can traverse. The remaining cells that are not scanned by the LiDAR will remain in teal, which indicates the unknown. With the occupancy grid map, the system can indicate the position of the detected tree with markers, as shown in Figure 2(b).

[0083] To perform tree detection in real-time, the occupancy grid map was extracted from the system automatically from time to time. The extracted occupancy grid map for a certain period (or a certain number of steps) was known as a submap, as shown in Figure 3. Furthermore, Figure 3 also showed that the occupied cells corresponding to the oil palm trees were represented with a circle / semicircle shape, replicating the trunk’s shape. Next, the extracted submap was fed to a CNN model that was trained to classify and determine the position of the circle / semicircle shape in the submap. Assuming all objects with a height above 1.5 m and a trunk-liked structure in the plantation are a tree, the system will first consider all the detected circle / semicircle shapes in the submap as potential trees. This example also proposed a mechanism to track the trees’ position and filter incorrect detection results from the circle / semicircle shape generated by the noise by cross-checking the detection results from different submaps.

[0084] The new tree detection approach proposed in this example detects trees from the circle / semicircle shapes in a 2D submap. This approach reduces computational resources and memory bandwidth and offers a wider Field-of-View (FoV) than cameras since the 2D submap was built by 360-degree laser scanning data. In contrast, the image captured by the camera was limited to a certain angle where the camera was mounted. Furthermore, analysing the 2D laser-scanning pattern generated by the LiDAR is not affected by the varieties and appearances of the tree. The detection can be precise most of the time as long as we can ensure all objects with a height above 1 .5 m and a trunk-like structure in the plantation are a tree.

[0085] Tree Tracking and clustering

[0086] This example proposes a new approach for tree detection with a 2D submap (i.e., the occupancy grid map constructed by the laser-scanning data from a 2D LiDAR for a defined period). This approach does not require complex calibration as both the localisation processes (i.e., SLAM and trees’ position estimation) are entirely based on the same 2D LiDAR. As mentioned in the proposed solution for tree detection, the newly developed system performs tree detection by analysing the submap generated by the 2D LiDAR from time to time. Alongside the new methodology for tree detection, this example also proposed a new methodology for tree tracking and clustering. In this approach, the system will compute the global position of the tree at the same time as detecting the tree from the submap. Thus, the tree can be tracked by its global position, as there could never be more than one tree in a single position. Therefore, by knowing the global position of the tree, the system can filter redundancy detection by clustering the trees’ position proposal (if more than one proposal has been made for the same tree) with a Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm. As shown in Figure 5., by applying the DBSCAN with a minimum number of samples for each cluster, the system can filter most of the False Positive detection results from the circle / semicircle shape generated by the noise by cross-checking the detection results from different submap.

[0087] Tree Pcsiticn Tracking and Re-mapping

[0088] The chain-based tree tracking and re-mapping algorithm is one of the key inventions of this example. This algorithm was developed by introducing a tracking Rol to improve the existing tracking algorithm. In this algorithm, the position of all the trees that are published on the map will be registered and chained to a list. This algorithm will track only the trees that fall in the tracking Rol instead of tracking all the trees in the list. The tracking Rol will always track the newly constructed area on the map for changes. Therefore, upon the change on the map, the algorithm will first track and update the position of the trees within the tracking Rol. After that, it will move the Rol in the reverse direction of the chain and update the corresponding tree. The algorithm will stop the position tracking and updating process once a tree position in the chain is found unaffected by the latest changes in the map. At this point, the algorithm will move back to the newly constructed area.

[0089] Fruitlet Detection and Counting

[0090] This example proposes a new fruitlet detection approach that does not depend on a camera-based tree detection component. The proposed fruitlet detection approach detects fruitlets using a ‘fruitlet-in’ and ‘fruitlet-out’ event. The ‘fruitlet-in’ event happens when the first detected fruitlet enters the Rol, while the ‘fruitlet-out’ event happens when the last detected fruitlet exits the Rol. The fruitlets count was then determined by computing the average number of fruitlets per frame that were present in the Rol within the ‘fruitlet-in’ and ‘fruitlet-out’ events. If the number of loose fruitlets counted is greater than the threshold number of loose fruitlets for classifying as harvest-ready, the system will then estimate the position of the detected fruitlets and associate it with the nearest tree object. Suppose the offset between the estimated position of the detected fruitlets and the position of the nearest tree object is smaller than a preset threshold (0.5 m). In that case, the markers for the corresponding tree object will be updated to green, indicating the tree is harvest-ready, as shown in Figure 9.

[0091] The invention aims to develop a system and method for detecting, classifying, and tagging palm trees with harvest-ready status in a semantic map representing a plantation area. The plantation manager can use the generated semantic map to plan for the right resources for harvesting, which provides more efficient agricultural resource management for harvesting activities, as shown In Figure 10.

[0092] The main objective of this invention is to automatically produce a semantic map that represents the actual plantation area with the harvest-readiness status of each of the oil palm trees in the area to provide more efficient agricultural resource management for harvesting activities. The invention achieves the objective by introducing a method to perform SLAM and tree detection simultaneously with the laser-scanning data produced by a 2D LiDAR. Besides, to classify the harvestreadiness status of the tree detected, the system also adopted a loose fruitlet detector to detect and count the loose fruitlets scattered around the tree. Furthermore, the system also improved the tree tracking and re-mapping component by introducing a more efficient chain-based tree tracking and remapping algorithm that instantly realises the fused position data used to construct the map to the published tree position.

[0093] Figure 11 shows the high-level architecture of the system. Generally, the system comprises a data acquisition module 100, a tree detection module 200, a fruitlet detection module 300, an environment representation module 400, and a visualisation module 500.

[0094] • Data Acquisition Module 100

[0095] The data acquisition module is responsible for communicating with the hardware sensor. It configures and launches the sensors during the setup stage and establishes a pipeline to fetch the data from the sensor to the system. For this example, a minimum of one 2D lidar and two RGB cameras are required. It is hardware-dependent. Theoretically, any 2D lidar with a scanning range greater than 10 meters and RGB cameras with a resolution greater than half-HD can be used. However, different hardware required different drivers and configurations to launch. The module for specific hardware might not work properly for another. Therefore, the data acquisition module needs to be developed based on the drivers and configuration for the respective hardware that users wish to use to collect data.

[0096] Furthermore, the data acquisition module also consists of the configuration / calibration file for users to configure the sensors. For example, this example uses the 2D LiDAR RPLIDAR S1 from SLAMTEC and Alvium 1800-U319 from Allied Vision, as shown in Figure 12. The specification and the mounting position of the hardware are shown in Table 1 . (The mounting position for another hardware might be slightly different, subject to the hardware’s size, focus length, etc.). Table 1 . LiDAR and Cameras’ Specification

[0097] • Tree Detection Module 200

[0098] With reference to Figure 13, the laser-scanning data acquired by the data acquisition module will be fetched to perform SLAM and generate the occupancy grid map. This invention used the SLAM system from Google’s Cartographer Project to perform Local SLAM and generate the occupancy grid map based on the laser- scanning data fetched to the system. The tree detection module will first extract the submap from the occupancy grid map built by the Cartographer’s local SLAM to perform tree detection. When the robot is moving, the LiDAR gets to scan the tree from different angles. Therefore, each frame can consist of partial information to construct a completed shape of the tree. In this invention, the extracted submap, which was generated by the Cartographer, combines a specific number of frames in sequence (which is known as steps) to produce the submap. In this way, the module can combine the laser-scanning information from different frames to construct a completed shape of a tree on the submap, which can then be extracted for tree detection.

[0099] Next, the extracted submap was fed to a CNN model trained to classify and determine the position of the circle / semicircle shape in the submap. If a tree were found on the submap with a score greater than the minimum score threshold, the module would then compute the position of the tree with respect to the robot. Lastly, since the final position of the tree computed is in the local coordinate (with respect to the robot), a transformation function was applied to compute the position of the detected tree to the global position (i.e., with respect to the semantic map). The transformation function gets the ‘transform’ of the robot’s local coordinate to the semantic map coordinate when the tree is detected and uses that ‘transform’ to compute the global position of the detected tree. The ‘transform’ is a ROS package that defines the translations and rotations needed to get from a source frame to a target frame.

[0100] • Fruitlet Detection Module 300

[0101] With respect to Figure 14, the fruitlet detection module first fetches the image data acquired by the data acquisition module to the system and performs image pre-processing, like downsampling, resizing, normalising, etc. Next, the pre- processed image was fed to a CNN model that was trained to detect loose fruitlets from the image. The fruitlet detection module divided the camera view into different regions. The ‘f ruitlet-in’ event happens when the first detected fruitlet enters the Rol, while the ‘fruitlet-out’ event happens when the last detected fruitlet exits the Rol. The module starts to count the fruitlet when the ‘f ruitlet-in’ event happens and stops counting when the ‘fruitlet-out’ event happens. The fruitlets count was then determined by computing the average number of fruitlets per frame that were present in the Rol within the ‘fruitlet-in’ and ‘fruitlet-out’ events. If the number of loose fruitlets counted is greater than the threshold number of loose fruitlets for classification as harvest-ready, the system will then estimate the position of the detected fruitlets with respect to the robot.

[0102] The position of the detected fruitlets was estimated as a ‘group’ (the system considers all the detected fruitlets as a grouped object) through grid analysis from the camera view with a static transform function to the respective position with respect to the robot’s current position. The estimated position of the grouped fruitlet object can be transformed with a static transform function because it doesn’t need to be very accurate, as it just needs to provide a rough location to be matched to the nearest tree position in order to tell that this groups of detected fruitlets belong to that specific tree. Similar to tree detection, a transformation function was applied to transform the computed position to the global coordinate (i.e., with respect to the semantic map). Since the tree’s harvest readiness status will be determined by the number of fruitlets scattered around the tree, the fruitlet detection and positioning result published by this module will be subscribed to by the environment representation module for the tree’s harvest readiness classification. • Environment Representation Module 400

[0103] The environment representation module is an extended real-time map generation system built on the SLAM system. In the present invention, an environment representation module is developed on top of the SLAM system from the Cartographer project. Cartographer is a SLAM system developed by Google and open-sourced with the Apache License Version 2.0. It is primarily designed for robots and autonomous vehicles to enable these machines to navigate and understand their surroundings in both 2D and 3D environments. Cartographers are widely used in the robotics community and have applications in various fields, including self-driving cars, drones, and mobile robotics. This sophisticated system is subdivided into two main components: Local SLAM and Global SLAM.

[0104] Local SLAM’s primary objective is to create highly detailed maps and precisely determine a robot’s position within a localised environment. It relies on data from two-dimensional and three-dimensional LiDAR and, in some cases, incorporates information from additional sensors like IMUs (Inertial Measurement Units) and odometry to enhance localisation accuracy. Leveraging loop closure detection algorithms, Local SLAM identifies previously visited locations, which helps rectify any accumulated mapping and pose estimation errors. It generates a detailed and up-to-date map of the immediate surroundings, ensuring that robots can operate safely and effectively. This invention constructs the map using a two- dimensional LiDAR with the CeresScanMatcher strategy. This strategy takes the initial guess as prior and finds the best spot where the scan match fits the submap by interpolating the submap and sub-pixel aligning the scan.

[0105] In contrast, Global SLAM takes on the larger-scale challenge of creating a consistent map of the entire environment by fusing information from various local maps. Maintaining map consistency becomes crucial as robots venture over greater distances or revisit areas. Global SLAM achieves this through loop closure detection, pose graph optimisation, and map stitching techniques. Loop closure detection identifies and reconciles previously visited locations, while pose graph optimisation refines the alignment between local maps to ensure they fit seamlessly together. The result is a unified global map that accurately represents the entire environment despite the challenges posed by robot movement and exploration. Additionally, integrating the Global Navigation Satellite System (GNSS) information into the system can provide a global position reference to help align local maps correctly within the larger global map. GNSS data can serve as a key constraint in the optimisation process during map merging and pose graph optimisation, ensuring that the entire map is geographically coherent.

[0106] The environment representation module subscripted to the result from the tree detection module, the fruitlet detection module, and the Cartographer’s SLAM in order to construct the final semantic map that represents the plantation area and generate the markers on the map that represent the trees in the plantation area. The changes in the markers on the map, which include updating the markers’ position and status (i.e., shape or colour) or initialising a new marker on the map, will only be realised after the map is refreshed. The environment representation workflow was triggered when the map needed to be refreshed. Users can configure the refresh frequency depending on the platform specification and application requirements. The higher the refresh frequency rate, the more computational resources are consumed.

[0107] The environment representation module consists of a submodule to track the changes of the markers and realise the changes on the map when the map is being refreshed. This submodule was named the “chain-based tree position tracking and re-mapping module”. This submodule was developed to improve the existing tracking algorithm by introducing a tracking Rol to realise the computational resources demanded to recognise the changes when refreshing the map. The submodule was named “chain-based” as the position of all the markers published on the map will be registered and chained to a list by this submodule. This submodule tracks only the markers that fall in the tracking Rol instead of monitoring all the markers in the list. In other words, only the markers that fall within the tracking Rol will be passed to the submodule for tracking and realising the changes when refreshing the map, while the remaining markers that are not in the tracking Rol will be treated as no changes and the submodule will not track and realise the changes of these markers. Since only a small subset of markers in the list are required to be tracked during the refresh of the map, the computational cost to refresh the map could be greatly reduced as compared to existing tracking algorithms, where all the markers were passed to the algorithm for tracking and realising the changes whenever the map needed to be refreshed, especially when dealing with a huge set of markers. Besides, the chain-based tree position tracking and re-mapping module also introduced a backward tracking methodology when the system needed to update the markers that didn’t fall within the current tracking Rol. Initially, the tracking Rol will be applied to the region centred on the robot’s current position. The reason for doing this is that the system is running the SLAM in real-time, and the region centred by the robot’s current position, i.e., the area scanned by the 2D LiDAR for the local SLAM algorithm to construct the map, will be updated from time to time. Furthermore, since the environment representation module is listening to the tree detection module and the fruitlet detection module for generating the markers, and both the tree detection and fruitlet detection are performed in real- time as well, the newly generated markers will always fall within the region within the camera view, which is also within the covering range of the LiDAR sensor. Thus, tracking the region centred by the robot’s current position will cover most cases where the published markers need to be updated and realised on the map.

[0108] However, there is also a change in that the markers outside the tracking Rol needed to be updated. For example, when the loop closure event happened, the position of the published markers needed to be updated to comply with the SLAM map changes. In this case, the backward tracking methodology will be applied. The backward tracking rules defined that the tracking Rol will move backwards from the newest published markers (closer to the robot’s current position) to the oldest published markers until no changes on the markers need to be realised. This can ensure that the tracking can only be applied to the selected region where some changes in the markers need to be realised. Upon the change on the map, the algorithm will first track and update the position of the markers within the tracking Rol. After that, it will move the Rol in the backward direction of the chain and update the corresponding markers. The algorithm will stop the position tracking and updating process once a marker’s position in the chain is unaffected by the map’s latest changes. As the tracking Rol only tracks until the point where no other changes need to be realised, this ensures no wastage of the computational resource to track and re-realise the published marker that no changes need to be realised.

[0109] Next, for all the published tree positions, the module will apply a clustering function based on DBSCAN to filter out the noise and cluster the tree that has a close position. DBSCAN, which stands for Density-Based Spatial Clustering of Applications with Noise, is a clustering algorithm used to identify clusters in a dataset by assessing the density distribution of data points, grouping those that are close to each other, and considering them as part of the same cluster. DBSCAN distinguishes between core points (data points with a sufficient number of neighbours within a specified radius), border points (points within the radius of a core point but with fewer neighbours), and noise points (isolated points). It does not require the pre-specification of the number of clusters and is robust to outliers, making it a valuable tool for identifying clusters of varying shapes and sizes in diverse applications like spatial data analysis and anomaly detection. In this case, as the tree detection module performs tree detection in real-time with the continuously streaming video from the camera, technically, there will always be more than one point for a detected tree, i.e., the tree was detected on each frame streamed from the camera. Therefore, the DBSCAN clustering and noise filtering algorithm was applied to cluster the points that belong to the same tree and filtered out the outlier point (if any) to compute the final position of the detected tree. In this way, since the final position of the tree was computed based on multiple points from multiple frames, the errors that occurred in a single frame that led to the errors in the final published result can be minimised.

[0110] Furthermore, after performing the clustering and noise filtering, the module will also compare the clustered tree position with all the existing tree positions to check whether a tree at a similar position has been published before. This additional checking process prevents duplicated detection of the same tree when the robot revisits the same tree or detects the same tree from another angle. If the tree were found to have existed on the map, instead of assigning the new tree ID to the newly detected tree instance, the system would assign the previous tree ID of the tree that the system found to have existed on the map. Or else, if the tree is found to be a new tree, the system will assign a new tree ID to it. Lastly, the module also consists of a tree harvest-readiness status tracking module that tracks the existing loose fruitlet around the tree to determine whether the tree is ready to be harvested. If loose fruitlets were found around the tree, the markers that represent the tree would be changed to green, indicating the tree’s status to be “harvest-ready”.

[0111] On the other hand, as mentioned in the previous paragraph, the environment representation module subscripted to the result from the tree detection module, fruitlet detection module, and Cartographer’s SLAM, and all these three modules operate in real-time. However, the environment representation workflow was triggered at a certain frequency for the chain-based tree position tracking and re-mapping module to consume the data from the tree detection module, the fruitlet detection module, and the Cartographer’s SLAM to track the changes and realise the markers on the generated map. Thus, the data published to the environment representation module needed to be queued on a ‘waiting list’ while waiting to be consumed by the workflow. The environment representation module creates three waiting lists to store the results:

[0112] (i) Tree Waiting List: To store the result published by the tree detection module. The tree waiting list subscriber subscribes to the output from tree detection, which is a position point of the detected tree, and appends it to a ‘tree waiting list’. The information in this list will be used to compute the global position of the tree on the map.

[0113] (ii) Fruitlet Waiting List: To store the results published by the fruitlet detection module. The fruitlet waiting list subscriber subscribes to the output from fruitlet detection, which is a position point of the fruitlet that is required to be matched to the tree to classify it as harvest-ready and append it to a ‘fruitlet waiting list’. The information in this list will be used to determine the status (harvest-readiness) of the targeted object (tree) on the map.

[0114] (iii) Tree Tracking List: To store the targeted object (tree)’s position published by the Cartographer. The tree tracking list subscriber subscribes to the output from the map builder class of the Cartographer, which consists of the tracking position of the published tree that is used to compute the offset of the published position upon changes in the semantic map and append it to a ‘tree tracking list’. At first, the global position of the published tree was computed by consuming the data in the Tree Waiting List, and it should be the same as the position data published by the Cartographer. However, the position value of this list will become different when there are changes in the map, like loop closure. In this case, the targeted object (tree) ’s position published by the Cartographer will be different so that the chain-based tree position tracking and re-mapping module can use this to track whether there is a change in the position of the marker needs to be realised when moving the tracking Rol in the backward direction on the chain. Besides the above three input lists, the environment representation module also creates an output list, Tree Marker List, to store the published markers in the ‘First In Last Out’ method. The Tree Marker List also served as the previously mentioned chained list that the chain-based tree position tracking and re-mapping module iterates when performing backward tracking.

[0115] • Visualisation Module 500

[0116] Finally, the users can visualise the camera view, semantic map, and the position and status of the detected tree on RViz. Figure 16 shows the sample user interface to visualise the results. It consists of two camera views for the video streamed from the left and right alvium camera. Next, the overall 2D map with the markers that indicate the harvest-readiness of the detected trees was provided in the centre. The output of the extracted submap, together with the detected circle / semicircle shape, was provided at the bottom right corner.

[0117] Experimental Result / Data

[0118] The system has undergone a validation test in a real-world plantation to verify whether the system’s performance meets the project’s requirements. The robot will be mounted on a wheelchair and manually moved around in the test area by the tester during the validation test.

[0119] The validation test will verify the performance of the system in these aspects:

[0120] 1 . Generate a virtual map to represent the given test area. The system needs to close the loop when returning to the starting point.

[0121] 2. Detect and track the trees in the given test area. The system needs to correctly detect all the trees in the given area and track their identity using different tree IDs.

[0122] 3. Tag the trees in the given area on the virtual map. The system needs to merge the repeated detection with the previous detection when detecting a tree from different sides or when the robot revisits the same tree. 4. Determine which tree in the given test area is ready to be harvested by counting the loose fruitlets on the ground around the tree and indicating it on the virtual map. The system needs to correctly indicate the harvestready status of all the trees in the given test area.

[0123] The validation test took place on a plantation in Sungai Pelek, Selangor. An L-shape region comprising 31 valid trees was selected as the test area. Figure 17 shows the satellite view of the test area from Google Maps. Besides, as the number of “harvest-ready” trees was insufficient in the given test area during the validation test, a certain number of mock “harvest-ready” trees were added randomly. The tester will manually place the loose fruitlet on the ground around the selected tree. Thus, the “harvest-ready” tree in this validation test will include both the actual “harvest-ready” trees and the mock “harvest-ready” tree.

[0124] Table 2. Ground Truth data of the trees in the test location

[0125] Figure 19 shows the sample output semantic map generated by the system. The semantic map consists of the markers that represent the position of the tree on the map as well as the harvest-readiness status of each tree. The harvest-ready tree will be marked with green colour.

[0126] Table 3. Configuration for tree detection

[0127] Table 4. Definition of each result category for tree detection

[0128] Table 5. Ground Truth data of the trees in the test location

[0129] Table 6. Configuration for fruitlet detection

[0130] Table 7. Definition of each result category for harvest-readiness determination

[0131] Table 8. Ground Truth data of the trees in the test location

[0132] It is preferred that the mobility platform 103 comprises an aerial map of the plantation that can be used for an initial travelling path on the ground. This enables platform operators to track the path on which the mobility platform 103 is travelling. The said aerial map can be obtained by way of a fixed-wing aerial vehicle, such as a drone. The aerial map may be overlaid with a graphical rendition of the geographic area of the plantation to form a composite map.

[0133] As used herein, the term “module” may refer to, be part of, or include an Application Specific Integrated Circuit (ASIC), an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor or a distributed network of processors (shared, dedicated, or grouped) and storage in networked clusters or data centres that executes code or a process, other suitable components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term “module” may also include memory (shared, dedicated, or grouped) that stores code executed by the one or more processors.

[0134] The term code, as used above, may include software, firmware, byte-code and / or microcode and may refer to programs, routines, functions, classes, and / or objects. The term shared, as used above, means that some or all code from multiple modules may be executed using a single (shared) processor. In addition, some or all code from multiple modules may be stored by a single (shared) memory. The term group, as used above, means that some or all code from a single module may be executed using a group of processors. In addition, some or all code from a single module may be stored using a group of memories.

[0135] The techniques described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on a non-transitory tangible computer-readable medium. The computer programs may also include stored data. Non-limiting examples of the non-transitory tangible computer- readable medium are non-volatile memory, magnetic storage, and optical storage.

[0136] Some portions of the above description present the techniques described herein in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. These operations, while described functionally or logically, are understood to be implemented by computer programs. Furthermore, it has also proven convenient at times to refer to these arrangements of operations as modules or by functional names without loss of generality.

[0137] Unless specifically stated otherwise as apparent from the above discussion, it is appreciated that throughout the description, discussions utilising terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0138] Certain aspects of the described techniques include process steps and instructions described herein in the form of an algorithm. It should be noted that the described process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.

[0139] The present disclosure also relates to a system for performing the operations herein. This system may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a tangible computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, readonly memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

[0140] Although an overview of the inventive subject matter has been described with reference to specific example embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of embodiments of the present disclosure. Such embodiments of the inventive subject matter may be referred to herein, individually or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single disclosure or inventive concept if more than one is, in fact, disclosed.

[0141] The embodiments illustrated herein are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. The Detailed Description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

[0142] As used herein, the term “or” may be construed in either an inclusive or exclusive sense. Moreover, plural instances may be provided for resources, operations, or structures described herein as a single instance. Additionally, boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and particular operations are illustrated in a context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within a scope of various embodiments of the present disclosure. In general, structures and functionality presented as separate resources in the example configurations may be implemented as a combined structure or resource. Similarly, structures and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within a scope of embodiments of the present disclosure as represented by the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

[0143] The foregoing description, for the purpose of explanation, has been described with reference to specific example embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the possible example embodiments to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The example embodiments were chosen and described in order to best explain the principles involved and their practical applications, to thereby enable others skilled in the art to best utilise the various example embodiments with various modifications as are suited to the particular use contemplated.

[0144] It will also be understood that, although the terms “first”, “second”, and so forth may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the present example embodiments. The first contact and the second contact are both contacts, but they are not the same contact. The terminology used in the description of the example embodiments herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used in the description of the example embodiments and the appended examples, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and / or “comprising” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0145] As used herein, the term “if” may be construed to mean “when” or “upon” or “in response to determining” or “in response to detecting” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] is detected” may be construed to mean “upon determining” or “in response to determining” or “upon detecting [the stated condition or event]” or “in response to detecting [the stated condition or event]” depending on the context.

Claims

CLAIMS1 . A system for monitoring a palm tree and an oil palm fruitlet thereof in a plantation in a lightweight and efficient manner, characterised in that, the system comprising: a data acquisition module (100) comprising a remote sensing device (101 ) and an image-capturing device (102) mountable to a mobility platform (103) configured for generating an occupancy grid map associated with the palm tree in the plantation and an image associated with the oil palm fruitlet scattered within a vicinity of the palm tree, respectively; a tree detection module (200) configured for generating a submap from the occupancy grid map to detect the palm tree therein and determine a tree position of the palm tree detected thereof, which is transformed to a global tree position on a semantic or global map prepared from the occupancy grid map; a fruitlet detection module (300) configured for detecting the oil palm fruitlet from the image, a fruitlet position, and a quantity of oil palm fruitlets using a grid analysis, wherein the fruitlet detection module (300) transforms the fruitlet position to a global fruitlet position to associate the same with the global tree position of the palm tree detected thereof; and an environment representation module (400) configured for comparing the tree position against a tree tracking list using a chain-based tree-tracking and re-mapping algorithm, and for assigning a harvest-ready status to the palm tree based on the quantity of oil palm fruitlets detected thereof, wherein the environment representation module (400) clusters the tree position using a density-based spatial clustering of applications with noise, DBSCAN, algorithm and initiates noise filtering for processing the palm tree detected thereof with a tree marker list.

2. The system according to Claim 1 , wherein the tree detection module (200), upon detection of the palm tree, computes a tree score for the palm tree detected thereof and compares the same against a threshold score.

3. The system according to Claim 1 , wherein the tree detection module (200) determines the tree position with respect to the mobility platform (103) prior to the transformation to the global tree position.

4. The system according to Claim 1 , wherein the fruitlet detection module (300), with respect to the grid analysis, employs a grid with a region of interest, through which the quantity of oil palm fruitlets is determined.

5. The system according to Claim 1 , wherein the fruitlet detection module (300) determines the fruitlet position with respect to the mobility platform (103) prior to the transformation to the global fruitlet position.

6. The system according to Claim 1 , wherein the image includes a still image, a set of still images, a moving image, or any combinations thereof.

7. The system according to Claim 1 , wherein the image-capturing device (102) includes a set of image-capturing devices mountable to opposing sides of the mobility platform (103) thereof.

8. The system according to Claim 1 , wherein the remote sensing device (101 ) includes a non-image-based depth sensing device mountable to an elevated position relative to the image-capturing device (102).

9. The system according to Claim 1 further comprising a visualisation module (500) configured for displaying data including the occupancy grid map, the submap, the semantic or global map, the image, the palm tree with the tree position including the global tree position, the oil palm fruitlet with the fruitlet position including the global fruitlet position, the tree marker list, the tree tracking list, the harvest-ready status, and camera views from the image-capturing device (102) thereof.

10. A method of monitoring a palm tree and an oil palm fruitlet thereof in a plantation in a lightweight and efficient manner, characterised in that, the method comprising the steps: providing a data acquisition module (100) comprising a remote sensing device (101 ) and an image-capturing device (102) mountable to a mobility platform (103) (S100); generating, by the remote sensing device (101 ), an occupancy grid map associated with the palm tree in the plantation (S101 ); generating, by the image-capturing device (102), an image associated with the oil palm fruitlet scattered within a vicinity of the palm tree (S102);generating, by a tree detection module (200), a submap from the occupancy grid map to detect the palm tree therein (S103); determining, by the tree detection module (200), a tree position of the palm tree detected thereof and transforming the same to a global tree position on a semantic or global map prepared from the occupancy grid map (S104); detecting, by a fruitlet detection module (300), the oil palm fruitlet from the image, a fruitlet position, and a quantity of oil palm fruitlets using a grid analysis (S105); transforming, by the fruitlet detection module (300), the fruitlet position to a global fruitlet position to associate the same with the global tree position of the palm tree detected thereof (S106); comparing, by an environment representation module (400), the tree position against a tree tracking list using a chain-based tree-tracking and remapping algorithm (S107); assigning, by the environment representation module (400), a harvest-ready status to the palm tree based on the quantity of oil palm fruitlets detected thereof (S108); clustering, by the environment representation module (400), the tree position using a density-based spatial clustering of applications with noise, DBSCAN, algorithm (S109); and initiating, by the environment representation module (400), noise filtering for processing the palm tree detected thereof with a tree marker list (S110).

Images