Plant detection and display system

Abstract

The plant detection and display system acquires images of a plurality of plants from an image acquisition device, performs a detection process on the acquired images using a trained model to classify each of the plurality of plants into a first category indicating a defective plant whose quality score is below a predetermined quality threshold or a second category indicating a normal plant, determines location information from the acquired images for each of the plurality of plants classified into the first category, uses the location information to generate a map of the plurality of plants classified into the first category including identifiers around each of the plurality of plants, and provides the generated map to a terminal device for output on a display device that visually displays the location of each plant classified as a defective plant in the first category.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority from U.S. Provisional Patent Application No. 63 / 123,786, filed December 10, 2020, which is incorporated by reference in its entirety.

[0002] [Field] The present disclosure relates generally to image processing, and more specifically, to plant detection and identification in vertical farming environments. [Background technology]

[0003] Vertical farming is a method of growing plants in multiple trays stacked vertically to maximize space and increase the amount of plants that can be grown in a given area. They are typically grown in carefully controlled environmental conditions to maximize yields. However, like traditional agriculture, plants require inspection to ensure they are healthy and disease-free. This manual visual inspection is time-consuming and labor-intensive, as it requires a human to analyze the plants in each tray. More specifically, it requires a human operator to physically analyze the growing trays to identify diseased or "bad" plants (or plant leaves) and monitor the changing health of the plants in the trays. This workflow is difficult to scale, as growing more plants also requires hiring more operators to inspect and track the health and status of the plants.

[0004] In a vertical farming environment, human inspection presents unique challenges as the structures that hold the trays tend to extend upward at heights that make direct human inspection difficult. One attempt to solve these shortcomings in monitoring and maintaining plant health in vertical farming involves the mounting and placement of image capture devices. This solution is a slight improvement over manual inspection as video can be captured to show a live feed of the plants in the various trays, and inspection is now done remotely and can be more easily performed on every tray. However, this solution still requires extensive human inspection to monitor and determine the health of the plants. Furthermore, plant health monitoring, when done remotely, requires a human user to go and find where and exactly which potentially defective leaves are displayed. This is problematic as it can be difficult to find where a particular growing tray is located within an array of vertically arranged growing trays, and where exactly the indicated leaves are on a particular growing tray.

[0005] Further attempts to address these shortcomings have utilized image processing techniques to perform object detection and identify diseased plants using machine learning. While these solutions automate the inspection process, object detection performed using machine learning still results in erroneous decisions (e.g., false negative and false positive results) and the results still require human verification to determine if the object detection is correct. More specifically, machine learning has limitations that result in mistakes that require a human operator or reviewer to review and monitor the machine learning results for false positives and false negatives. Summary of the Invention

[0006] The above drawbacks are remedied by a disclosed plant detection and display system, which includes a server with at least one memory storing instructions and at least one processor, the at least one processor being configured to execute the stored instructions to acquire images of a plurality of plants from an image acquisition device, perform a detection process on the acquired images using a trained model for classifying each of the plurality of plants into a first category indicative of a defective plant having a quality score below a predetermined quality threshold or a second category indicative of a normal plant, where the trained model is a machine learning model trained using one or more visual characteristics associated with the plurality of plants indicative of a defective plant, determine location information for each of the plurality of plants classified into the first category from the acquired images, utilize the location information to generate a map of the plurality of plants classified into the first category including identifiers around each of the plurality of plants, and provide the generated map to a terminal device for output on a display device providing a visual representation of the location of each plant classified into the first category as a defective plant.

[0007] According to another embodiment, the plant detection and display system is embodied in a vertical farming system including one or more growing structures, each of the growing structures including one or more trays in which a plurality of plants are grown, each of the one or more growing trays including an image capture device that moves over and captures images of each growing tray, and a server. The server includes at least one memory storing instructions and at least one processor, and when the at least one processor executes the stored instructions, it is configured to acquire images of a plurality of plants in each of the one or more cultivation trays from each image acquisition device of the cultivation structure, and perform a detection process on the acquired images using the learned model to classify, for each of the one or more cultivation trays, each of the plurality of plants into a first category indicative of a defective plant having a quality score below a predetermined quality threshold or into a second category indicative of a normal plant, where the learned model is a machine learning model trained using one or more visual characteristics associated with the plurality of plants indicative of a defective plant, from the acquired images, determine location information for each of the plurality of plants classified into the first category, utilize the location information to generate a map of the plurality of plants classified into the first category including identifiers in the vicinity of each of the plurality of plants, and provide the generated map to the terminal device for output on a display device that provides a visual representation of the location of each plant classified as a defective plant in the first category.

[0008] These and other objects, features, and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure, taken in conjunction with the accompanying drawings and the appended claims. [Brief description of the drawings]

[0009] [Figure 1] 1 shows a plant detection and display system.

[0010] [Figure 2A] 1 shows aspects of the cultivation structure at the growing site. [Figure 2B] 1 shows aspects of the cultivation structure at the growing site. [Figure 2C] 1 shows aspects of the cultivation structure at the growing site.

[0011] [Diagram 3] FIG. 2 is a block diagram detailing the hardware components of the server.

[0012] [Figure 4] FIG. 11 is a block diagram detailing modules representing computer-executable instructions executed by the server.

[0013] [Diagram 5] FIG. 1 is a block diagram showing the structure of a machine learning model.

[0014] [Figure 6] 1 is a graphical user interface output by the detection and display system.

[0015] [Figure 7] 1 is a graphical user interface output by the detection and display system used to provide feedback based on detection results.

[0016] [Figure 8] 1 is a graphical user interface output by the detection and display system used to provide feedback based on detection results.

[0017] [Figure 9] 4 is a flow diagram detailing an algorithm for controlling the display of a graphical user interface on a mobile computing device.

[0018] [Figure 10A] 1 illustrates a process for acquiring and detecting images at a growing site. [Figure 10B]1 illustrates a process for acquiring and detecting images at a growing site.

[0019] [Figure 11A] 1 shows a process for acquiring and detecting images at a growing site and displaying the same. [Figure 11B] 1 shows a process for acquiring and detecting images at a growing site and displaying the same.

[0020] [Figure 12A] 1 shows a process for acquiring and detecting images at a growing site and displaying the same. [Figure 12B] 1 shows a process for acquiring and detecting images at a growing site and displaying the same.

[0021] [Figure 13] 1 illustrates an embodiment for scaling a captured image with detection results for display on a mobile computing device. [Figure 14] 1 illustrates an embodiment for scaling a captured image with detection results for display on a mobile computing device. [Figure 15] 1 illustrates an embodiment for scaling a captured image with detection results for display on a mobile computing device. [Figure 16] 1 illustrates an embodiment for scaling a captured image with detection results for display on a mobile computing device.

[0022] [Figure 17] 1 shows a robotic device that can be controlled based on the results of a detection system.

[0023] [Figure 18] 1 is a flow diagram illustrating an analysis algorithm that uses the results of the detection system.

[0024] [Figure 19] 19 is a graph showing an example output of the analysis algorithm of FIG. 18.

[0025] Throughout the drawings, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components, or portions of the illustrated embodiments. Moreover, while the present disclosure will now be described in detail with reference to the drawings, it is done so in connection with the exemplary embodiments for purposes of illustration. It is intended that changes and modifications can be made to the exemplary embodiments described without departing from the true scope and spirit of the present disclosure as defined by the appended claims. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the following exemplary embodiments are merely examples for implementing the present disclosure, and may be appropriately modified or changed depending on the individual configuration of the device to which the present disclosure is applied and various conditions. Therefore, the present disclosure is not limited to the following exemplary embodiments, and according to the drawings and embodiments described below, the embodiments described below can be applied / implemented in situations other than those described below as examples.

[0027] Described herein is a system for analyzing and detecting defective plants during the growing process. In one embodiment, the system described herein is deployed in a vertical farming environment in which a series of plants are grown in multiple vertically stacked growing trays. Plants are living, constantly changing objects that must meet certain quality checks before they can be harvested and sold. Currently, this quality check is performed by human workers who must look at each leaf to identify where issues such as mold or disease are occurring. This means that growing facilities must spend additional money to analyze and monitor the health of the plants as they scale up. The system according to the present disclosure improves its ability to identify both damaged and defective leaves within a tray of multiple plants with multiple leaves by using a machine learning model that has an improved ability to identify which of these leaves are defective. In this way, the system advantageously performs plant health tracking over a period of time while accounting for environmental factors present in the growing facility. By continuously capturing images of the growing trays over this period of time and analyzing them with the trained machine learning model, diseased or defective leaves within a particular growing tray can be quickly and accurately identified, allowing them to be more quickly cut from the lot of healthy plants growing therein. In addition to quickly and accurately identifying diseased or defective leaves, the system advantageously identifies the exact location of leaves in a particular tray that need to be removed. The system generates an augmented reality view of a particular tray on a terminal device (e.g., mobile phone, tablet, etc.) and displays therein an identifier indicating which of the leaves should be removed. Due to the sensitivity of the trained model, defective leaves may not appear to be defective and a significant amount of time may be required to analyze all leaves in a particular tray to identify which leaves need to be removed. The augmented reality view provided on the terminal device instantly displays the location of the defective leaves in the tray so that they can be removed.In another embodiment, the annotated images showing the location of the defective leaves can be provided as a map to a robotic device that can be used to remove the defective leaves without the individual having to go to the growing rack and manually remove the identified defective leaves. Thus, the present disclosure improves the field of vertical farming by providing improved and more sensitive trained machine learning models that analyze captured images of the growing trays and improve how individuals or other systems are notified of the exact location of the defective leaves so that the leaves can be removed from the growing environment by human or mechanical means.

[0028] FIG. 1 illustrates an exemplary plant detection and display system 10. The system 10 includes at least one growing location 11 including a plurality of growing trays 12 in which plants or other vegetation are grown. In one embodiment, the growing trays are stacked vertically in one or more growing tray support racks to maximize the growing area, as shown in FIGS. 2A-2C below. Each growing tray is monitored, individually and collectively, by an image capture device 14. In one embodiment, each individual growing tray includes a respective image capture device that captures a series of images of the plants in the growing tray, which images can be input to and analyzed by a machine learning model trained to detect defects in the plants being grown. The growing location 11 also includes one or more sensors 16 positioned throughout the growing location to sense and monitor one or more growing characteristics of the growing location 11 at a given time. In one embodiment, the one or more sensors 16 include one or more of a temperature sensor (a) for measuring the temperature throughout the growing area 11, a moisture sensor (b) for measuring the moisture content of the growing area 11, a soil sensor (c) for measuring one or more properties of the soil at the growing area, and a pH sensor (d) for measuring the pH at the growing area. In another embodiment, the one or more sensors may further include a plant height sensor for measuring the height of the plants, which may be realized via a laser or ultrasonic sensor or an infrared sensor, a humidity sensor for measuring the humidity of the air, a light sensor for measuring foot candles and / or lux, a temperature sensor for water, an electrical conductivity sensor for water, a water flow sensor for measuring the flow of water through pipes at various locations, an air flow sensor for measuring the air flow at various locations, and a spectrometer for measuring the spectral signal at the growing area. The one or more sensors 16 may be positioned to monitor the entire growing area 11 and / or may be associated with a specific tray of plants at the growing area 11 to more accurately monitor the properties that affect the growth of the plants.

[0029] One or more servers 20 are provided and include one or more processors that execute one or more programs stored thereon to control the growing environment. This may include, for example, a program with computer executable instructions to control any characteristic at the growing site, including but not limited to light, heat, and water provided to allow plants to grow in the growing trays. Additionally, the server may receive inputs from any sensors 16 and images captured by the image capture device 14, and may use these inputs to make decisions regarding the quality of the leaves of a particular growing tray and to derive correlations between sensed environmental characteristics and defective leaves in a particular growing tray 12. Details regarding the detection and analysis of captured images are described in more detail below. It should be understood that the figure is an illustration of a server 20.

[0030] The network 50 facilitates a connection between one or more servers 20 at the growing location 11 and one or more terminal devices such that the terminal devices can receive data communicated by the one or more servers 20 over the network 50 to generate various types of graphical user interfaces that can represent the data received from the one or more servers 20. The network 50 may be any suitable network. For example, one or more portions of the network 50 may include an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more thereof. The network 50 may include one or more interconnected networks. In operation, the network 50 may be any network that enables two-way communication between one or more computing devices.

[0031] As shown, the network 50 connects a first terminal device 60 and a second terminal device 70 to one or more servers 20. In one embodiment, the first terminal device 60 and the second terminal device 70 are different types of computing devices. For example, the first terminal device 60 is a computing device such as a laptop, while the second terminal device 70 is a mobile computing device such as a smartphone. In these instances, as described below, the first terminal device 60 may be used during a model training phase by a training module running on the server 20, which displays and annotates images captured by the one or more image capture devices 14 to generate a training set of images with labels indicating which leaves in the captured images are identified as damaged.

[0032] Once the model training is complete, future images of the growing tray captured by one or more image capture devices 14 are input to the trained model for analysis and classification. The trained model generates a map of the growing tray 12 being analyzed to classify individual leaves of the plant as "defective" or "non-defective" and indicate the location of each leaf classified as "defective." In one embodiment, the second terminal device 70 may be selectively positioned near the growing tray 12 and may have images of the growing tray 12 captured by the second terminal device displayed thereon, augmented with one or more identifiers overlaid on the captured image to indicate defective leaves. This overlay is generated based on the map generated during the classification operation so that defective leaves can be removed.

[0033] 2A-2C show an exemplary structure at the growing location 11. In FIG. 2A, a support structure 200 for multiple growing trays 12a-12c is provided. Desirably, each growing location includes multiple support structures 200, each supporting multiple growing trays 12, to maximize yields by providing a larger growing area obtained from vertically stacked growing trays 12. The support structure 200 includes at least one growing area 210. As shown in FIG. 2A, the support structure 200 includes three growing areas, indicated by reference numbers 210a-210c. It should be noted that the support structure 200 may, and preferably does, include more than two growing areas 210 supporting individual growing trays 12. The diagrams provided herein are for illustrative purposes only and to facilitate understanding of the described operations. Each of the cultivation areas 210a-210c includes a lower support 212a-212c, a separation section 214a-214c, and an upper support 216a-216c. As shown here, only the upper support 216a is visible in FIG. 2A. However, it should be understood that the cultivation areas 210b and 210c include the upper support 216b and 216c, respectively. In certain embodiments, the lower support 212 may also function as the upper support 216 for the cultivation area located toward the ground surface. The lower support 212a-212c are configured to support the cultivation trays 12a-12c, respectively. The separators 214a-c are configured to separate the lower supports 212a-c and the upper supports 216a-c to allow space between the growing trays so that growing devices such as watering devices, lights, etc. can be configured according to the requirements of the plants being grown in the growing trays 12a-c, which may or may not be of the same plant variety. The upper supports 216a-c include a cross support 218. As shown in FIG. 2A, only the cross support 218a is visible, but it should be understood that each of the growing areas 210b and 210c also includes a respective cross support 218b and 218c.The cross support 218 includes a carriage 220 that supports at least one image capture device 222, shown in FIG. 2B.

[0034] Referring to FIG. 2B, a cross section of a growing area 210a is shown. Each growing area at a particular location has a unique growing area identifier whose value is communicated to the server 20 of FIG. 1 when data of a commission regarding the plants in the particular growing area is communicated to the server 20, as described below. As seen here, a portion of a lower support 212a is depicted, which supports a growing tray 12a in which a number of plants are growing. Each growing tray 12 includes a tray identifier that uniquely identifies the particular growing tray 12 in the particular growing area 210. The tray identifier is obtained and also communicated to the server 20 when data regarding the plants in the particular growing tray 12 is communicated to the server 20, as described below.

[0035] A portion of the upper support 216a is disposed above the lower support 212a and the growing tray 12a and is supported by a separator 214a (shown in FIG. 2A but omitted here). The upper support 216 visualizes a number of tray markers 224a-d at predefined positions within the growing tray 12, as shown in FIG. 2C by circles with different types of hatching. The various tray markers 224a-d are disposed at known distances from each other and from the edge of the growing tray 12a. In one embodiment, each set of tray markers 224a-d is disposed along the same line across the growing tray 12a. As shown, each set 224a-d includes two tray markers, but this is for illustrative purposes only, and each set of tray markers 224a-d can include any number of tray markers. Additionally, while only four sets of tray markers are shown, it should be understood that any number of sets of tray markers 224a-d can be arranged depending on the size of the growing area and / or growing tray. In certain embodiments, each individual tray marker within a respective set of tray markers includes a secondary indicator that precisely identifies a particular area of ​​the growing tray 12a. In other embodiments, the tray markers are not provided in sets, and each tray marker is uniquely identifiable when acquired and indicates a particular area within the growing tray.

[0036] The carriage 220 is disposed on the cross support 218 and supports one or more image capture devices 222 thereon, as shown in FIG. 2A. The one or more image capture devices 222 supported by the carriage 220 are the same as the image capture devices 14, which are generally described as part of the system and shown in FIG. 1. The cross support 218 is movable in the x-direction along the length of the upper support 216a such that the one or more image capture devices 222 supported by the carriage 220 are positioned to capture images along the entire surface of the growing tray 12a. In one embodiment, the carriage is controlled to move at predetermined times and remain at various positions along the length of the upper support for predetermined periods of time such that one or more images (e.g., still images or video) of the plants in the growing tray 12a are captured. The image data also captures any tray markers 224 present within the field of view. When the image is analyzed, the tray indicator advantageously provides a precise location of the image within the growing tray 12, since the distance of individual plant leaves can be determined based on the distance from one or more particular tray markers and the outer edge of the growing tray 12a. Data obtained from the images captured by the one or more image capture devices 222 is transmitted to the server 20 of FIG.

[0037] Figure 3 illustrates the hardware components of the server 20 shown in Figure 1. The server 20 is illustrated in Figure 1 as being located at the growing location 11. However, it should be understood that the server 20 need not be physically located at the growing location 11. Rather, the server 20 may be connected to image capture devices and sensors at the growing location via the communications network 50 of Figure 1, and the server 20 may be, for example, a cloud server or other remotely accessible server configured to perform the operations described in this disclosure.

[0038] The server 20 includes one or more CPUs 302 configured to control the overall operation of the server 20 by using a set of stored instructions and / or one or more computer programs in conjunction with data stored in a RAM 304 or a ROM 306. The one or more CPUs may include one or more dedicated hardware or a graphics processing unit (GPU) other than the CPU 302, and the GPU or dedicated hardware may perform some of the processes performed by the CPU 302. Examples of dedicated hardware include an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), etc. The RAM 304 temporarily stores computer programs and data read from the ROM 306, data provided from the outside via a communication interface 310, and / or data read from the storage 312. The ROM 306 stores programs and data that do not need to be modified. The ROM includes one or more control programs that control the operation of the server 20.

[0039] The input / output unit 308 includes one or more input components, including but not limited to a joystick, a jog dial, a touch panel, a keyboard, a mouse, etc., and receives user operations and inputs various commands to the CPU 302. The input / output unit 308 includes an output component including a display and / or an output port for enabling connection to an external display device so that a graphical user interface can be displayed thereon. Furthermore, the output component may include an audio output device such as a speaker. The communication interface 310 communicates with external devices such as a PC, a smartphone, a camera, a sensor, etc. The communication with the external devices may be performed by wire using a local area network (LAN) cable, a serial digital interface (SDI) cable, a wireless connection, etc., or may be performed wirelessly via an antenna. The communication interface 310 is configured to communicate using a standard communication protocol such as WIFI, Bluetooth, NFC, etc.

[0040] Server 20 also includes storage 312, which may be either a hard disk drive, a solid state storage device, or the like, and may be internal to server 20 or may be a removable storage device. Storage 312 includes a number of control programs, which are applications that may be executed by one or more CPUs 302 to perform certain types of operations. The control programs stored in storage 312 include computer-executable instructions that are loaded into memory and configure one or more CPUs to perform certain operations. Storage 312 also allows for program data and other data in which operations are performed, as shown in FIG. 4.

[0041] 4 is a block diagram illustrating each program and data object stored in storage 312. For purposes of illustration, storage 312 includes an application area 420 that stores application programs comprised of computer-executable instructions that configure one or more CPUs 302 of server 20 to perform various operations. Storage 312 also includes a data area 130 that stores object data that, when executed, may be used by one or more applications stored in application area 420.

[0042] The application area 420 includes a training module 422, a detection module 424, and a display control module 426. The training module 422 is a set of instructions that, when executed, configures one or more processors to generate training data used to train a machine learning model. This includes acquiring a series of images of plants growing in various growing trays and labeling some of the acquired images to indicate which plants or leaves of the plants are defective as determined by one or more plant characteristics such as shape, color, size, surface characteristics, etc. The acquired images annotated by a user using the training module can be stored in an image repository 436 in a data area 430 of the storage 312. The training module 422 provides the labeled training data to a training model, which is a machine learning model trained using the training data to identify which leaves are defective from the acquired images of leaves. The training module 422 validates the training such that the classification of leaves as defects has an accuracy score of at least 80% when compared to a dataset labeled by humans. Once training is complete, the trained model 434 is stored in the data area 430 and used by the detection module 424.

[0043] The machine learning model trained via training module 422 is a convolutional neural network (CNN). An advantage of using a CNN for a system according to the present disclosure is the ability to automatically detect important features of the images used to train the model based on labeled images that indicate that the leaves (or plants) are defective. Furthermore, the computational efficiency provided by a CNN makes it particularly ideal for processing data to classify images of plant leaves as defective or not defective. This is particularly important due to the commonality between objects (e.g., leaves) in the acquired images. The ability to identify patterns and specific features of images labeled as defective improves the functionality of the model once trained, which ultimately improves the detection performed on unlabeled image data.

[0044] An exemplary structure of the CNN model 500 is shown in FIG. 5. The CNN model 500 shown in FIG. 5 was designed with a focus on speed of processing images without compromising accuracy. More specifically, the CNN model 500 was designed to identify multiple object characteristics including, but not limited to, color, surface pattern, edges, size, and leaf shape. The feature maps of the larger convolutional layers focus on color, edges, shape, and size, while the deeper layers focus more intensively on the surface details of the leaves. Although the focus of the deeper layers is shown to be on the surface details, these deeper layers also analyze the image for all features, and at a more refined size than the global and previously placed layers. The number of convolutional layers is minimized to place the emphasis primarily and more heavily on the leaf size, which may enable faster training times and provide higher accuracy. The activation layers are standard rectified linear units because they are more computationally efficient and do not resemble sigmoid curve activation layers. Additionally, hyperparameters were tuned with a particular focus on properties related to intersection over union (IOU), false negative / false positive tolerance, and batch sizing. IOU is particularly relevant for leaf detection because the leaves are all very close to each other and many overlap. This can result in training locations having overlapping ground truth, but only one of the overlapping leaves will be detected. As a result, IOU-related hyperparameters were tuned by increasing the IOU overlap parameter so that boundary locations are highly overlapping.

[0045] Model 500 includes an initial convolutional layer 502 where an image is input, analyzes the entire layer of the image (e.g., RGB), and outputs a tensor with dimensions of 1x8K neurons across the layer. From there, the tensor flows through multiple bottleneck layers 504a-e, which are shortcut convolutional layers that reduce the overall number of nodes required to process the image and also reduce the degradation of data at each successive layer. At each layer, the tensor input to it is uncompressed, filtered, and recompressed before exiting to the subsequent layer.

[0046] Bottleneck layers are used to reduce computation needs and reduce the size of data in the network. Each bottleneck has different manually defined factors for expansion factor, repetition factor that sets the number of times the layer is repeated, output channels, and stride. For example, the number of input channels at the first bottleneck layer is less than the input channels at the subsequent bottleneck layers. The expansion factor at each bottleneck determines when data computations are performed, including how tensors are expanded and recompressed. During analysis, the parameters that define the stride at each layer indicate the size and movement of the filter across the image input to the layer to perform the analysis.

[0047] The current CNN model 500 analyzes the shape of the leaves contained in the acquired images at a macro level. For other features, analysis at the macro scale as well as at the micro scale and identifying features from the images labeled as ground truth is important to better detect defective leaves present in the subsequent unlabeled image data. Therefore, the model needs to be flexible to obtain information at the macro and micro levels. To achieve this, the analysis starts using a large filter, but soon moves to a smaller size filter to extract smaller details to determine and identify where one leaf ends and another begins. The smaller filter also allows for identifying other features of the leaves such as color, edges, surface properties, etc.

[0048] After flowing through bottleneck layers 504a-e, the tensors are output to a pointwise convolutional layer 506 to analyze each point in the image, and the results flow to a pooling layer 508 to reduce the resolution of the feature maps. A final pointwise convolutional network is placed downstream of the pooling layer 508 to apply a final compression and output a binary classification as defective or non-defective leaves.

[0049] In an exemplary operation, the training module 422 is executed by one or more CPUs 302 of the server and receives images of growing trays at a growing location. The received images may be transferred to the image repository 436. The training module 422 generates a graphical user interface (GUI) that includes the acquired images of the growing trays. The GUI includes a labeling function that allows a human user to manually annotate the received images of the growing trays with ground truth identifiers located above leaves that are visually identified as defective. The positions of the ground truth indicators are defined at specific x and y coordinates in the image, and also include information identifying the height and width of the ground truth indicators. The labeling information identifying the respective positions and sizes of each ground truth box is output to a data file (e.g., an extensible markup language (XML) file) and, together with the original images, are associated with each other as a "labeled dataset."

[0050] An exemplary GUI generated by the training module and output to a display device is shown in FIG. 6. There, an image of a plant in a growing tray is displayed within the GUI, and a user can use an input device, such as a mouse or touch screen, to review the image and manually identify individual leaves that are defective. Upon receiving user input via the GUI, the training module 422 identifies location information associated with each ground truth indicator entered by the user (visually represented by multiple boxes in FIG. 6). The training module 422 further outputs ground truth identifier size information representing the size of the ground truth identifier entered by the user. The location information and ground truth size information are output to a data file having a predefined format (e.g., XML). The training module 422 associates the data file with the original image as a labeled dataset. This process is repeated until a sufficient number of images and associated data files are completed.

[0051] After generating a sufficient number of images and associated data files, the training module 422 trains a machine learning model as described above in Figure 5. After training for a period of time, the training can be stopped and the model 500 can be frozen and saved in storage 312 as a trained model 434, which can later be used to analyze unlabeled image data to identify defective leaves.

[0052] The detection module 424 requires that there is a completed trained model 434 stored in the storage 312. When a detection operation needs to be performed, the detection module 424 acquires raw image data from one or more image capture devices that acquire images of the respective growing trays. As used herein, the term raw image data refers to images that contain a plurality of objects that are to be classified into at least two categories, and that are not labeled such that there is no indication of whether the objects in the acquired images belong to one of the at least two categories. The acquired images are provided to the trained model 434 to determine the category to which each object in the acquired images belongs. The acquired images are provided to the trained model 434 to determine the category to which each object in the acquired images belongs. In operation, the objects in the acquired images are plant leaves, and the trained model analyzes each raw image input thereto to classify the plant leaves as defective or non-defective. While this classification is taking place, the detection module 424 generates a GUI that includes the objects in the images that the model has determined to be "defective." An exemplary GUI output by the detection module is shown in FIG. 7. The GUI generates annotated images with classifiers in the form of boxes that overlay the raw image to help the user visualize which plant leaves are defective. As shown in Figure 7, the classifiers are bounding boxes and include a label indicating a confidence score that the determined classification is accurate.

[0053] Execution of the detection module 424 causes the server 20 to acquire an image from an image acquisition device which the server 20 transfers and stores as raw image data in an image repository 436 in the storage 312. Also stored in association with the raw image data is the location where the image was acquired. The detection module 424 provides the stored image data to a trained model 434 which uses it to perform object detection on the raw image and produce an inference as to which category the detected object belongs to (e.g., defective or non-defective). The resulting object detections and inferences are output on the image displayed in a GUI (e.g., FIG. 7) in the form of boxes with associated positions and sizes, each showing the detected results of a plant or leaf. These detections also have a confidence percentage, with 100% being the highest confidence indicating that the object in the image has been correctly classified as a defect. The user can set a threshold for what percentage is valid for a confident detection, and if the user selects a threshold of 80%, all detections below 80% will not be classified as defects. One confidence threshold defines the percentage at which a detection must be declared a positive detection and is used during training. This confidence threshold cannot be changed during or after training and is a static preset value. After training, there is also a confidence threshold setting, which can be changed, and does the same for inference of the learned model. In this embodiment, this confidence threshold is changeable. The classification made by the detection module is output as a detection data file that contains the location of each box based on x and y coordinates, along with size information of each box, indicating the height and width of each box in pixels. As shown in FIG. 7, a new image file is generated in parallel with the generation of a text file that is the original image analyzed by drawing static boxes on the confidence score image as shown above.

[0054] In one embodiment, the detection module 424 performs detection on the raw image data and then performs feedback functionality on the classified image as shown in FIG. 7 and FIG. 8. During the feedback functionality, the detection module can retrieve the original raw image data and a detection data file containing the location of the classifiers and the size of the classifiers from the image repository. The GUI uses the data of the location and size of the classifiers to overlay the classifiers on the raw image data. Unlike the image shown in FIG. 7, in this embodiment, the classifiers are generated for user selection. An exemplary GUI for performing the feedback functionality is shown in FIG. 8. The GUI is generated based on the detection data file containing the x, y coordinates indicating the location of the classifiers determined during the object detection operation and the width and height of the box. The GUI then generates these boxes representing the classifiers at their respective locations and sizes on top of the original image. The boxes are not embedded or permanently drawn on top of the original image. The boxes are selectable by the user to delete boxes that the user determines to be inappropriate classifications. The user can select one or more boxes for deletion. The user can also "add" boxes by selecting a location and drawing a bounding box around one or more objects (e.g., leaves) that are not classified as defects but that the user believes should have been classified as defects. When the user confirms the image, the feedback function ends. When one or more classifier boxes are deleted, an updated detection data file is generated by copying the original detection data file, deleting the location and size information of the deleted classifier boxes, and adding the location and size information of the newly added classifier boxes. The updated detection data file can be saved in association with the original image and added to the labeled dataset used to initially train the learned model 434. This can provide updated learning data for subsequent retraining of the model. In one embodiment, the updated detection data file is in XML format and stored in the image repository 436 along with its corresponding original image file on which the feedback operation was performed.These can be added to the original "labeled dataset" that was originally used to train the object detection model. This "updated labeled dataset" can have multiple generated XML detection files and original images added before being retrained. In one embodiment, the user can choose when to start retraining the model, which can be done by running the retrain function of the training module 422. In another embodiment, a threshold for new images and / or updated detection data files can be set, and when this threshold is reached, the training module 422 will automatically retrain the model using the updated labeled dataset. This allows the output of the object detection model to be reviewed and adjusted by a human, and re-input to the model as an updated labeled dataset that adjusts the model's accuracy to what the human wants to detect and what he or she does not want to detect. During the retraining operation, the training model 432, which is in the same state as the learned model 434 after the initial training, is accessed by the training module 422, and the updated labeled dataset is input to the training model for subsequent training. After this is done, the updated learned model 434 is saved in storage and used for subsequent object detection operations.

[0055] In another embodiment, after obtaining the detection results, the display control module 426 can be executed to perform an augmented reality display on a terminal device such as the mobile phone 70 shown in FIG. 1. In this embodiment, the mobile phone 70 includes hardware similar to that of the server 20 shown in FIG. 3, plus certain additional elements specific to a mobile computing device. More specifically, as is known, the mobile computing device may be a smartphone that includes input and output components of one or more mobile capture devices, such as front and rear cameras that can be operated to capture images. Additionally, the mobile computing device includes one or more touch screen displays that can selectively display the display information and receive input from a user to perform control operations to control the mobile computing device. This exemplary post-detection process is described with reference to FIGS. 9-16.

[0056] A flow diagram detailing an exemplary algorithm for generating an augmented display is shown in Figure 9. In an exemplary operation, the mobile computing device includes a corresponding display control module stored in its storage, which is separately executed by one or more processors of the mobile computing device to perform the operations described herein. In another embodiment, the mobile computing device includes a terminal application that, when executed, establishes communication between the mobile computing device and the server 20 over a communication network, such as the network 50 shown in Figure 1. In an embodiment in which the application running on the mobile computing device is a terminal application, the server's display control module 426 receives data acquired by the mobile computing device, performs the described processing, and sends the processed information back to the mobile computing device for display on the touch screen of the mobile computing device.

[0057] As shown in Figure 9, in block 902, the detection information determined by the detection module 424 is provided to the mobile computing device along with plant location information from the tray database 438 in Figure 4. From this information, the mobile computing device can use to augment an image of the growing tray currently being captured by the image capture device of the mobile computing device, such that the augmented display more clearly shows the exact location of one or more plant leaves classified as defective in the currently captured image of the growing tray.

[0058] In block 904, an augmented display application is executed on the mobile computing device to dynamically display detection results associated with the particular growing tray. As a result of blocks 902 and 904, the mobile computing device obtains object detection results and identifies leaves whose location on the known object (tray) is classified as defective. In block 906, the mobile computing device initiates image acquisition using one or more image acquisition devices of the mobile computing device. The image acquisition operation acquires one or more images of the particular growing tray containing the plurality of plants that have undergone object detection, and the live images being acquired by the mobile computing device are displayed on the display, as shown in block 908. Upon displaying the images, two processing operations occur substantially in parallel. The first processing operation identifies the particular object in the displayed image that is classified as shown in blocks 910-914, and the position and orientation of the mobile computing device are determined in blocks 916-918. The results of both of these processing operations are used to generate the augmented display, as described in block 920.

[0059] Turning first to determining position and orientation, the position of a mobile computing device with a camera is collected or determined, at block 916. The position of the mobile computing device can be collected in many different ways, such as a global positioning system (GPS), a compass, an inertial measurement unit (IMU), a quick response (QR) code orientation using a camera or a reference point marker, an augmented reality (AR) location link, or any other method that defines the user's position relative to the surrounding environment. Once the position of the mobile computing device is determined, at block 918, the orientation of the mobile computing device relative to the growing tray is determined. The position and orientation information is provided to a display control module and used with the detection information to generate an augmented image for display on the mobile computing device, as described below.

[0060] Returning to block 908, the image being acquired by the mobile computing device is also analyzed for tray markers or other indicators that can be used to identify the growing tray or a portion thereof being acquired at the current time in block 910. If no markers are found as shown in block 911, a prompt may be provided to the user via a display on the mobile computing device suggesting moving the mobile computing device to acquire other areas of the growing tray and identify the growing tray information from within the image being acquired. In the event that the image currently being acquired includes one or more tray markers or indicators that identify a particular growing tray as determined in block 912, the location of the mobile computing device is determined in block 914 and the image acquired by the mobile computing device is overlaid with an indication of the location of the plants classified as defective according to the detection results obtained in block 902.

[0061] This image overlay and user position determination is repeated continuously because the images being captured are live view images and the location of the mobile computing device is also constantly changing. Since the object detection positions and sizes of the detected objects and the user's position are known, the display control module 426 can generate an augmented reality view by overlaying the object detection positions on the live view images being captured by the image capture device of the mobile computing device.

[0062] Based on the object detection results generated upon completion of the detection module process 424, the location of the growing tray is known and the location of the leaves classified as defective is known and captured in the detection result data file. By using the X, Y location of the acquired image and the detection results, the location of all the leaves classified as defective for a particular tray is precisely defined. Furthermore, a Z region including the two-dimensional region of X, Y can be determined. This allows the display control module 426 to generate a graphical user interface (GUI) that allows a view of the classified leaves from a plane (e.g., the X, Z plane shown in Figures 10-12) by displaying where the classified leaves can be found using the X, Y coordinates and the Z region. As previously described in Figures 6-8, in addition to overlaying an identifier identifying the location of the leaves in the live view image classified as defective, a confidence detection value associated with each classified leaf may be displayed near the object indicating the location of the defective leaves on the mobile device.

[0063] In another embodiment, a threshold value indicating whether an object has been detected and classified as being a defect may be set via input received at the mobile computing device while the augmented live view image is being displayed. If the threshold value is changed via the screen of the mobile device, the display of the object indicating the location of the pale leaf is updated. If the user changes the threshold value from a first threshold value to a second threshold value via the screen of the mobile computing device, the second threshold value is transmitted from the mobile computing device to the server 20, the mobile computing device receives location information from the server 20 of classified objects having a confidence value equal to or greater than the second threshold value, and the augmented display is updated to display only objects (e.g., leaves) classified as defects having a confidence value equal to or greater than the second threshold value.

[0064] 10-16 further illustrate how blocks 908-920 in FIG. 9 are performed. FIGS. 10A and 10B illustrate the image acquisition and object detection process performed to obtain the detection result provided to the mobile device in block 902, as described in FIG. 9. FIGS. 10A and 10B include similar structures as described in FIG. 2 and need not be further defined. Rather, the following description focuses on the operations for acquiring an image of a particular growing tray and a plant growing therein. FIGS. 10A and 10B illustrate a side view of an XZ direction of a particular growing tray 1002 provided at a growing location. The growing tray 1002 includes one or more reference marks 1004a-b in an area containing a plant, which are used for calibration and uniquely identifying an area or location in the growing tray 1002.

[0065] An image acquisition structure 1010 is provided in the Z direction above the growing tray 12. The image acquisition structure 1010 includes a carriage 1014 with a camera (not shown). In one embodiment, the image acquisition structure is a rail system placed above the growing tray to acquire a top view of the plants in the growing tray 12. The carriage 1014 is movable in the X direction as shown. The image acquisition structure 1010 includes one or more camera position identifiers 1012a and 1012b that are used to track the position of the camera on the carriage 101 during image acquisition. In one embodiment, the one or more camera position identifiers are RFID tags provided at predefined positions on the rails and are read by a reader located on the carriage as the carriage moves along the rails during the image acquisition operation. In FIG. 10B, the carriage 1014 stops at the position where the camera position identifier 1012b is provided and the image acquisition operation begins. As shown here, the location of the camera position identifier 1012b defines an image capture field, indicated by the arrow labeled 1016, that includes the area of ​​the growing tray 1002 including the tray markers 1004c and 1004d. The location information from both the camera position indicator 1012b and the tray markers 1004c and 1004d is provided to the server 20 along with the captured images for detection processing performed by the detection module 424. Thus, not only can the detection module 424 identify which leaves are defective, but the detection module 424 can also use the camera position identifier 1012 to identify where on the carriage 1014 each image was captured by the camera since this location is known. Finally, the AI ​​model can determine where the pale leaves are located.

[0066] 11A and 11B show the location of the pale leaf, which is one way in which the detection module 424 determines that the leaf 1100 in the growing tray 1002 is defective and the area 1102 captured by the image capture device of the carriage 1014. FIG. 11A is a view in the Z direction, and FIG. 11B is a view in the Y direction. In this example, the defective leaf 1100 is located between the tray marker 1004c and the tray marker 1004d, both of which represent reference marks in the tray. After the detection module 424 determines the location of the defective leaf 1100, the user can use the mobile computing device to locate the exact location of the defective leaf 1100, as shown in FIG. 12A and FIG. 12B. The image capture device on the mobile computing device is initiated. The location information determined during the detection operation 424 to identify the defective leaf is transmitted to the mobile computing device 1202. Because the image capture device of the mobile computing device 1202 is positioned facing the side of the growing tray 1002, the image capture operation captures tray marker 1004c and tray marker 1004d. From this, an application running on the mobile computing device determines the location of the defective leaves 1100 shown in FIG. 12A. A guided GUI is displayed to help the user see the location of the defective leaves 1100 in the growing tray 1002. This is possible because the location of the tray markers is known and the XY distance in pixels from the tray markers in the captured image can be obtained and used to locate the defective leaves 1100, thus locating each leaf classified as defective.

[0067] In another embodiment, when the orientation of the mobile device is changed from a horizontal orientation to a vertical orientation, the GUI displayed on the mobile device may be changed from an image acquired from above the cultivation tray 1002 (FIG. 12A) to an image acquired from the side of the cultivation tray 1002 (FIG. 12B). When the orientation of the mobile device is changed from a vertical orientation to a horizontal orientation, the image displayed on the mobile device may be changed from an image acquired from the side of the cultivation tray 1002 (FIG. 12B) to an image acquired from above the cultivation tray 1002 (FIG. 12A). The orientation of the mobile device can be detected by a tilt sensor mounted on the mobile device.

[0068] 13-16 illustrate various methods for converting an image captured by an image capture device on the carriage 1014 and subjected to object detection into a GUI on the display of a mobile computing device that includes an augmented display showing an image of a growing tray with indicators identifying one or more leaves that have been determined to be defective.

[0069] In FIG. 13, a captured image of a plant tray is shown, including arrows showing how the captured image is transformed for display on a mobile computing device with an image capture device capturing a live view of the growing tray. The original captured image is subjected to detection using the trained model and includes the location and size of each leaf classified as defective as included in the detection data file with xy position values ​​and hw size values ​​associated with each classification. In this embodiment, the live view image being captured is of an area of ​​the growing tray. Reference points on the actual plant tray have known positions and distances from each other, allowing the captured image to be transformed to the area the mobile device is looking at in order to correctly overlay the detections contained in that area.

[0070] 14 illustrates another embodiment in which a live view image being acquired by a mobile computing device includes an object classified as a defect known to be at an x,y pixel location on the image, the object having a defined width and height in pixels, the known width and height. In this embodiment, where the reference point is not in view of the acquired image, a set or determined location for the acquired image can be established from the reference point, allowing the mobile device with a camera to accurately display a generated overlay identifying the location in the image with the object classified as a defect.

[0071] FIG. 15 illustrates another embodiment where a live view image with a known width and height in pixels is known where objects classified as defects are located at x,y pixel locations on the image, and the classified objects have a defined width and height in pixels. In this embodiment, the reference markers are within the area of ​​the captured image, so generating the detection area can be done by mapping the captured area to the live view area of ​​the mobile computing device. This is further illustrated in FIG. 16, which shows a captured image with a detection process, a growing tray from which the image was captured, and a mobile computing device with an image capturer having a live view of the growing tray. As previously described, the original captured image has been subjected to a detection process by the detection module 424 and includes detection data identifying the location and size of each object in the image classified as a defect. The captured image is of an area of ​​the plant tray. The reference points on the actual growing tray have known locations and distances from each other and are within the view of the original captured image. This allows pixel locations of objects classified as defects to be mapped to actual plant tray locations and allows the GUI to be augmented with a live view image of the growing tray by overlaying on the live view image an indicator identifying the location within the growing tray of leaves classified as defects.

[0072] In another embodiment, the results of the detection module processing can be used as input to control a robotic device that automatically picks or removes leaves classified as defective. An exemplary robotic device 1700 is shown in FIG. 17. The robotic device 1700 includes a support structure 1702 that is disposed on a moving device 1704. The moving device 1704 is configured to move the growing tray 1701 below the support structure 1702 such that the growing tray 1701 is positioned at a predefined location where the robotic device can remove the defective leaves from within the growing tray 1701. To do this, the support structure includes a crossbar 1710 that supports a cutting device 1712. The crossbar 1710 is configured to move in a direction indicated by the arrow labeled 1711. The cutting device 1712 is configured to move in a direction indicated by the arrow labeled 1713 and toward and away from the growing tray 1701 disposed below the support structure. In operation, the movement and position of the crossbar 1710 and the cutting device 1712 are determined based on the detection data file generated upon completion of the detection module 424. Since the particular growing tray on which the detection was performed is known, the robotic device can identify the growing tray currently positioned under the support device based on one or more tray identifiers included in the detection result data file. Furthermore, the movement of the robotic device can be controlled based on the detection position that allows the image to indicate a point of interest in the growing tray under the support structure to the robotic controller that controls the movement of the crossbar 1710 and the cutting device 1712 to move to the point of interest. Upon reaching the point of interest, the robotic controller signals the cutting device to move in a direction toward the growing tray 1701 to grab or otherwise cut off the leaf classified as defective. The movement control is performed by the robotic controller that maps the image to real world coordinates that serve as input for the robotic device to move as described above. In operation, if the cutting device 1712 was to move to a pixel position in the image as defined in the detection data file, it will move to the corresponding position in the real world.Using this method, the detection location can be operated by passing the detection center point coordinates (x+width / 2, y+height / 2) to the robot either as a pixel location, G-code, or another format that the robotic device can operate in. As a result, the processing performed by the detection module 424 can result in faster and more accurate removal of defective objects from a particular growing tray.

[0073] and combining information regarding the number and location of defective plants in one or more growing trays with information obtained from environmental sensors at the growing location to track, quantify, and qualify the condition of the plants in one or more trays at the growing location. This process is performed by the analysis module 428 in Figure 4 being loaded into memory, which configures one or more processors of the server 20 to perform the analysis process. Figure 18 illustrates the operations performed by the analysis module 428 when executed.

[0074] In blocks 1802-1808, images of leaves in one or more growing trays are acquired and analyzed using the learned model during a detection process performed by the detection module 424, producing detection results and image data that are stored in the tray database 438 of FIG. 4 and shown in block 1820 in FIG. 18. These have been described above and do not need to be discussed again and are incorporated herein by reference. Furthermore, blocks 1812-1816 represent a feedback process performed by the detection module 424 and have been described above with respect to FIGS. 6-8. Details of this feedback for updating the detection results for further model training purposes and / or providing a more accurate set of detection results also do not need to be discussed further and are incorporated herein by reference. The detection results stored in block 1820 are provided to the analysis module 428 together with environmental information obtained from one or more sensors 1840 that sense information about the environment at the growing location 1830. The information obtained in block 1840 includes information obtained by one or more sensors 16 representing any one or more of the following sensors: (a) temperature sensors measuring the temperature of the entire growing area 11, (b) moisture sensors measuring the amount of moisture at the growing area 11, (c) soil sensors measuring one or more properties of the soil at the growing area, and (d) pH sensors measuring the pH at the growing area. The environmental information obtained in block 1840 is stored in the tray database 438 on a tray-specific basis identifying the environmental conditions at a particular tray, a rack-specific basis identifying the environmental conditions across all trays within a particular support structure, an area-specific basis indicating the environmental conditions across a particular area within the growing area, and a location-specific basis indicating the environmental conditions at the growing area globally.

[0075] Information from the environmental conditions database 1830 and the detection results in the tray database 1820 are provided as inputs to an analytics module 428 running on the server. The analytics module 428 can perform predictions, monitoring, historical tracking, alerts, and growing trends at growing locations. The resulting text files from the object detection models can be used to quantify and qualify the condition or health and how the condition and health is changing by creating a database of results for plant growth at various tray locations and tracking object detection results across tray locations. The ability to identify defective leaves on a growing tray allows the ability to count / track the number of defective leaves on that tray and also allows the ability to compare the number of defective leaves to historical growing conditions (temperature, humidity, pH, etc...) and track the change in the number of defective leaves from day to day. The results of this analytics process can generate a graph, as shown in FIG. 19, that identifies the quality of plants growing at a particular location over a series of growing days, determined based on the number of leaves classified as defective on each growing day. In one embodiment, this is calculated by the total number of leaves on a tray minus the number of abnormal / defective leaves detected on that tray divided by the total number of leaves on that tray to give a percentage quality score, which is expressed as ((total leaves-defective leaves) / total leaves)=% quality score.

[0076] The detection and display system according to the present disclosure advantageously automatically identifies the condition of the plant leaves using a trained machine learning object detection model that operates on images containing multiple plant leaves and can cover a large number of leaves very quickly and successfully distinguish individual leaves from each other. Furthermore, based on the detection and classification of leaves as defective, the system can be used to track the quality of all plants on the growing tray, give a quality score and generate historical quality data. The results of the detection can be used as a direct input to control an automation solution that now allows picking and sorting to be done by robotic devices. Furthermore, the detection and display system can convert and augment the live view image of a particular growing tray with identifiers such as box / coordinates / location of all leaves classified as defective.

[0077] Aspects of the present disclosure may also be implemented by a computer of a system or apparatus (or a device such as a CPU, a microprocessing unit (MPU) or the like) that reads and executes a program recorded in a memory device to perform the functions of the above-described embodiments, and may also be implemented by a method in which the steps are performed, for example, by a computer of the system or apparatus that reads and executes a program recorded in a memory device to perform the functions of the above-described embodiments. For this purpose, the program is provided to the computer, for example, via a network or from various recording media (e.g., non-transitory computer-readable media) that function as storage devices.

[0078] Although the present disclosure has been described with reference to the exemplary embodiments, it should be understood that the present disclosure is not limited to the disclosed exemplary embodiments.

Claims

1. A server comprising at least one memory for storing instructions and at least one processor, The at least one processor, when executing the stored instructions, acquiring images of a plurality of plants growing in at least one growing tray from an image capture device; performing a detection process on the acquired images of the plants growing in the at least one growing tray using a trained model for classifying each of the plurality of plants into a first category indicative of a defective plant or a second category indicative of a normal plant, the trained model being a machine learning model trained using one or more visual characteristics of leaves of defective plants; determining, from the acquired images, position information within at least one growing tray for each of the plurality of plants classified into the first category, the position information being determined using one or more tray markers associated with the at least one growing tray and captured when the images of the plurality of plants were acquired; utilizing the one or more tray markers in the location information to generate a map of the plurality of plants classified in the first category including an identifier around each of the plurality of plants classified in the first category; providing the generated map and a live view image of the at least one growing tray captured by the image capture device to an application running on a terminal device to generate an augmented reality view displayed on a display of the terminal device, the augmented reality view including the live view image of the plurality of plants on the at least one growing tray and the identifiers providing a visual representation of a location of each plant classified into the first category as a defective plant. The server is configured as follows:

2. The detection process includes a first analysis and a second analysis, the first analysis identifying leaf shapes in the images of the plurality of plants, and the second analysis identifying defective leaves. The server of claim 1 .

3. Execution of the instructions comprises: receiving a request from the terminal device to change each plant classified in the first category to be classified in the second category; updating the generated map based on the received change requests; Using the updated map, each plant that has been changed from the first category to the second category is prevented from being identified in the image. The server of claim 1 , further configuring the at least one processor to:

4. Execution of the instructions comprises: storing the updated map in association with the images acquired as modified image data in the at least one memory, the updated map including plants classified into the first category using the trained model and plants modified by a user; Providing the modified image data to a training module used to generate the trained model to generate an updated trained model. The server of claim 3 , further configuring the at least one processor to:

5. Execution of the instructions comprises: Using the generated map to control a picker to cause the picker to remove each of the plants classified into the first category using the location information within the map. The server of claim 1 , further configuring the at least one processor to:

6. Execution of the instructions comprises: continuously acquiring images of a plurality of plants from the image acquisition device over a certain period of time; For each of the continuously acquired images, a detection process is performed using the trained model to classify each of the plurality of plants into the first category indicating a defective plant having a quality score below a predetermined quality threshold or into a second category indicating a normal plant, the trained model being a machine learning model trained using one or more visual features associated with the plurality of plants indicating a defective plant; generating a quality score representative of the plurality of plants based on a number of each of the plurality of plants classified into the first category; Using the quality scores to control one or more parameters used to cultivate the plurality of plants. The server of claim 1 , further configuring the at least one processor to:

7. The one or more tray markers are located within each of the at least one growing tray. The server of claim 1 .

8. At least one growing tray in the acquired image is supported by a support structure. The server of claim 1 .

9. 1. A method for classifying plants, comprising the steps of: acquiring images of a plurality of plants growing in at least one growing tray from an image capture device; performing, by at least one processor, a detection process on the acquired images of the plurality of plants growing in the at least one growing tray using a trained model for classifying each of the plurality of plants into a first category indicative of a defective plant or a second category indicative of a normal plant, the trained model being a machine learning model trained using one or more visual features of leaves of defective plants; determining, by the at least one processor, position information within at least one growing tray for each of the plurality of plants classified into the first category from the acquired images, the position information being determined using one or more tray markers associated with the at least one growing tray and captured when the images of the plurality of plants were acquired; using the one or more tray markers in the location information to generate a map of the plurality of plants classified in the first category including an identifier around each of the plurality of plants classified in the first category; providing the generated map and a live view image of the at least one growing tray being captured by the image capture device to an application running on a terminal device, generating an augmented reality view to be displayed on a display of the terminal device, the augmented reality view including the live view image of the plurality of plants on the at least one growing tray and the identifier providing a visual representation of a location of each plant classified as a defective plant; A method comprising:

10. The detection process includes a first analysis and a second analysis, the first analysis identifying leaf shapes in the images of the plurality of plants, and the second analysis identifying defective leaves.

10. The method of claim 9.

11. receiving a request from the terminal device to change each plant classified in the first category to be classified in the second category; updating the generated map based on the received change requests; and using the updated map to prevent each plant that has been changed from the first category to the second category from being identified in the image; and The method of claim 9 further comprising:

12. storing, in at least one memory, the updated map in association with the captured image as modified image data, the updated map including plants classified into the first category using the trained model and plants modified by a user; providing the modified image data to a training module used in generating the trained model to generate an updated trained model; The method of claim 11 further comprising:

13. controlling a picker using the generated map to cause the picker to remove each of the plants classified in the first category using the location information within the map. The method of claim 9 further comprising:

14. continuously acquiring images of a plurality of plants from the image acquisition device over a period of time; performing a detection process for each of the continuously acquired images using the trained model to classify each of the plurality of plants into the first category indicative of a defective plant having a quality score below a predetermined quality threshold or into a second category indicative of a normal plant, the trained model being a machine learning model trained using one or more visual features associated with the plurality of plants indicative of a defective plant; generating a quality score representative of the plurality of plants based on a number of each of the plurality of plants classified into the first category; using the quality scores to control one or more parameters used to cultivate the plurality of plants; The method of claim 9 further comprising:

15. A recording medium for storing instructions, When the instructions are executed by at least one processor, the at least one processor acquiring images of a plurality of plants growing in at least one growing tray from an image capture device; performing a detection process on the acquired images of the plurality of plants growing in the at least one growing tray using a trained model for classifying each of the plurality of plants into a first category indicative of a defective plant or a second category indicative of a normal plant, the trained model being a machine learning model trained using one or more visual features of leaves of defective plants; determining, from the acquired images, position information within at least one growing tray for each of the plurality of plants classified into the first category, the position information being determined using one or more tray markers associated with the at least one growing tray and captured when the images of the plurality of plants were acquired; using the one or more tray markers in the location information to generate a map of the plurality of plants classified in the first category including an identifier around each of the plurality of plants classified in the first category; providing the generated map and a live view image of the at least one growing tray being captured by the image capture device to an application running on a terminal device, generating an augmented reality view to be displayed on a display of the terminal device, the augmented reality view including the live view image of the plurality of plants on the at least one growing tray and the identifier providing a visual representation of a location of each plant classified as a defective plant; A recording medium for executing the above.

Images