User platform and method

Abstract

The user platform comprises an input unit that receives one or more inputs from a user, a display, and a processor operably connected to the input unit and the display. The processor is configured or programmed to control the display to display suggested cut points for crops, and the processor is configured or programmed to verify that the suggested cut points are properly positioned, delete the suggested cut points, or modify the suggested cut points based on one or more inputs received by the input unit.

Description

Technical Field

[0001] Cross - References to Related Applications This application claims the benefit of priority of U.S. Provisional Application No. 63 / 447,471, filed Feb. 22, 2023. The entire content of this application is incorporated herein by reference.

[0002] Background 1. Field of the Invention The present invention relates to an agricultural cutting system and a method for generating an agricultural cutting point.

Background Art

[0003] 2. Description of Related Art Conventionally, cutting operations in agriculture have been manual, which are expensive and time - consuming. For example, when the target crop is a grapevine, in the agricultural cutting operation of pruning the grapevine, a skilled person has to walk through the vineyard and manually prune the grapevine. Furthermore, since the techniques for pruning grapevines can vary from person to person, it may reduce the reliability and consistency of grapevine pruning. Such low reliability and lack of consistency are undesirable because they may have an adverse effect on the health and growth of the grapevine and the quality of the grapes produced from the grapevine.

Summary of the Invention

[0004] For the reasons described above, there is a need for an agricultural cutting system and a method for generating an agricultural cutting point that can be used cheaply and reliably to perform a cutting operation on a target crop and generate a cutting point on the target crop.

Problems to be Solved by the Invention

[0005] Summary of the Invention Preferred embodiments of the present invention relate to an agricultural cutting system and a method for generating an agricultural cutting point.

Means for Solving the Problems

[0006] A method according to a preferred embodiment of the present invention includes generating suggested cut points for a crop using a processor, displaying the suggested cut points on a user platform, and receiving, via the user platform, at least one input for confirming that the suggested cut points are properly positioned, for deleting the suggested cut points, and for modifying the suggested cut points.

[0007] In a method according to a preferred embodiment of the present invention, the method further includes, after receiving the input to modify the proposed cut point, moving the proposed cut point to a new desired location using the user platform.

[0008] In a method according to a preferred embodiment of the present invention, the method further includes displaying the proposed cutting point on the user platform after the proposed cutting point has been moved to the new desired position, and receiving input via the user platform confirming that the proposed cutting point is properly positioned at the new desired position.

[0009] In a method according to a preferred embodiment of the present invention, the method further includes receiving user comments via the user platform regarding the reason why the proposed cutting point was moved to the new desired position.

[0010] In a method according to a preferred embodiment of the present invention, generating the suggested cutting points for the crop includes generating a plurality of suggested cutting points for the crop, and displaying the suggested cutting points on the user platform includes displaying the plurality of suggested cutting points on the user platform, wherein for each of the plurality of suggested cutting points for the crop, at least one of the inputs for confirming that the suggested cutting point is properly positioned, for deleting the suggested cutting point, and for modifying the suggested cutting point is received via the user platform.

[0011] In a preferred embodiment of the present invention, the method further includes receiving input via the user platform that confirms that each of the plurality of proposed breakpoints is acceptable to the user.

[0012] In a preferred embodiment of the present invention, the method further includes detecting one or more agricultural features of the crop using an object detection model, and displaying the one or more agricultural features detected using the object detection model on the user platform when the proposed cut point is displayed on the user platform.

[0013] In a method according to a preferred embodiment of the present invention, the method further includes generating a proposed cutting plane including the proposed cutting point and the proposed cutting angle or cutting direction, and displaying the proposed cutting plane on the user platform.

[0014] In a method according to a preferred embodiment of the present invention, the method further includes receiving input to modify the proposed cutting plane via the user platform, and changing the proposed cutting plane to a new desired position and / or a new desired cutting angle or cutting orientation using the user platform.

[0015] In a method according to a preferred embodiment of the present invention, the method further includes switching the cutting system, which includes the user platform, from a semi-automatic mode in which a plurality of suggested cutting points, including the suggested cutting point, are displayed on the user platform before the cutting system performs an operation step, to a fully automatic mode in which the cutting system automatically performs the operation step, wherein the switching of the cutting system from the semi-automatic mode to the fully automatic mode is performed based on the user's approval rate for the plurality of suggested cutting points.

[0016] In a method according to a preferred embodiment of the present invention, the cutting system is switched from the semi-automatic mode to the fully automatic mode when the user's approval rate is equal to or greater than an approval rate threshold.

[0017] In a method according to a preferred embodiment of the present invention, the cutting system switches from the semi-automatic mode to the fully automatic mode when the user's approval rate is equal to or greater than an approval rate threshold and a predetermined number of proposed cutting points have been reviewed by the user via the user platform.

[0018] A method according to a preferred embodiment of the present invention further includes indicating via the user platform that the user's approval rate is equal to or greater than an approval rate threshold, and receiving via the user platform an input to switch the disconnection system from the semi-automatic mode to the fully automatic mode.

[0019] In a method according to a preferred embodiment of the present invention, the method further includes switching a cutting system, which includes the user platform, between a semi-automatic mode in which a plurality of suggested cutting points, including the suggested cutting point, are displayed on the user platform before the cutting system performs an operation step, and a fully automatic mode in which the cutting system automatically performs the operation step, wherein the switching of the cutting system between the semi-automatic mode and the fully automatic mode is performed based on input received via the user platform.

[0020] In a method according to a preferred embodiment of the present invention, the method further includes receiving, via the user platform, input for modifying one or more rules used by the processor to generate the proposed breakpoints.

[0021] The method according to a preferred embodiment of the present invention includes displaying the agricultural crop on the user platform and creating a new cutting point or a new cutting plane for the agricultural crop using the user platform. Creating the new cutting point or the new cutting plane includes designating a specified position for the new cutting point or the new cutting plane and displaying the new cutting point or the new cutting plane at the specified position on the user platform.

[0022] In the method according to a preferred embodiment of the present invention, the method further includes detecting one or more agricultural features of the agricultural crop using an object detection model and displaying the one or more agricultural features detected using the object detection model on the user platform.

[0023] In the method according to a preferred embodiment of the present invention, the method further includes displaying a proposed cutting point or a proposed cutting plane generated by a processor on the user platform.

[0024] In the method according to a preferred embodiment of the present invention, the method further includes switching a cutting system including the user platform from a manual mode in which a new cutting point or a new cutting surface is created using the user platform before an operation step is executed by the cutting system to a fully automatic mode in which the operation step is automatically executed by the cutting system. Creating the new cutting point or the new cutting surface for the crop using the user platform includes creating a plurality of new cutting points or new cutting surfaces for the crop using the user platform. Displaying the new cutting point or the new cutting surface on the user platform includes displaying the plurality of new cutting points or new cutting surfaces. The switching of the cutting system from the manual mode to the fully automatic mode is based on a prediction rate of the plurality of new cutting points or new cutting surfaces, and the prediction rate includes a rate at which the plurality of new cutting points or new cutting surfaces match a plurality of proposed cutting points or proposed cutting surfaces generated by a processor.

[0025] A user platform according to a preferred embodiment of the present invention is a user platform comprising an input unit that receives one or more inputs from a user, a display, and a processor operably connected to the input unit and the display, wherein the processor is configured or programmed to control the display to display proposed cutting points for a crop, and the processor is configured or programmed to confirm that the proposed cutting points are properly arranged, delete the proposed cutting points, or modify the proposed cutting points based on the one or more inputs received by the input unit.

[0026] In the user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to move the proposed cutting points to a new desired position based on the one or more inputs received by the input unit.

[0027] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to control the display to display the proposed disconnect point after the proposed disconnect point has been moved to the new desired position, and the processor is configured or programmed to confirm, based on input received by the input unit, that the proposed disconnect point is appropriately positioned at the new desired position.

[0028] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to receive user comments via the input unit regarding the reason why the proposed disconnect point was moved to the new desired position.

[0029] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to control the display to display a plurality of suggested breakpoints, including the suggested breakpoint, and for each of the plurality of suggested breakpoints, the processor is configured or programmed to verify that the suggested breakpoint is properly positioned, delete the suggested breakpoint, or modify the suggested breakpoint, based on one or more inputs received by the input unit.

[0030] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to confirm, based on the input received by the input unit, that each of the plurality of proposed disconnections is acceptable to the user.

[0031] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to control the display to show one or more agricultural features of the crop, the one or more agricultural features of the crop being detected using an object detection model.

[0032] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to control the display to display a proposed cutting plane including the proposed cutting point and the proposed cutting angle or cutting orientation.

[0033] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to modify the proposed cutting plane based on one or more inputs received by the input unit, thereby changing the proposed cutting plane to a new desired position and / or a new desired cutting angle or cutting orientation.

[0034] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to switch the cutting system, which includes the user platform, from a semi-automatic mode in which a plurality of suggested cutting points, including the suggested cutting point, are displayed on the display before the cutting system performs an operation step, to a fully automatic mode in which the cutting system automatically performs the operation step, the switching of the cutting system from the semi-automatic mode to the fully automatic mode is performed based on the user's approval rate for the plurality of suggested cutting points.

[0035] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to switch the disconnection system from the semi-automatic mode to the fully automatic mode when the user's approval rate is equal to or greater than an approval rate threshold.

[0036] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to switch the cutting system from the semi-automatic mode to the fully automatic mode when the user's approval rate is equal to or greater than an approval rate threshold and a predetermined number of proposed cutting points have been reviewed by the user via the user platform.

[0037] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to perform control to display an indication when the user's approval rate is equal to or greater than an approval rate threshold, and the processor is configured or programmed to switch the disconnection system from the semi-automatic mode to the fully automatic mode based on input received by the input unit.

[0038] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to switch the cutting system, which includes the user platform, between a semi-automatic mode in which a plurality of suggested cutting points, including the suggested cutting point, are displayed on the display before the cutting system performs an operation step, and a fully automatic mode in which the cutting system automatically performs the operation step, and the processor is configured or programmed to switch the cutting system between the semi-automatic mode and the fully automatic mode based on input received by the input unit.

[0039] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to modify one or more rules used to generate the proposed disconnection point based on the input received by the input unit.

[0040] A user platform according to a preferred embodiment of the present invention comprises an input unit for receiving input from a user, a display, and a processor operably connected to the input unit and the display, wherein the processor is configured or programmed to control the display to display crops, the processor is configured or programmed to create new cutting points or new cutting surfaces of crops based on one or more inputs received by the input unit, and the processor is configured or programmed to control the display to display the new cutting points or new cutting surfaces.

[0041] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to control the display to show one or more agricultural features of the crop, the one or more agricultural features of the crop being detected using an object detection model.

[0042] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to control the display to show a proposed cutting point or proposed cutting plane.

[0043] In a user platform according to a preferred embodiment of the present invention, the processor is configured or programmed to switch the cutting system, which includes the user platform, from a manual mode in which the user platform is used to create the new cutting point or new cutting surface before the cutting system performs an operation step to a fully automatic mode in which the cutting system automatically performs the operation step, wherein creating the new cutting point or new cutting surface on the crop includes creating a plurality of new cutting points or new cutting surfaces on the crop, and displaying the new cutting points or new cutting surfaces on the user platform includes displaying the plurality of new cutting points or new cutting surfaces, and the processor is configured or programmed to switch the cutting system from the manual mode to the fully automatic mode based on a prediction rate of the plurality of new cutting points or new cutting surfaces, wherein the prediction rate includes the rate at which the plurality of new cutting points or new cutting surfaces match a plurality of proposed cutting points or proposed cutting surfaces.

[0044] The above and other features, elements, steps, configurations, characteristics, and advantages of the present invention will become more apparent from the following detailed description of preferred embodiments of the invention with reference to the accompanying drawings. [Brief explanation of the drawing]

[0045] A patent or application file includes at least one color drawing. The patent or application file also includes a black and white line drawing corresponding to each of the at least one color drawing.

[0046] Figure 1 shows a front perspective view of a cutting system according to a preferred embodiment of the present invention.

[0047] Figure 2 shows an enlarged view of a part of the cutting system according to a preferred embodiment of the present invention.

[0048] Figures 3A and 3B show an example block diagram of a cloud system including a cutting system, a cloud platform, and a user platform according to a preferred embodiment of the present invention.

[0049] Figure 4 is a flowchart showing the cutting point generation process according to a preferred embodiment of the present invention.

[0050] Figures 5A and 5B show examples of images captured in the imaging step according to a preferred embodiment of the present invention, where Figure 5A shows an example of an image captured in the imaging step in black and white, and Figure 5B shows an example of an image captured in the imaging step in color.

[0051] Figures 6A and 6B are flowcharts illustrating an example of the disparity estimation step according to a preferred embodiment of the present invention, where Figure 6A shows the flowchart in black and white and Figure 6B shows the flowchart in color.

[0052] Figures 7A and 7B are flowcharts illustrating other examples of the disparity estimation step according to a preferred embodiment of the present invention, where Figure 7A shows the flowchart in black and white and Figure 7B shows the flowchart in color.

[0053] Figures 8A and 8C show point cloud generation steps according to preferred embodiments of the present invention, with Figure 8A showing the point cloud generation steps in black and white and Figure 8C showing the point cloud generation steps in color.

[0054] Figures 8B and 8D show a point cloud registration step according to a preferred embodiment of the present invention, where Figure 8B shows the point cloud registration step in black and white, and Figure 8D shows the point cloud registration step in color.

[0055] Figures 9A and 9C show an example of a depth-based thresholding process according to a preferred embodiment of the present invention, where Figure 9A shows the depth-based thresholding process in black and white, and Figure 9C shows the depth-based thresholding process in color.

[0056] Figures 9B and 9D show an example of an outlier removal process according to a preferred embodiment of the present invention, where Figure 9B shows the outlier removal process in black and white, and Figure 9D shows the outlier removal process in color.

[0057] Figures 10A and 10B show an example of a component division step according to a preferred embodiment of the present invention, where Figure 10A shows the component division step in black and white and Figure 10B shows the component division step in color.

[0058] Figures 11A, 11B, and 11C show examples of segmented images generated in the component division step according to a preferred embodiment of the present invention, where Figures 11A and 11B show examples of segmented images in black and white, and Figure 11C shows an example of a segmented image in color.

[0059] Figures 12A and 12B show examples of images annotated using a computer-implemented labeling tool according to a preferred embodiment of the present invention, where Figure 12A shows the image in black and white and Figure 12B shows the image in color.

[0060] Figures 13A and 13B show an expansion process according to a preferred embodiment of the present invention, with Figure 13A showing the expansion process in black and white and Figure 13B showing the expansion process in color.

[0061] Figures 14A and 14B show examples of agricultural feature detection steps according to preferred embodiments of the present invention, with Figure 14A showing an example of the agricultural feature detection step in black and white and Figure 14B showing an example of the agricultural feature detection step in color.

[0062] Figures 15A and 15B show examples of feature images generated in the agricultural feature detection step according to a preferred embodiment of the present invention, where Figure 15A shows the feature image in black and white and Figure 15B shows the feature image in color.

[0063] Figures 16A and 16B show examples of images annotated using a computer-implemented labeling tool according to a preferred embodiment of the present invention, where Figure 16A shows the image in black and white and Figure 16B shows the image in color.

[0064] Figures 17A and 17B show examples of the cutting point generation step according to a preferred embodiment of the present invention, with Figure 17A showing an example of the cutting point generation step in black and white and Figure 17B showing an example of the cutting point generation step in color.

[0065] Figure 18 is a flowchart showing the cutting point generation step according to a preferred embodiment of the present invention.

[0066] Figure 19 is a flowchart showing the process for determining the cutting point angle according to a preferred embodiment of the present invention.

[0067] Figures 20A and 20B show feature images illustrating the process for determining the cutting point angle according to a preferred embodiment of the present invention, where Figure 20A shows the feature image in black and white and Figure 20B shows the feature image in color.

[0068] Figures 21A and 21B show examples of the cutting point projection step according to a preferred embodiment of the present invention, with Figure 21A showing an example of the cutting point projection step in black and white and Figure 21B showing an example of the cutting point projection step in color.

[0069] Figures 22A and 22B show examples of the cutting point registration step according to a preferred embodiment of the present invention, with Figure 22A showing an example of the cutting point registration step in black and white, and Figure 22B showing an example of the cutting point registration step in color.

[0070] Figures 23A and 23B show examples of trace modules according to preferred embodiments of the present invention, with Figure 23A showing an example of a trace module in black and white and Figure 23B showing an example of a trace module in color.

[0071] Figures 24A and 24B show examples of agricultural feature projection steps according to preferred embodiments of the present invention, with Figure 24A showing an example of the agricultural feature projection step in black and white and Figure 24B showing an example of the agricultural feature projection step in color.

[0072] Figures 25A and 25B show examples of agricultural feature registration steps according to preferred embodiments of the present invention, with Figure 25A showing an example of the agricultural feature registration step in black and white, and Figure 25B showing an example of the agricultural feature registration step in color.

[0073] Figures 26A and 26B show an example of a user platform according to a preferred embodiment of the present invention, where Figure 26A shows the user platform in black and white and Figure 26B shows the user platform in color.

[0074] Figures 27A and 27B show an example of a user platform according to a preferred embodiment of the present invention, where Figure 27A shows the user platform in black and white and Figure 27B shows the user platform in color.

[0075] Figures 28A and 28B show an example of a user platform according to a preferred embodiment of the present invention, where Figure 28A shows the user platform in black and white and Figure 28B shows the user platform in color.

[0076] Figures 29A and 29B show an example of a user platform according to a preferred embodiment of the present invention, where Figure 29A shows the user platform in black and white and Figure 29B shows the user platform in color.

[0077] Figure 30A is a flowchart showing a process according to a preferred embodiment of the present invention.

[0078] Figure 30B is a flowchart showing a process according to a preferred embodiment of the present invention. [Modes for carrying out the invention]

[0079] Detailed description of preferred embodiments Figure 1 shows a front perspective view of a cutting system 1 according to a preferred embodiment of the present invention. As shown in Figure 1, the cutting system 1 may have a moving body or the like. However, the cutting system 1 can be installed on a moving body or a cart that can be towed by a person, or on a self-propelled or self-propelled cart or moving body.

[0080] As shown in Figure 1, the cutting system 1 includes a base frame 10, side frames 12 and 14, a horizontal frame 16, and a vertical frame 18. The side frames 12 and 14 are mounted on the base frame 10, and the side frames 12 and 14 directly support the horizontal frame 16. The vertical frame 18 is mounted on the horizontal frame 16. One or more devices, such as a camera 20, a robotic arm 22, and / or a cutting tool 24, may be mounted and supported on, for example, the vertical frame 18, and / or other frames among frames 10, 12, 14, or 16.

[0081] The base frame 10 has a base frame motor 26 that can move the side frames 12 and 14 along the base frame 10 so that one or more devices can be moved in the depth direction (z-axis shown in Figure 1). The horizontal frame 16 has a horizontal frame motor 28 that can move the vertical frame 18 along the horizontal frame 16 so that one or more devices can be moved in the horizontal direction (x-axis shown in Figure 1). The vertical frame 18 has a vertical frame motor 30 that can move one or more devices along the vertical frame 18 in the vertical direction (y-axis shown in Figure 1). Each of the base frame motor 26, horizontal frame motor 28, and vertical frame motor 30 may be, for example, a screw motor. A screw motor can provide relatively high precision for precisely moving and positioning one or more devices. However, each of the base frame motor 26, horizontal frame motor 28, and vertical frame motor 30 may be any motor that provides a continuous torque of, for example, about 0.2 Nm or more, preferably about 0.3 Nm or more.

[0082] Each of the base frame motor 26, horizontal frame motor 28, and vertical frame motor 30 may be designed and / or sized according to the total weight of one or more devices. Furthermore, the couplers for each of the base frame motor 26, horizontal frame motor 28, and vertical frame motor 30 may be modified according to the diameter of the motor shaft and / or the corresponding mounting hole pattern.

[0083] The base frame 10 can be mounted on the base 32, and the base electronics system 34 can also be mounted on the base 32. Multiple wheels 36 can be mounted on the base 32. For example, as shown in Figures 3A and 3B, the multiple wheels 36 can be controlled by the base electronics system 34, which may have a power supply 35 for driving an electric motor 37, etc. As an example, the multiple wheels 36 can be driven by an electric motor 37 having a target capacity of about 65 kW to about 75 kW, and the power supply 35 for the electric motor 37 may be a battery with a capacity of about 100 kWh.

[0084] The base electronics system 34 also includes a processor and memory elements programmed or configured to perform autonomous navigation of the cutting system 1. Furthermore, as shown in Figure 1, a LiDAR (light detection and ranging) system 38 and a Global Navigation Satellite System (GNSS) 40 are installed or supported, for example, on the base frame 10 or base 32 and / or other frames among frames 10, 12, 14, or 16, so that the position data of the cutting system 1 can be determined. The LiDAR system 38 and GNSS 40 may be used for obstacle avoidance and navigation when the cutting system 1 is moving autonomously. Preferably, for example, the cutting system 1 is implemented using a remote control interface and can communicate via one or more of Ethernet, USB, wireless communication, and GPS RTK (real-time kinematics). The remote control interface and communication device may be included in either or both of the base electronics system 34 and the imaging electronics system 42 (described later). As shown in Figure 1, the cutting system 1 may include a display device 43 that displays data and / or images obtained by one or more devices and information provided by the base electronic system 34 (e.g., position, speed, battery life of the cutting system 100), or may be communicably connected to such a display device 43. The display device 43 may include an input device that accepts user input operations and may include one or more buttons or switches, a display such as a liquid crystal or OLED display, a storage device including a semiconductor storage medium such as flash memory, and a processor that executes one or more computer programs stored in the storage device. The input device and the display of the display device 43 may be implemented as a touchscreen 43a. Alternatively, data and / or images obtained by one or more devices and provided by the base electronic system 34 may be displayed to the user through a user platform 45, which will be described in more detail below.

[0085] Figure 2 is an enlarged view of a part of the cutting system 1 having one or more devices. As shown in Figure 2, one or more devices include, for example, a camera 20, a robotic arm 22, and a cutting tool 24, which may be mounted on the vertical frame 18 and / or other frames among frames 10, 12, 14, or 16. Further devices among the one or more devices may also be mounted on, for example, the vertical frame 18 and / or other frames among frames 10, 12, 14, or 16.

[0086] Camera 20 may include a stereo camera, an RGB camera, etc. As shown in Figure 2, camera 20 may have a body 20a that includes a first camera / lens 20b (e.g., left camera / lens) and a second camera / lens 20c (e.g., right camera / lens). Alternatively, body 20a may include two or more cameras / lenses. The resolution of camera 20 may be, for example, 1536 × 2048 pixels or 2448 × 2048 pixels, but camera 20 may have a different resolution. Camera 20 may have, for example, a PointGrey CM3-U3-31S4C-CS or PointGrey CM3-U3-50S5C sensor and a 3.5mm f / 2.4 or 5mm f / 1.7 lens, and a field of view of 74.2535 × 90.5344 or 70.4870 × 80.3662. Camera 20 may have other sensors and lenses, and may also have different fields of view.

[0087] One or more light sources 21 may be mounted on one or more sides of the camera body 20a. The light sources 21 may have LED light sources oriented in the same direction as one or more devices, such as the camera 20, along the z-axis as shown in Figure 1. The light sources 21 may provide illumination to one or more objects imaged by the camera 20. For example, the light sources 21 may act as a flash to compensate for ambient light when imaged by the camera 20 during daytime operation. During nighttime operation, the light sources 21 may act as a flash for the camera 20, or the light sources may provide steady illumination to the camera 20. In a preferred embodiment, one or more light sources 21 may have, for example, 100-watt LED modules, but LED modules with different wattages (e.g., 40 watts or 60 watts) may be used.

[0088] The robot arm 22 may include robot arms known to those skilled in the art, such as the Universal Robot 3 e-series robot arms and the Universal Robot 5 e-series robot arms. For example, the robot arm 22, also known as an articulated robot arm, may have multiple joints that act as axes enabling degrees of freedom of motion. Here, the more rotary joints the robot arm 22 has, the higher the degrees of freedom of motion it has. For example, the robot arm 22 may have 4 to 6 joints, and these will provide the same number of axes of rotation for motion.

[0089] In a preferred embodiment of the present invention, the controller may be configured or programmed to control the movement of the robot arm 22. For example, the controller may be configured or programmed to control the movement of the robot arm 22 to which a cutting tool 24 is attached, and to position the cutting tool 24, according to steps described later. For example, the controller may be configured or programmed to control the movement of the robot arm 22 based on the position of the cutting point located on the target crop.

[0090] In a preferred embodiment of the present invention, the cutting tool 24 has a body 24a and a blade portion 24b, as shown, for example, in Figure 2. The blade portion 24b may have a driven blade that moves relative to a fixed blade and is actuated to perform a cutting operation together with the fixed blade. The cutting tool 24 may have, for example, a cutting device disclosed in U.S. Patent Application No. 17 / 961,666, which is titled “End Effector Including Cutting Blade And Pulley Assembly,” which is incorporated herein by reference in its entirety.

[0091] In a preferred embodiment of the present invention, the cutting tool 24 may be attached to the robot arm 22 using a robot arm mount assembly 23. The robot arm mount assembly 23 may be, for example, the robot arm mount assembly disclosed in U.S. Patent Application No. 17 / 961,668, which is titled “Robotic Arm Mount Assembly Including Rack And Pinion” and is incorporated herein by reference in its entirety.

[0092] The cutting system 1 may have an imaging electronic system 42, which can be installed on the side frame 12 or side frame 14, for example, as shown in Figure 1. The imaging electronic system 42 may supply power to and control each of the base frame motor 26, the horizontal frame motor 28, and the vertical frame motor 30. That is, the imaging electronic system 42 may have a power supply that provides power to each of the base frame motor 26, the horizontal frame motor 28, and the vertical frame motor 30. The imaging electronic system 42 may also have a processor and memory element programmed or configured to control each of the base frame motor 26, the horizontal frame motor 28, and the vertical frame motor 30. The processor and memory element of the imaging electronic system 42 may also be configured or programmed to control one or more devices having a camera 20, a robot arm 22, a robot arm mount assembly 23, and a cutting tool 24. The processor and memory element of the imaging electronic system 42 may also be configured or programmed to process image data obtained by the camera 20.

[0093] As described above, the imaging electronic system 42 and the base electronic system 34 may include processors and memory elements. The processors may be hardware processors, multipurpose processors, microprocessors, dedicated processors, digital signal processors (DPS), and / or other types of processing components configured or programmed to process data. The memory elements may include one or more volatile, non-volatile, and / or interchangeable data storage elements. For example, the memory elements may include magnetic, optical, and / or flash storage elements that can be integrated whole or partially with the processor. The memory elements may store instructions and / or instruction sets or programs that can be read and / or executed by the processor.

[0094] In another preferred embodiment of the present invention, the imaging electronics system 42 may be partially or completely implemented by the base electronics system 34. For example, each of the base frame motor 26, the horizontal frame motor 28, and the vertical frame motor 30 may receive power from and / or be controlled by the base electronics system 34 rather than the imaging electronics system 42.

[0095] In a further preferred embodiment of the present invention, the imaging electronic system 42 may be connected to a separate power supply (one or more) from the base electronic system 34. For example, the power supply may be included in one or both of the imaging electronic system 42 and the base electronic system 34. The base frame 10 may also be detachably attached to the base 32 so that the base frame 10, side frames 12 and 14, horizontal frame 16, vertical frame 18, and the components installed thereon can be installed on another mobile body or the like.

[0096] The base frame motor 26, horizontal frame motor 28, and vertical frame motor 30 can move one or more devices in three separate directions or along three separate axes. However, in another preferred embodiment of the present invention, only a portion of one or more devices, such as a camera 20, a robotic arm 22, and a cutting tool 24, may be moved by the base frame motor 26, horizontal frame motor 28, and vertical frame motor 30. For example, the base frame motor 26, horizontal frame motor 28, and vertical frame motor 30 may move only the camera 20. Furthermore, the cutting system 1 may be configured to move the camera 20 linearly along only a single axis while the camera captures multiple images, as will be described later. For example, the horizontal frame motor 28 may be configured to move the camera 20 linearly across a target crop such as a grapevine, and the camera 20 may capture multiple images of the grapevine.

[0097] The imaging electron system 42 and the base electron system 32 of the cutting system 1 are, for example, VIDIA(R) JETSON TMThe mobile platform may be provided by being partially or fully implemented by edge computing, such as by an AGX computer. In a preferred embodiment of the present invention, edge computing provides all of the computational and communication needs of the disconnected system 1. Figures 3A and 3B show an example block diagram of a cloud system that includes a mobile platform and interacts with a cloud platform and a user platform 45. In a preferred embodiment, the user platform 45 may include an input device 45a that accepts user input operations and may include one or more buttons or switches, a display device 45b including a display such as a liquid crystal or OLED display, a storage device 45c including a semiconductor storage medium such as flash memory, and a processor 45d operably connected to the input device 45a, the display device 45b, and the storage device 45c, and which executes one or more computer programs stored in the storage device. The input device 45a and the display device 45b of the user platform 45 may be implemented as touchscreens. For example, the user platform 45 may include a computer or a portable device such as a smartphone, tablet computer, or remote control. The user platform 45 may be used in the field where the cutting system 1 performs agricultural work, or it may be used in a location away from the field where the cutting system 1 performs agricultural work.

[0098] As shown in Figures 3A and 3B, edge computing on a mobile platform includes a cloud agent, which is a service-based component that facilitates communication between the mobile platform and the cloud platform. For example, the cloud agent can receive command and instruction data from the cloud platform (e.g., a web application on the cloud platform) and forward the command and instruction data to the corresponding component on the mobile platform. As another example, the cloud agent can send operational and production data to the cloud platform. Preferably, the cloud platform may include software components and data storage to maintain the overall operation of the cloud system. Preferably, the cloud platform provides enterprise-level services with on-demand capability, fault tolerance, and high availability (e.g., Amazon Web Services). TM The cloud platform includes one or more application programming interfaces (APIs) for communicating with the mobile platform and the user platform 45. Preferably, the APIs are protected with a high level of security, and the capabilities of each API may be automatically adjusted according to the computing load. The user platform 45 provides a dashboard (e.g., on a touchscreen 45a) for controlling the cloud system and receiving data acquired by the mobile platform and the cloud platform. The dashboard can be implemented by a web-based application (e.g., an internet browser), a mobile application, a desktop application, etc.

[0099] As an example, the edge computing of the mobile platform shown in Figures 3A and 3B can obtain data from hardware GPS (Global Positioning System) (e.g., GNSS 40) and LiDAR data (e.g., from LiDAR system 38). The mobile platform can also obtain data from camera 20. The edge computing of the mobile platform may have, for example, temporary storage for storing the raw data obtained by camera 20. The edge computing of the mobile platform may also have, for example, persistent storage for storing processed data. Specifically, camera data stored in temporary storage may be processed by an artificial intelligence (AI) model, the camera data may then be stored in persistent storage, and a cloud agent may retrieve and transmit the camera data from persistent storage.

[0100] Figure 4 is a flowchart showing a cutpoint generation process according to a preferred embodiment of the present invention. The cutpoint generation process shown in Figure 4 has multiple steps, including an imaging step S1, a disparity estimation step S2, a component division step S3, an agricultural feature detection step S4, a point cloud generation step S5, a point cloud registration step S6, a cutpoint generation step S7, a cutpoint projection step S8, a cutpoint registration step S9, a mega registration step S10, an operation step S11, an agricultural feature projection step S12, and an agricultural feature registration step S13. These will be described in more detail below.

[0101] In a preferred embodiment of the present invention, the disparity estimation step S2, the component division step S3, and the agricultural feature detection step S4 can be performed simultaneously. Alternatively, one or more of the disparity estimation step S2, the component division step S3, and the agricultural feature detection step S4 can be performed individually or in series.

[0102] In a preferred embodiment of the present invention, imaging step S1 includes the cutting system 1 moving to a waypoint located in front of the target crop (e.g., a grapevine). The waypoint may be pre-configured or programmed in the onboard memory of the cutting system 1 or retrieved from remote storage, and may be determined based on the distance or time from the previous waypoint. Upon reaching the waypoint located in front of the target crop, the cutting system 1 stops and uses the camera 20 to capture multiple images of the target crop.

[0103] In a preferred embodiment of the present invention, imaging step S1 includes capturing multiple images of the target crop from multiple viewpoints (e.g., multiple positions of the camera 20) using the camera 20. For example, at each viewpoint, the camera 20 is controlled to capture a first image (e.g., a left image) using a first lens 20a and a second image (e.g., a right image) using a second lens 20b. By controlling the horizontal frame motor 28, the camera 20 can be moved to multiple positions in front of the target crop in the horizontal direction (x-axis direction in Figure 1), thereby reaching multiple viewpoints (positions of the camera 20). The number of viewpoints can be determined based on the field of view of the camera 20 and the number of viewpoints required to capture an image of the entire target crop. In a preferred embodiment, the multiple images captured by the camera 20 are stored in the local storage of the cutting system 1.

[0104] Figures 5A and 5B show examples of multiple images captured in imaging step S1. For example, a first image (image L0) is captured from viewpoint 0 using the first lens 20a, and a second image (image R0) is captured using the second lens 20b. After images L0 and R0 are captured from viewpoint 0, the camera 20 is moved to viewpoint 1. From viewpoint 1, a first image (image L1) is captured using the first lens 20a, and a second image (image R1) is captured using the second lens 20b. Similarly, the first image (image L2) and the second image (image R2) are captured from viewpoint 2, the first image (image L3) and the second image (image R3) from viewpoint 3, the first image (image L4) and the second image (image R4) from viewpoint 4, the first image (image L5) and the second image (image R5) from viewpoint 5, and the first image (image L6) and the second image (image R6) from viewpoint 6. In the examples shown in Figures 5A and 5B, the camera 20 is moved from left to right to reach viewpoints 0 to 6, but it is also possible to move the camera 20 from right to left to reach viewpoints 0 to 6.

[0105] In a preferred embodiment of the present invention, imaging step S1 may include downsampling the image captured by the camera 20, for example, by a factor of 2. Imaging step S1 may also include rectifying each pair of stereo images (e.g., image L0 and image R0) captured using the first lens 20a and the second lens 20b. This involves the process of reprojecting the image planes (left image plane and right image plane) onto a common plane parallel to the line between the camera lenses.

[0106] In the imaging step S1, once an image of the target crop is captured, the disparity estimation step S2 can be performed. In a preferred embodiment of the present invention, the disparity estimation step S2 is an example of a depth estimation step that generates a depth estimate of the crop. The disparity estimation step S2 includes estimating the depth of pixels contained in the image captured in the imaging step S1 using a disparity estimation model. In a preferred embodiment, the disparity estimation model generates a disparity map 46 corresponding to the images captured from viewpoints 0 to 6 in the imaging step S1. The disparity estimation step S2 can be performed using a number of approaches, including artificial intelligence (AI) deep learning approaches or classical computer vision approaches, as will be described in more detail below.

[0107] Figures 6A and 6B are flowcharts illustrating an example of a disparity estimation step S2 performed using an AI deep learning approach. Figures 6A and 6B show an AI disparity estimation model 44 used to generate a disparity map 46-0 corresponding to viewpoint 0 shown in Figures 5A and 5B. The input to the AI ​​disparity estimation model 44 includes a parallelized stereo image pair containing a first image (image L0) and a second image (image R0), respectively, captured from viewpoint 0 in the imaging step S1. The output of the AI ​​disparity estimation model 44 includes a disparity map 46-0 corresponding to viewpoint 0.

[0108] Using the AI ​​disparity estimation model 44, disparity maps 46 corresponding to each of viewpoints 0 to 6 can be generated. For example, using a parallelized stereo image pair containing a first image (image L1) and a second image (image R1) captured at viewpoint 1 in imaging step S1, disparity map 46-1 corresponding to viewpoint 1 can be generated. Similarly, using a parallelized stereo image pair containing a first image (image L2) and a second image (image R2) captured at viewpoint 2 in imaging step S1, disparity map 46-2 corresponding to viewpoint 2 can be generated. Furthermore, using images captured at viewpoints 3 to 6, disparity maps 46-3 to 46-6 corresponding to viewpoints 3 to 6 can be generated.

[0109] In a preferred embodiment of the present invention, the AI ​​disparity estimation model 44 matches each pixel in a first image (e.g., image L0) with the corresponding pixel in a second image (e.g., image R0) based on a correspondence. This aims to determine that pairs of pixels in the first and second images are projections of the same physical point in space. The AI ​​disparity estimation model 44 then calculates, for example, the distance between each pair of matching pixels. In a preferred embodiment of the present invention, the AI ​​disparity estimation model 44 generates a disparity map 46 based on the fact that depth is inversely proportional to disparity, for example, that the greater the disparity, the closer the object in the image is. In a preferred embodiment, the disparity map 46 includes pixel differences mapped to real-world depth (depth) based on the configuration of the camera 20, including, for example, intrinsic and extrinsic parameters of the camera 20.

[0110] Figures 6A and 6B show examples of the disparity estimation model 44 including AI (deep learning) frameworks, such as stereo matching AI frameworks like the RAFT-Stereo architecture. The RAFT-Stereo architecture is based on optical flow and uses a recurring neural network approach. The RAFT-Stereo architecture may include, for example, feature encoders 44a and 44b, a context encoder 44c, a correlation pyramid 44d, and a disparity estimator component / gated recurrent unit (GRU) 44e. Feature encoder 44a is applied to the first image (e.g., image L0), and feature encoder 44b is applied to the second image (e.g., image R0). Feature encoders 44a and 44b map the first and second images, respectively, to a dense feature map used to construct a correlation volume. Feature encoders 44a and 44b determine individual features of the first and second images, including density, texture, and pixel intensity.

[0111] In a preferred embodiment of the present invention, the context encoder 44c is applied only to the first image (e.g., image L0). The context features generated by the context encoder 44c are used to initialize the hidden state of the update operator and are also injected into the GRU 44e during each iteration of the update operator. The correlation pyramid 44d constructs a three-dimensional correlation volume using the feature maps generated by the feature encoders 44a and 44b. In a preferred embodiment, the disparity estimator component / gated regressive unit (GRU) 44e estimates the disparity of each pixel in the image. For example, the GRU 44e predicts a set of disparity fields from the initial starting point using the current estimate of disparity in each iteration. Between each iteration, the correlation volume is indexed using the current estimate of disparity to generate a set of correlation features. The correlations, disparities, and context features are then concatenated and injected into the GRU 44e. The GRU 44e updates the hidden state and uses the updated hidden state to predict the disparity update.

[0112] In a preferred embodiment of the present invention, AI deep learning frameworks / approaches other than the RAFT-Stereo architecture can be used to perform the disparity estimation step S2 to generate disparity maps 46 corresponding to viewpoints (e.g., viewpoints 0-6). For example, disparity maps 46 can be generated by performing the disparity estimation step S2 using AI deep learning approaches such as EdgeStereo, HSM-Net, LEAStereo, MC-CNN, LocalExp, CRLE, HITNet, NOSS-ROB, HD3, gwcnet, PSMNet, GANet, and DSMNet. The neural networks of the above-mentioned AI approaches, including the RAFT-Stereo architecture, can be trained using artificially generated / synthesized datasets.

[0113] Alternatively, the disparity estimation step S2 can be performed using a classical computer vision approach. A classical computer vision approach may include a stereo semi-global block matching (SGMB) function 48, which is an intensity-based approach that generates a dense disparity map 46 for 3D reconstruction. More specifically, the SGMB function 48 is a geometric approach algorithm that uses the eigenparameters and exextrinsic parameters of a camera (e.g., camera 20) used to capture images for generating the disparity map 46.

[0114] Figures 7A and 7B are flowcharts illustrating an example in which the disparity estimation step S2 is performed using a classical computer vision approach. In Figures 7A and 7B, for example, the SGMB function 48 is used to generate the disparity map 46-0 corresponding to viewpoint 0 shown in Figures 5A and 5B. The input to the SGMB function 48 includes a parallelized stereo image pair containing a first image (e.g., image L0) and a second image (e.g., image R0) captured from viewpoint 0 in the imaging step S1. The output of the SGMB function 48 includes the disparity map 46-0 corresponding to viewpoint 0. The SGMB function 48 can be used to generate disparity maps 46 corresponding to viewpoints 0 to 6.

[0115] In a preferred embodiment of the present invention, a first image (e.g., image L0) and a second image (e.g., image R0) are input to the camera parallelization and destraining module 47 before the SGMB function 48, as shown, for example, in Figures 7A and 7B. The camera parallelization and destraining module 47 performs a camera parallelization step and a destraining step before generating a disparity map 46 using the SGMB function 48. In the camera parallelization step, a transformation process is performed to project the first image and the second image onto a common plane. In the image destraining step, a mapping process is performed to map the coordinates of the output destrained image to the input camera image using distortion coefficients. After the camera parallelization and destraining module 47 has performed the camera parallelization step and the destraining step, the camera parallelization and destraining module 47 outputs the pair of parallelized and destrained images to the SGMB function 48.

[0116] In a preferred embodiment of the present invention, parameters of the SGMB function 48, such as window size, minimum parallax, or maximum parallax, can be fine-tuned according to factors including image size, lighting conditions, and camera mounting angle of camera 20 in order to optimize parallax. The uniqueness ratio parameter can also be fine-tuned to filter out noise. The parameters of the SGMB function 48 may also include post-processing parameters used to avoid speckle artifacts, namely speckle window size and speckle range, which can also be fine-tuned based on operating conditions including lighting conditions and camera mounting angle of camera 20.

[0117] In a preferred embodiment of the present invention, the point cloud generation step S5 includes generating a point cloud 49 corresponding to the viewpoint from which the image was captured in the imaging step S1. For example, the point cloud generation step S5 includes generating point clouds 49-0 to 49-6 corresponding to viewpoints 0 to 6 shown in Figures 5A and 5B. Each of the point clouds 49-0 to 49-6 is a set of data points in space, each point having a position represented as a set of real-world Cartesian coordinates (x, y, z points).

[0118] As shown in Figures 8A and 8C, for example, the disparity map 46 generated in the disparity estimation step S2 can be reprojected onto three-dimensional data points by the point cloud generation module 491 to form a point cloud 49 generated in the point cloud generation step S5. The point cloud generation step S5 includes the process of converting the two-dimensional disparity (depth) map 46 into a set of points (X, Y, Z coordinates) of the point cloud 49. In a preferred embodiment, the point cloud generation module 491 can convert the disparity (depth) map 46 into three-dimensional points of the point cloud 49 using the camera parameters of the camera 20. For example, the equation Z = fB / d can be used to convert the disparity (depth) map 46 into three-dimensional points, where f is the focal length (in pixels), B is the baseline (in meters), and d is the disparity (depth) map 46. After Z is determined, X and Y can be calculated using the camera projection equations (1) X = uZ / f and (2) Y = vZ / f. Here, u and v are pixel positions in a 2D image space, X is the x-axis in the real world, Y is the y-axis in the real world, and Z is the z-axis in the real world.

[0119] In a preferred embodiment, the disparity map 46 can be transformed into three-dimensional points used to generate a point cloud 49 by using built-in API functions and an inverse projection matrix obtained using the intrinsic and exextrinsic parameters of the camera 20. For example, the values ​​of X (real-world x-position), Y (real-world y-position), and Z (real-world z-position) can be determined based on the following matrix.

number

[0120] In the matrix above, X is the x-coordinate in the real world (x-axis), Y is the y-coordinate in the real world (y-axis), and Z is the z-coordinate in the real world (z-axis). Variables x and y are values ​​corresponding to the coordinates in the calibrated 2D left or right image (e.g., image L0 or image R0) captured in imaging step S1, and variable z = 1. Disparity (x,y) is the disparity value determined from the disparity map 46. For example, the disparity map 46 can be a 1-channel 8-bit unsigned, 16-bit signed, 32-bit signed, or 32-bit floating-point disparity image. Variable Q can be a 4x4 viewpoint transformation matrix, which is a disparity-to-depth mapping matrix, and can be obtained using a program such as stereoRectify based on the following variables: That is, the eigenmatrices of the first camera (e.g., the first camera 20b), the first camera distortion parameters, the eigenmatrices of the second camera (e.g., the second camera 20c), the second camera distortion parameters, the image size used for stereo calibration, the rotation matrix from the coordinate system of the first camera to the coordinate system of the second camera, and the translation vector from the coordinate system of the first camera to the second camera. For example, the variable Q can be represented by the following matrix: c x1 c is the distance (in pixels) from the left edge of the parallelized 2D image (e.g., parallelized image L0) to the point where the optical axis (e.g., the axis between the center of the first camera / lens 20b and the physical object) intersects the image plane of the parallelized 2D image, and c x2 c is the distance (in pixels) from the left edge of the parallelized 2D image (e.g., parallelized image R0) to the point where the optical axis (e.g., the axis between the center of the second camera / lens 20c and the physical object) intersects the image plane of the parallelized 2D image, and c y F is the distance (in pixels) from the left edge of the parallelized 2D image (e.g., parallelized image L0) to the point where the optical axis (e.g., the axis between the center of the first camera / lens 20b or the second camera / lens 20c and the physical object) intersects the image plane of the parallelized 2D image, where f is the focal length (in pixels), and T xis the distance between the first camera 20b and the second camera 20c.

number

[0121] Therefore, the variable W can be expressed by the following equation. The variable W can be used to convert the values ​​of X (real-world x-position), Y (real-world y-position), and Z (real-world z-position) from pixels to units of distance (e.g., millimeters).

number

[0122] The point cloud generation step S5 may include generating point clouds 49-0 to 49-6 corresponding to viewpoints 0 to 6, respectively, as shown in Figures 5A and 5B. For example, point cloud 49-0 can be generated based on disparity map 46-0 corresponding to viewpoint 0, point cloud 49-1 can be generated based on disparity map 46-1 corresponding to viewpoint 1, point cloud 49-2 can be generated based on disparity map 46-2 corresponding to viewpoint 2, and similarly, point clouds 49-3 to 49-6 can be generated based on disparity maps 46-3 to 46-6 corresponding to viewpoints 3 to 6, respectively.

[0123] In a preferred embodiment of the present invention, the point cloud registration step S6 includes determining one or more spatial transformations (e.g., scaling, rotation, and / or translation) to combine / align the point clouds (e.g., point clouds 49-0 to 49-6) generated in the point cloud generation step S5. More specifically, the point cloud registration module 1161 is used to align the point cloud 49 generated in the point cloud generation step S5 to generate a mega point cloud 116, as shown in Figures 8B and 8D, for example.

[0124] In a preferred embodiment, the point cloud registration step S6 may be performed based on one or more assumptions, including that the horizontal frame 16 is precisely horizontal and oriented correctly, and that the physical distance between each viewpoint (e.g., viewpoints 0-6) is a predetermined value, such as approximately 15 cm or approximately 20 cm. Based on such one or more assumptions, it may be sufficient to perform translation along the X-axis (the axis of the horizontal frame 16) to obtain the mega point cloud 116. In a preferred embodiment, a 4x4 transformation matrix can be used to transform individual point clouds 49 from one viewpoint to individual point clouds 49 from another viewpoint, such that each element of the transformation matrix represents translation and rotation information. For example, in the point cloud registration step S6, a 4x4 transformation matrix can be used to sequentially transform each of the point clouds (e.g., point clouds 49-0 to 49-6) generated in the point cloud generation step S5 to generate the mega point cloud 116.

[0125] In a preferred embodiment of the present invention, a depth-based thresholding step can be performed after the point cloud generation step S5. The depth-based thresholding step includes removing points from the point cloud 49 that have a depth greater than a set depth-based threshold. The depth-based threshold is, for example, a user-configurable depth value (a value in the z-direction shown in Figure 1). For example, the depth-based threshold can be set based on the length of the robot arm 22 or the working space of the robot arm 22. For example, if the length of the robot arm is 1.5 meters, or if the working space of the robot arm extends 1.5 meters in the depth direction (z-direction in Figure 1), the depth-based threshold can be set to a value of 1.5 meters. In other words, the depth-based threshold can be set based on the working range in the depth direction of the cutting system 1.

[0126] Since a disparity map 46 is generated using the image captured in the imaging step S1, each point cloud 49 generated in the point cloud generation step S5 is generated using the disparity map 46, which includes both the foreground and background. For example, the target crop (e.g., grapevines) is included in the foreground of the disparity map 46, and the background of the disparity map 46 is not of interest. The depth-based thresholding step can remove points from the point cloud 49 that correspond to the background of the disparity map 46. Figures 9A and 9C show examples of the point cloud 49A before the depth-based thresholding step and the same point cloud 49B after the depth-based thresholding step. The depth-based thresholding step can reduce the number of points included in the point cloud 49. Therefore, the computation time and memory requirements, which are affected by the number of points included in the point cloud 49, can be reduced.

[0127] In a preferred embodiment of the present invention, a statistical outlier removal step can be performed after the point cloud generation step S5. The statistical outlier removal step can be performed after the depth-based thresholding step, or it can be performed before or simultaneously with the depth-based thresholding step. The statistical outlier removal step includes a process of removing trailing points and dense points generated in the disparity estimation step S2 from undesirable regions of the point cloud 49. For example, the statistical outlier removal step may include a process of removing trailing points and dense points from undesirable regions of the point cloud 49 that include portions of the point cloud 49 corresponding to the edges of an object, for example, the edges of a vine.

[0128] In a preferred embodiment, the statistical outlier removal step includes removing points that are farther from their neighbors compared to the mean of the point cloud 49. For example, the mean distance between a given point and its neighbors is calculated by calculating the distance between that point and a predetermined number of neighbors for each given point in the point cloud 49. Parameters of the statistical outlier removal step include a neighbor parameter and a ratio parameter. The neighbor parameter sets how many neighbors to consider when calculating the mean distance for a given point. The ratio parameter can set a threshold level based on the standard deviation of the mean distance of the entire point cloud 49, and determines the extent to which the statistical outlier removal step removes points from the point cloud 49. In a preferred embodiment, the lower the ratio parameter, the more aggressively the statistical outlier removal step filters / removes points from the point cloud 49. Figures 9B and 9D show examples of front views of point cloud 49C before and after the statistical outlier removal step, as well as side views of point cloud 49E before and after the statistical outlier removal step. A statistical outlier removal step can be used to reduce noise and the number of points from undesirable regions in the point cloud 49, thereby reducing computation time.

[0129] In the example described above, the depth-based thresholding step and statistical outlier removal step are performed on the individual point clouds (e.g., point clouds 49-0 to 49-6) generated in the point cloud generation step S5. However, in addition to, or alternatively to, performing the depth-based thresholding step and statistical outlier removal step on the individual point clouds, the depth-based thresholding step and statistical outlier removal step can also be performed on the megapoint cloud 116 generated by the point cloud registration step S6.

[0130] In a preferred embodiment of the present invention, component division step S3 includes identifying different segments (e.g., individual components) of the crop in question. For example, if the crop in question is a grapevine, component division step S3 may include identifying different segments of the grapevine, including the trunk, each individual cordon, each individual spur, and each individual cane.

[0131] In a preferred embodiment, the component segmentation step S3 is performed using an instance segmentation AI architecture 50. The instance segmentation AI architecture 50 may include a fully convolutional network (FCN) and may rely on an instance mask representation scheme that dynamically segments each instance in an image. Figures 10A and 10B show an example of the component segmentation step S3 using the instance segmentation AI architecture 50 to identify different segments of a target crop (e.g., a grapevine). The input to the instance segmentation AI architecture 50 includes an image of the target crop. For example, as shown in Figures 10A and 10B, the input to the instance segmentation AI architecture 50 includes an image (e.g., image L2) captured in the imaging step S1. The instance segmentation AI architecture 50 receives the image input and outputs a segmented image 51. The segmented image 51 includes one or more masks that identify different segments / individual components of the target crop contained in the image input to the instance segmentation AI architecture 50. For example, Figures 10A and 10B show that the instance segmentation AI architecture 50 outputs a segmented image 51 that includes masks that identify different segments of a grapevine, such as the main trunk, each individual main branch, each individual short shoot, and each individual branch. Figures 10A and 10B show that the segmented image 51 includes masks that include a main trunk mask 52, a main branch mask 54 that masks individual main branches, a short shoot mask 56 that masks individual short shoots, and a branch mask 58 that masks individual branches.

[0132] In a preferred embodiment of the present invention, the instance segmentation AI architecture 50 may include mask generation, which is separated into mask kernel prediction and mask feature learning, each generating a convolution kernel and a feature map to be convolved. The instance segmentation AI architecture 50 can significantly reduce or prevent inference overhead by a novel non-maximum suppression (NMS) technique for matrices. This technique takes an image (e.g., image L2 shown in Figures 10A and 10B) as input and directly outputs instance masks (e.g., trunk mask 52, main branch mask 54, short branch mask 56, and branch mask 58) and corresponding class probabilities in a fully convolutional, box-free, and grouping-free paradigm.

[0133] In a preferred embodiment of the present invention, the component division step S3 includes using the instance segmentation AI architecture 50 to identify different segments of the target crop (e.g., a grapevine) that are included in one or more of the multiple images captured in the imaging step S1. For example, Figures 11A to C show images L0 to L6 captured in the imaging step S1 and input to the instance segmentation AI architecture 50 in the component division step S3, and segmented images 51-0 to 51-6 output by the instance segmentation AI architecture 50 when images L0 to L6 are input to the instance segmentation AI architecture 50.

[0134] In a preferred embodiment, the instance segmentation AI architecture 50 employs adaptive learning and dynamic convolutional kernels for mask prediction, and a deformable convolutional network (DCN) is used. For example, the SoloV2 instance segmentation framework can be used to perform the component partitioning step S3. However, the instance segmentation AI architecture 50 may include instance segmentation frameworks other than the SoloV2 framework to perform the component partitioning step S3. For example, the instance segmentation AI architecture 50 may include the Mask-RCNN framework, which includes a deep neural network that can be used to perform the component partitioning step S3. Alternatively, the instance segmentation AI architecture 50 may also include instance segmentation frameworks such as SOLO, TrnsorMask, YOLACT, PolarMask, and BlendMask to perform the component partitioning step S3.

[0135] In a preferred embodiment of the present invention, the instance segmentation AI architecture 50 is trained using a segmentation dataset tailored to an instant segmentation task for a specific target crop. For example, if the target crop is a grapevine, the segmentation dataset is tailored to an instant segmentation task for a grapevine. The segmentation dataset includes multiple images selected based on factors such as whether the images were captured under appropriate operating conditions and whether the images contain an appropriate level of variety. Once multiple images to be included in the segmentation dataset have been selected, the multiple images are cleansed and annotated. For example, the multiple images in the segmentation dataset can be manually annotated using a computer-implemented labeling tool, as will be described in detail below.

[0136] Figures 12A and 12B show an example of image 60 of a segmentation dataset annotated using a computer-implemented labeling tool. The computer-implemented labeling tool includes a user interface that allows the formation of polygon masks around segments / individual components of the target crop. For example, if the target crop is a grapevine, the labeling tool's user interface can form polygon masks around different segments of the grapevine, including the main trunk, each individual main branch, each individual short shoot, and each individual branch. Polygon masks can also be formed around other objects included in image 60, such as poles or trellises used to support parts of the target crop. Each polygon mask formed around a segment of the target crop or other object is assigned a label that indicates the instance of the target crop or object segment around which the polygon mask is formed. For example, Figures 12A and 12B show a trunk polygon mask 62 formed around the main trunk, a main branch polygon mask 64 formed around individual main branches, a short branch polygon mask 66 formed around individual short branches, and a branch polygon mask 68 formed around individual branches.

[0137] In a preferred embodiment of the present invention, the labeling tool enables a specific type of annotation called group-identification-based labeling, which can be used to annotate discrete portions of the same segment / individual component using the same label. In other words, group-identification-based labeling can be used to annotate discrete portions of the same instance using the same label. Figures 12A and 12B show an example where the crop in question is a grapevine and group-identification-based labeling can be used to annotate discrete portions of the same branch using the same label. For example, in image 60 shown in Figures 12A and 12B, a first branch 70 overlaps / intersects with a second branch 72 in image 60, and image 60 contains a first discrete portion 72a and a second discrete portion 72b, which are spaced apart from each other in image 60 but are parts of the same second branch 72. Group identification-based labeling creates a first polygon mask 74 around the first discrete portion 72a, a second polygon mask 76 around the second discrete portion 72b, and the same label is assigned to the first polygon mask 74 and the second polygon mask 76, i.e., a common label, to indicate that the first discrete portion 72a and the second discrete portion 72b are parts of the same second branch 72.

[0138] In a preferred embodiment of the present invention, approximately 80% of the segmentation dataset is used as a training set for training and teaching the network of the instance segmentation AI architecture, and approximately 20% of the segmentation dataset is used as a validation / test set for the network included in the instance segmentation AI architecture 50. However, these percentages can be adjusted so that more or less of the segmentation dataset is used as the training set and the validation / test set.

[0139] In a preferred embodiment of the present invention, an augmentation process can be used to create additional images for the segmentation dataset from existing images included in the segmentation dataset. As shown in Figures 13A and 13B, the augmentation process may include editing and modifying the original captured images 78 to create new images that can be included in the segmentation dataset to create a good distribution of images for the network of the instance segmentation AI architecture 50 to learn / train. By including multiple different relative augments applied to the original captured images 78 in the augmentation process, the network of the instance segmentation AI architecture 50 can learn and generalize a wide range of lighting conditions, textures, and spatial augments.

[0140] Figures 13A and 13B show examples of augments that can be performed on the original captured image 78 during the augmentation process. For example, the augmentation process may include non-perpendicular augments such as color jitter augmentation 80, equalization augmentation 82, Gaussian blur augmentation 84, and sharpening augmentation 86, and / or spatial augments such as perpendicular augmentation 88 and affine augmentation 90. Non-perpendicular augments can be included in a custom data loader that operates on the fly and reduces memory constraints. Spatial augments can be manually added and saved before the network of the instance segmentation AI architecture 50 is trained with the updated segmentation dataset.

[0141] In a preferred embodiment of the present invention, agricultural feature detection step S4 includes detecting specific agricultural features of a target crop. For example, if the target crop is a grapevine, agricultural feature detection step S4 may include detecting one or more buds on the grapevine. Agricultural feature detection step S4 can be performed using an object detection model 92, such as an AI deep learning object detection model. Figures 14A and 14B show an example of agricultural feature detection step S4 in which an object detection model 92 is used to detect / identify specific agricultural features of a target crop (e.g., a grapevine). The input to the object detection model 92 includes an image of the target crop. For example, as shown in Figures 14A and 14B, the input to the object detection model 92 may include a first image (e.g., image L2) captured in imaging step S1. The object detection model 92 receives the input of the first image and outputs a feature image 94 which includes bounding boxes 96 surrounding the specific agricultural features shown in the first image. For example, Figures 14A and 14B show that the object detection model 92 outputs a feature image 94 that includes bounding boxes 96 surrounding the buds contained in the first image.

[0142] In a preferred embodiment of the present invention, the agricultural feature position 95 of an agricultural feature (e.g., a sprout) can be defined by the x and y coordinates of the center point of the bounding box 96 surrounding the agricultural feature. For example, the agricultural feature position 95 can be defined by the x and y coordinates of a pixel in the feature image 94 that includes the center point of the bounding box 96 surrounding the agricultural feature. Alternatively, the agricultural feature position 95 can be defined using the x and y coordinates of another point within or on the bounding box 96 (e.g., the lower left corner, lower right corner, upper left corner, or upper right corner of the bounding box 96). Thus, the agricultural feature position 95 can be determined for each agricultural feature (e.g., a sprout) detected in the agricultural feature detection step S4.

[0143] In a preferred embodiment of the present invention, the agricultural feature detection step S4 includes detecting / identifying agricultural features contained in each of a plurality of images captured in the imaging step S1 using an object detection model 92. For example, Figures 15A and 15B show images L0 to L6 captured in the imaging step S1 and input to the object detection model 92 in the agricultural feature detection step S4, and feature images 94-0 to 94-6 output by the object detection model 92 based on images L0 to L6. The object detection model 92 may include a model backbone, a model neck, and a model head. The model backbone is mainly used to extract important features from a given input image (e.g., image L2 in Figures 14A and 14B). In a preferred embodiment, a cross-stage partial (CSP) network can be used as the model backbone to extract useful features from the input image. The model neck is mainly used to generate a feature pyramid. The feature pyramid helps the object detection model 92 to generalize well when scaling objects of agricultural features (e.g., grape buds). The performance of the object detection model 92 is improved by identifying the same object (e.g., grape buds) at different scales and sizes. The model head is primarily used for the final detection of agricultural features. The model head applies anchor boxes to agricultural features contained in the image features and generates a final output vector with class probabilities, object scores, and bounding boxes 96 for the feature image 94.

[0144] In a preferred embodiment of the present invention, the agricultural feature detection step S4 is performed using an object detection model 92 such as YoloV5. However, the agricultural feature detection step S4 can also be performed using other models such as Yolov4. The trained object detection model 92 can be converted into a TensorRT optimization engine for faster inference.

[0145] The object detection model 92 can be trained using a detection dataset tailored to an object detection task related to a particular agricultural feature. For example, if the agricultural feature is grape buds, the detection dataset can be tailored to an object detection task related to grape buds. The detection dataset contains multiple images, selected based on factors such as whether the images were captured under appropriate operating conditions and whether the images contain an appropriate level of variety. Once the multiple images to be included in the detection dataset have been selected, the images are cleansed and annotated. For example, images in a detection dataset tailored to an object detection task related to grape buds can be manually annotated using a computer-implemented labeling tool.

[0146] Figures 16A and 16B show an example of image 98 included in a detection dataset annotated using a computer-implemented labeling tool. The computer-implemented labeling tool includes a user interface that allows polygon masks to be formed around specific agricultural features 100 of a target crop. For example, if the agricultural feature 100 is a grapevine bud, the labeling tool's user interface can form a polygon mask 102 around each grapevine bud. In a preferred embodiment, polygon masks 102 of different sizes can be formed around agricultural features 100 in image 98. For example, the size of the polygon mask 102 can be determined based on the size of the specific agricultural feature 100 around which the polygon mask 102 should be formed. For example, if the size of the specific agricultural feature 100 in image 98 is small due to a large distance between the specific agricultural feature 100 and the camera used to capture image 98, then the size of the polygon mask 102 formed around the specific agricultural feature 100 will be small. More specifically, in a preferred embodiment, the size of each polygon mask 102 formed around an agricultural feature 100 in the image 98 can be determined / adjusted based on a predetermined ratio of the pixel area of ​​the agricultural feature 100 to the total pixel area of ​​the polygon mask 102. For example, the size of the polygon mask 102 formed around an agricultural feature 100 in the image 98 can be determined / adjusted such that the ratio between the pixel area of ​​the agricultural feature 100 and the total pixel area of ​​the polygon mask 102 is a predetermined ratio of 50% (i.e., the area of ​​the agricultural feature 100 is 50% of the total area of ​​the polygon mask 102). Alternatively, each polygon mask 102 can be the same size regardless of the size of the particular agricultural feature 100 around which the polygon mask 102 is to be formed.

[0147] In a preferred embodiment of the present invention, approximately 80% of the detection dataset is used as a training set for training and teaching the network of the object detection model 92, and approximately 20% of the detection dataset is used as a validation / test set for the network of the object detection model 92. However, these percentages can be adjusted so that a large or small portion of the dataset is used as the training set and the validation / test set.

[0148] In a preferred embodiment of the present invention, an augmentation process can be used to create additional images for the detection dataset from existing images included in the detection dataset. As shown in Figures 13A and 13B, the augmentation process may include editing and modifying the original captured images 78 to create new images to be included in the detection dataset in order to create a good distribution of images in the detection dataset for the object detection model 92 network to train / learn. By including multiple different relative augments applied to the original captured images 78, the augmentation process can enable the object detection model 92 network to learn and generalize across a wide range of lighting conditions, textures, and spatial augments.

[0149] Figures 13A and 13B show examples of augments that can be performed on the original captured image 78 during the augmentation process. For example, Figures 13A and 13B show that the augmentation process can include non-perspective augments such as color jitter augmentation 80, equalization augmentation 82, Gaussian blur augmentation 84, and sharpening augmentation 86, and / or spatial augments such as perspective augmentation 88 and affine augmentation 90. In a preferred embodiment, non-perspective augments can be included in a custom data loader that operates on the fly and reduces memory constraints. Spatial augments can be manually added and saved before the object detection model 92 network is trained with the updated dataset.

[0150] In a preferred embodiment of the present invention, the cutting point generation step S7 includes generating two-dimensional cutting points 108 using a cutting point generation module 104. If the target crop is a grapevine, the cutting point generation module 104 generates two-dimensional cutting points 108 for a branch of the grapevine. Preferably, the cutting point generation module 104 generates two-dimensional cutting points 108 for each branch included in the grapevine. As an example, Figures 17A and 17B show an example of a two-dimensional cutting point 108 on a cutting point image 106. The location of the two-dimensional cutting point 108 can be represented by x and y coordinates. For example, the location of the two-dimensional cutting point 108 can be defined by the x and y coordinates of a pixel in the cutting point image 106 that contains the two-dimensional cutting point 108.

[0151] As shown in Figures 17A and 17B, the cut-point generation module 104 receives input including a mask from the segmented image 51 generated by the instance segmentation AI architecture 50 in the component division step S3, and the agricultural feature locations 95 of agricultural features (e.g., buds) detected in the agricultural feature detection step S4. For example, Figures 17A and 17B show that the input to the cut-point generation module 104 includes a mask from the segmented image 51 (e.g., segmented image 51-2) and the agricultural feature locations 95 of agricultural features (buds) contained in the corresponding feature image 94 (e.g., feature image 94-2). Both of these are generated using the image L2 captured from viewpoint 2 in the imaging step S1.

[0152] In a preferred embodiment of the present invention, the cut point generation module 104 generates a two-dimensional cut point 108 by performing an agricultural feature association step S18-1, an agricultural feature identification step S18-2, and a cut point generation step S18-3. Figure 18 shows a flowchart of the cut point generation step S7, which includes the agricultural feature association step S18-1, the agricultural feature identification step S18-2, and the cut point generation step S18-3.

[0153] In the agricultural feature association step S18-1, the agricultural features detected in the agricultural feature detection step S4 are associated with specific segments / individual components of the target crop identified in the component division step S3. For example, if the agricultural feature is a grapevine bud, each bud detected in the agricultural feature detection step S4 is associated with a specific branch of the grapevine identified in the component division step S3. In the example shown in Figures 17A and 17B, when the bud position 95 is compared with the branch mask 58 of the segmented image 51, if the agricultural feature position 95 (bud position 95) falls within / exists within the specific branch mask 58, then the bud associated with bud position 95 is considered to be located on / attached to the branch associated with the specific branch mask 58. For example, if the pixel of agricultural feature position 95 in feature image 94 corresponds to a pixel in the branch mask 58 of segmented image 51, then it can be determined that the agricultural feature position 95 (bud position 95) falls within / exists within the branch mask 58. In this way, the buds detected in the agricultural feature detection step S4 can be associated with the specific branch / branch mask 58 identified in the component division step S3.

[0154] When comparing the bud position 95 with the branch mask 58 of the segmented image 51, it is possible that the agricultural feature position 95 (bud position 95) does not fall within / exist within a particular branch mask 58. For example, because buds are attached to the outer surface of branches, the agricultural feature position 95 (bud position 95) may be adjacent to the branch mask 58 and therefore not fall within / exist within the branch mask 58. To address this problem, a search radius is assigned to the agricultural feature position 95. If it is determined that the agricultural feature position 95 is located within the area of ​​the branch mask 58, the agricultural feature position 95 is maintained. However, if it is determined that the agricultural feature position 95 is not located within the area of ​​the branch mask 58, the search radius is used to determine whether the agricultural feature position 95 is located within a predetermined distance from the branch mask 58. Using the search radius, if it is determined that a branch mask 58 is located within a predetermined distance from the agricultural feature position 95, the position of the agricultural feature position 95 is moved to a point within the area of ​​the branch mask 58, for example, the nearest point within the area of ​​the branch mask 58. On the other hand, if the search radius is used to determine that the branch mask 58 is not located within a predetermined distance from the agricultural feature location 95, then it is determined that the agricultural feature location 95 is either not located on the branch mask 58 or is not associated with the branch mask 58.

[0155] The agricultural feature identification step S18-2 includes assigning each agricultural feature an identifier relating to a specific segment / individual component of the target crop to which the agricultural feature was associated in the agricultural feature association step S18-1. For example, if the agricultural feature is a grape bud, each bud is assigned an identifier relating to a specific branch / branch mask 58 to which the bud was associated in the agricultural feature association step S18-1.

[0156] The agricultural feature identification step S18-2 may include identifying the starting point 57 of the branch mask 58 located at the connection point between the short-prune mask 56 and the branch mask 58. For example, the connection point between the short-prune mask 56 and the branch mask 58 can be identified by pixels that fall into both the short-prune mask 56 and the branch mask 58. This connection point indicates the overlap between the short-prune mask 56 and the branch mask 58. Once the starting point 57 of the branch mask 58 is identified, each bud detected in the agricultural feature detection step S4 can be assigned an identifier relating to the specific branch / branch mask 58 associated with the bud in the agricultural feature association step S18-1, based on the distance from the starting point 57 of the branch mask 58 to the respective bud. In the examples shown in Figures 17A and 17B, agricultural feature position 95-1 is closest to the starting point 57 of the branch mask 58 (the connection point between the short-prune mask 56 and the branch mask 58), agricultural feature position 95-2 is second closest to the starting point 57 of the branch mask 58, and agricultural feature position 95-3 is third closest to the starting point 57 of the branch mask 58. Agricultural feature positions 95-1, 95-2, and 95-3 are illustrated on the cutting point image 106 in Figures 17A and 17B.

[0157] Based on the respective distances from the starting point 57 of the branch mask 58 to the agricultural feature locations 95-1, 95-2, and 95-3, each agricultural feature can be assigned an identifier relating to a specific segment / individual component of the target crop to which the agricultural feature is associated. For example, a bud with agricultural feature location 95-1 can be assigned as the first bud of the branch associated with the branch mask 58, a bud with agricultural feature location 95-2 can be assigned as the second bud of the branch associated with the branch mask 58, and a bud with agricultural feature location 95-3 can be assigned as the third bud of the branch associated with the branch mask 58.

[0158] Step S18-3, the cut point generation step, includes executing a cut point generation algorithm to generate a two-dimensional cut point 108. The cut point generation algorithm uses one or more rules to generate a two-dimensional cut point 108 based on one or more identifiers assigned to agricultural features in the agricultural feature identification step S18-2. For example, if the agricultural feature is a grapevine bud and the specific segment / individual component of the target crop is a specific branch / branch mask 58 of the grapevine, the rule may include generating a two-dimensional cut point 108 between a first bud having agricultural feature position 95-1 and a second bud having agricultural feature position 95-2 when the branch contains multiple buds (when multiple agricultural feature positions 95 are located within the branch mask 58). More specifically, the rule may include generating a cut point 108 at the midpoint (approximately 50th percentile) between agricultural feature position 95-1 and agricultural feature position 95-2. Alternatively, the rule may include generating a breakpoint 108 at another point between agricultural feature location 95-1 and agricultural feature location 95-2 (e.g., approximately the 30th percentile or approximately the 70th percentile). Alternatively, the rule may include generating a breakpoint 108 at a predetermined distance from agricultural feature location 95-1. One or more of the above rules may also include not generating a breakpoint if the branch contains a single bud or does not contain a bud, for example, if a single agricultural feature location 95 is located within the branch mask 58 or if agricultural feature location 95 is not located there.

[0159] In preferred embodiments of the present invention, one or more of the above rules may differ from or be modified from the rules described above. For example, if a branch contains more than two buds (if more than two agricultural feature positions 95 are located within the branch mask 58), one or more of the above rules may include generating a cutting point 108 between a second bud having agricultural feature position 95-2 which is the second closest to the starting point 57 of the branch mask 58, and a third bud having agricultural feature position 95-3 which is the third closest to the starting point 57 of the branch mask 58.

[0160] In a preferred embodiment of the present invention, the two-dimensional cut point 108 generated in the cut point generation step S18-3 may not be located on a branch or within the branch mask 58. For example, if the cut point 108 is generated at the midpoint (approximately 50th percentile) between agricultural feature position 95-1 and agricultural feature position 95-2, and the branch between agricultural feature position 95-1 and agricultural feature position 95-2 is bent or curved, the generated cut point 108 may not be located on a branch or within the branch mask 58. To address this problem, a search radius is assigned to the cut point 108. If it is determined that the cut point 108 generated in the cut point generation step S18-3 is located within the area of ​​the branch mask 58, the position of the cut point 108 is maintained. On the other hand, if it is determined that the cut point 108 generated in the cut point generation step S18-3 is not located within the area of ​​the branch mask 58, the search radius is used to determine whether the cut point 108 generated in the cut point generation step S18-3 is located within a predetermined distance from the branch mask 58. If the search radius is used to determine that the cutting point 108 is located within a predetermined distance from the branch mask 58, the position of the cutting point 108 is moved to a point within the area of ​​the branch mask 58, for example, to the point within the area of ​​the branch mask 58 closest to the cutting point 108 generated in the cutting point generation step S18-3. On the other hand, if the search radius is used to determine that the cutting point 108 is not located within a predetermined distance from the branch mask 58, the cutting point 108 is deleted.

[0161] In a preferred embodiment of the present invention, the cutting point angle is determined for a two-dimensional cutting point 108. An example of the process used to determine the cutting point angle is shown in the flowchart of Figure 19. In step S19-1, it is identified which agricultural feature positions 95 the cutting point 108 was generated between. For example, as shown in Figures 20A and 20B, it is identified that the cutting point 108 was generated between agricultural feature positions 95-1 and 95-2. In step S19-2, the angle of the portion of a particular segment / individual component of a crop where the cutting point 108 is located is determined using the agricultural feature positions 95 identified in step S19-1. For example, the angle of the portion of the branch where the cutting point 108 is located is determined by forming a line 126 connecting agricultural feature positions 95-1 and 95-2. Once the angle of the portion of the crop that is the specific segment / individual component where the cutting point 108 is located is determined in step S19-2, the cutting point angle of the cutting point 108 can be determined in step S19-3 by forming a line 127 perpendicular to line 126 at a certain angle to line 126, for example. Line 127 may also be formed at another angle to line 126, for example, 30 degrees or 45 degrees to line 126. The angle of line 127 defines the cutting point angle of the cutting point 108, which is the angle with respect to the specific segment / individual component of the crop where the cutting point 108 is located.

[0162] In a preferred embodiment of the present invention, the cutting point generation step S7 includes generating a set of two-dimensional cutting points 108 using a cutting point generation module 104 with a plurality of images captured from a plurality of viewpoints (e.g., viewpoints 0 to 6) in the imaging step S1. For example, the cutting point generation step S7 may include generating a set of two-dimensional cutting points 108 for each viewpoint from which an image was captured in the imaging step S1 (e.g., one cutting point 108 for each branch) using the cutting point generation module 104. The cutting point generation module 104 generates a first set of cutting points 108 based on the mask of segmented image 51-0 (see Figures 11A-C) and agricultural feature positions 95 from feature image 94-0 (see Figures 15A and 15B), generates a second set of cutting points 108 based on the mask of segmented image 51-1 (see Figures 11A-C) and agricultural feature positions 95 from feature image 94-1 (see Figures 15A and 15B), generates a third set of cutting points 108 based on the mask of segmented image 51-2 (see Figures 11A-C) and agricultural feature positions 95 from feature image 94-2 (see Figures 15A and 15B), and generates a first set of cutting points 108 based on the mask of segmented image 51-3 (see Figures 11A-C) and feature image 94-3 (see Figure 15A and 15B). A fourth set of cutting points 108 can be generated based on agricultural feature locations 95 from (see A and 15B), a fifth set of cutting points 108 can be generated based on the mask of segmented image 51-4 (see Figures 11A-C) and agricultural feature locations 95 from feature image 94-4 (see Figures 15A and 15B), a sixth set of cutting points 108 can be generated based on the mask of segmented image 51-5 (see Figures 11A-C) and agricultural feature locations 95 from feature image 94-5 (see Figures 15A and 15B), and a seventh set of cutting points 108 can be generated based on the mask of segmented image 51-6 (see Figures 11A-C) and agricultural feature locations 95 from feature image 94-6 (see Figures 15A and 15B).

[0163] In a preferred embodiment of the present invention, the cutting point projection step S8 includes generating a three-dimensional cutting point 114 using a cutting point projection module 110. As shown in Figures 21A and 21B, the cutting point projection module 110 receives an input including a set of two-dimensional cutting points 108 generated in the cutting point generation step S7 and a corresponding disparity map 46 generated in the disparity estimation step S2. In Figures 21A and 21B, the set of two-dimensional cutting points 108 is shown on the cutting point image 106. For example, the input to the cutting point projection module 110 may include a third cutting point image 106 containing a set of two-dimensional cutting points 108 generated in the cutting point generation step S7 based on the mask of the segmented image 51-2 and the agricultural feature positions 95 of the feature image 94-2, and a corresponding disparity map 46 generated in the disparity estimation step S2 based on images L2 and R2. In other words, both the set of 2D cutting points 108 and the corresponding disparity map 46 are generated based on images taken from the same viewpoint, for example, viewpoint 2 shown in Figures 5A and 5B.

[0164] The cutting point projection module 110 outputs a three-dimensional cutting point 114, as shown, for example, in Figures 21A and 21B. For illustrative purposes, Figures 21A and 21B show the three-dimensional cutting point 114 on a three-dimensional cutting point cluster 112. The cutting point projection module 110 generates a three-dimensional cutting point 114 corresponding to the two-dimensional cutting point 108 by slicing the position of the two-dimensional cutting point 108 from the disparity map 46 and reprojecting the sliced ​​disparity with a known camera configuration of a camera (e.g., camera 20). For example, pixels in the cutting point image 106 containing the two-dimensional cutting point 108 can be identified, and the corresponding pixels in the disparity map 46 can be identified. The depth value of the corresponding pixel from the disparity map 46 can be used as the depth value of the two-dimensional cutting point 108. In this way, the two-dimensional cutting point 108 can be projected onto a three-dimensional cutting point 114 including X, Y, and Z coordinates.

[0165] In an alternative preferred embodiment of the present invention, the cutting point projection module 110 receives input including a set of two-dimensional cutting points 108 generated in the cutting point generation step S7 and a crop depth estimate obtained from a LiDAR sensor (e.g., LiDAR system 38), a time-of-flight (TOF) sensor, or another depth sensor capable of generating crop depth estimates. For example, the crop depth estimate can be obtained from point cloud data generated by a LiDAR sensor calibrated to have a coordinate system aligned with the coordinate system of camera 20, and the set of two-dimensional cutting points 108 can be generated based on images captured using camera 20, including an RGB camera. The cutting point projection module 110 generates three-dimensional cutting points 114 by determining the depth values ​​of the two-dimensional cutting points 108 based on the crop depth estimate, thereby generating three-dimensional cutting points 114 corresponding to the two-dimensional cutting points 108. For example, the coordinates (pixels) of the cutting point image 106, which includes the two-dimensional cutting point 108, can be identified, and the corresponding coordinates in the depth estimation of crops, such as the corresponding coordinates in the point cloud data generated by the LiDAR sensor, can be identified. The depth values ​​of the corresponding coordinates from the depth estimation of crops can be used as the depth values ​​of the two-dimensional cutting point 108. In this way, the two-dimensional cutting point 108 can be projected onto a three-dimensional cutting point 114 that includes X, Y, and Z coordinates.

[0166] In a preferred embodiment of the present invention, the cutting point projection step S8 includes generating a set of three-dimensional cutting points 114 for each of the multiple viewpoints (e.g., viewpoints 0 to 6) whose images were captured by the camera 20 in the imaging step S1. For example, using the cutting point projection module 110, a first set of three-dimensional cutting points 114 can be generated using a first set of two-dimensional cutting points 108 and a parallax map 46-0; a second set of three-dimensional cutting points 114 can be generated using a second set of two-dimensional cutting points 108 and a parallax map 46-1; a third set of three-dimensional cutting points 114 can be generated using a third set of two-dimensional cutting points 108 and a parallax map 46-2; a fourth set of three-dimensional cutting points 114 can be generated using a fourth set of two-dimensional cutting points 108 and a parallax map 46-3; a fifth set of three-dimensional cutting points 114 can be generated using a fifth set of two-dimensional cutting points 108 and a parallax map 46-4; a sixth set of three-dimensional cutting points 114 can be generated using a sixth set of two-dimensional cutting points 108 and a parallax map 46-5; and a seventh set of three-dimensional cutting points 114 can be generated using a seventh set of two-dimensional cutting points 108 and a parallax map 46-6.

[0167] In a preferred embodiment of the present invention, when a set of three-dimensional cutting points 114 (for example, the first to seventh sets of three-dimensional cutting points 114) is generated in the cutting point projection step S8, the set of three-dimensional cutting points 114 is joined / aligned with each other in the cutting point registration step S9 to form a set of mega cutting points 115. For illustrative purposes, Figures 22A and 22B show a set of three-dimensional cutting points 114 on a three-dimensional cutting point group 112 corresponding to multiple viewpoints, and a set of mega cutting points 115 on a mega cutting point group 117. The mega cutting point group 117 can be formed by merging a set of mega cutting points 115 with a mega point group 116 generated in the point group registration step S6.

[0168] In a preferred embodiment, a set of three-dimensional cutting points 114 are joined / aligned to one another by a cutting point registration module 1151 that determines one or more spatial transformations (e.g., scaling, rotation, and translation) to align the set of three-dimensional cutting points 114. For example, similar to the point cloud registration step S6, the cutting point registration step S9 may be performed based on one or more assumptions, including that the horizontal frame 16 is exactly horizontal and oriented correctly, and that the physical distance between each viewpoint (e.g., viewpoints 0-6) is a predetermined value. Based on one or more such assumptions, it may be sufficient to perform translation along the X-axis (the axis of the horizontal frame 16) to obtain a set of mega-cutting points 115. In a preferred embodiment, a 4x4 transformation matrix can be used to transform individual sets of three-dimensional cutting points 114 from one viewpoint to another, such that each element of the transformation matrix represents translation and rotation information. For example, each of the sets of three-dimensional cutting points 114 can be sequentially transformed using a 4x4 transformation matrix to complete the cutting point registration step S9 and generate a set of mega-cutting points 115.

[0169] The set of 3D cutting points 114 is generated based on images taken from different viewpoints (e.g., viewpoints 0-6 in Figures 5A and 5B). Therefore, even after a spatial transformation intended to align the set of 3D cutting points 114 is performed in the cutting point registration step S9, the set of 3D cutting points 114 may not be perfectly aligned with each other. Thus, to identify 3D cutting points 114 that represent the same cutting point but belong to different sets of cutting points 114, i.e., 3D cutting points 114 that represent the same cutting point but are still slightly misaligned with each other even after the transformation of the set of 3D cutting points 114, a search radius (e.g., 4 cm) is assigned to each of the 3D cutting points 114. When sets of 3D cutting points 114 are combined / aligned to generate a set of mega-cutting points 115, the search radius of the 3D cutting point 114 is used to determine whether one or more other 3D cutting points 114 from another set of 3D cutting points 114 are located within the search radius of that 3D cutting point 114. If one or more other 3D cut points 114 are located within the search radius of that 3D cut point 114, that 3D cut point 114 and the one or more other 3D cut points 114 are merged into a megacut point 115 that is included in the set of megacut points 115.

[0170] In a preferred embodiment of the present invention, two or more 3D cuts 114 from different sets of 3D cuts 114 must be merged to generate a single megacut 115. For example, if, when a set of 3D cuts 114 is merged / aligned, there are no other 3D cuts 114 from another set of 3D cuts 114 that are located within the search radius of a given 3D cut 114, then no megacut 115 is generated. As another example, three or more 3D cuts 114 from different sets of 3D cuts 114 must be merged to generate a single megacut 115. Alternatively, a megacut 115 may be generated based on a single 3D cut 114 even if, when a set of 3D cuts 114 is merged / aligned, there are no other 3D cuts 114 from another set of 3D cuts 114 that are located within the search radius of that 3D cut 114.

[0171] The mega-cutting points 115 are generated by combining / aligning sets of 3D cutting points 114 that are generated based on images taken from different viewpoints (e.g., viewpoints 0-6 in Figures 5A and 5B). However, in some cases, due to the different viewpoints from which the images were taken, the first 3D cutting point included in the first set of 3D cutting points 114 and the second 3D cutting point included in the second set of 3D cutting points 114 may be located in significantly different positions, even though the first and second 3D cutting points are located on the same specific segment / individual component of the crop (e.g., the same branch). For example, buds detected based on images taken from one viewpoint (e.g., viewpoint 6) on a particular branch may be different from buds detected based on images taken from another viewpoint (e.g., viewpoint 2). As a result, the agricultural features detected from one viewpoint (e.g., viewpoint 6) may differ from those detected from another viewpoint (e.g., viewpoint 2). Consequently, the position of the first 3D crosspoint generated based on the image captured from one viewpoint may differ significantly from the position of the second 3D crosspoint generated based on the image captured from the other viewpoint. For example, an agricultural feature (e.g., a sprout) detected based on an image captured from one viewpoint (e.g., viewpoint 6) may be hidden or otherwise invisible in an image captured from another viewpoint (e.g., viewpoint 2). In such cases, the same agricultural feature will not be detected in the agricultural feature detection step S4 for the image captured from the other viewpoint (viewpoint 2). Furthermore, an agricultural feature may be incorrectly detected in the agricultural feature detection step S4 for an image captured from one viewpoint (e.g., viewpoint 6), and that agricultural feature may not be detected in the agricultural feature detection step S4 for an image captured from the other viewpoint (viewpoint 2). In each of these cases, the first 3D cutting point included in the first set of 3D cutting points 114 and the second 3D cutting point included in the second set of 3D cutting points 114 will be in significantly different locations, even though the first and second 3D cutting points are located on the same specific segment / individual component (e.g., the same branch) of the crop.Therefore, when the set of 3D cutting points 114 are joined / aligned with each other in the cutting point registration step S9, the first 3D cutting point and the second 3D cutting point are not located within each other's search radius and are not merged with each other in the cutting point registration step S9. As a result, the first mega-cutting point 115-1 is generated based on the first 3D cutting point, and the second mega-cutting point 115-2 for the same branch is generated based on the second 3D cutting point. For example, Figures 23A and 23B show the first mega-cutting point 115-1 generated based on the first 3D cutting point generated based on an image taken from one viewpoint, and the second mega-cutting point 115-2 generated based on the second 3D cutting point generated based on an image taken from another viewpoint.

[0172] In a preferred embodiment of the present invention, it is desirable to have only one megacut point 115 for each specific segment / individual component of a crop. That is, it is desirable to have only one megacut point 115 for each branch of a grapevine. Accordingly, a preferred embodiment of the present invention includes a trace module 120 that can be used to identify and remove one or more megacut points 115 when multiple megacut points 115 are assigned to a specific segment / individual component of the crop in question. For example, the trace module 120 can be used to identify and remove one or more megacut points 115 when multiple megacut points 115 are assigned to a branch of a grapevine.

[0173] In a preferred embodiment of the present invention, the mega-cut points 115 generated in the cut point registration step S9 are merged with the mega-point cloud 116 generated in the point cloud registration step S6 to form the mega-cut point cloud 117 in the mega-registration step S10. The mega-cut point cloud 117 is used by the trace module 120. As shown in Figures 23A and 23B, for example, the trace module 120 fits a cylinder 122 around a specific segment of the target crop and traces the specific segment, starting from a first mega-cut point 115 (first mega-cut point 115-1) that is closest to the connection point between the short shoot and the branch. The trace module 120 can determine that the mega-cut point 115-1 is closest to the connection point between the short shoot and the branch by using the short shoot mask 56 and branch mask 58 included in one or more of the segmented images 51 generated in the component division step S3. Branch masks 58 included in one or more of the segmented images 51 are projected onto three-dimensional coordinates using one or more corresponding disparity maps 46, allowing the three-dimensional space of the branches that the cylinder 122 traces around to be determined. The trace module 120 uses the cylinder 122 to trace a specific segment of the target crop from a first megacut point 115-1 to the free end 124 of that particular segment. If multiple megacut points 115 exist in the area traced by the cylinder 122, one or more megacut points following the first megacut point can be identified as false megacut points and removed from the set of megacut points 115. In the example shown in Figures 23A and 23B, the second megacut point 115-2 is identified as a false megacut point and removed from the set of megacut points 115. As a result, the first megacut point 115-1 remains as the only remaining megacut point for the particular branch traced by the trace module 120. In a preferred embodiment, each branch represented in the megacut point group 117 can be traced simultaneously by different cylinders 122 of the trace module 120. Alternatively, the trace module 120 can be used to trace each branch in series (one after the other) until each branch is traced by the trace module 120.

[0174] In a preferred embodiment of the present invention, a mega-cutting angle can be determined for each of one or more mega-cutting points 115. The mega-cutting angle is the angle at which the blade portion 24b of the cutting tool 24 is directed when a cutting operation is performed at the mega-cutting point 115. In a preferred embodiment, the mega-cutting angle can be determined based on the cutting angles of the cutting points 108 corresponding to the mega-cutting point 115. For example, if the mega-cutting point 115 corresponds to cutting points 108 generated from each of a plurality of viewpoints, the cutting angles of these cutting points 108 are averaged to determine the mega-cutting angle. Alternatively, the mega-cutting angle can be determined by averaging the angles of the portion of the branch where the cutting point 108 is located.

[0175] In a preferred embodiment of the present invention, the operation step S11 shown in Figure 4 can be performed based on a set of mega-cutting points 115. Operation step S11 includes controlling one or more of the horizontal frame motor 28, vertical frame motor 30, robot arm 22, or robot arm mount assembly 23 to position the blade portion 24b of the cutting tool 24 and perform a cutting operation at the mega-cutting point 115. In a preferred embodiment, one or more of the horizontal frame motor 28, vertical frame motor 30, robot arm 22, or robot arm mount assembly 23 are controlled via a robot operating system (ROS) and a free-space motion planning framework such as "MoveIt!". This framework is used to plan the movement of the robot arm 22 and cutting tool 24 between two points in space without collision. For example, the free-space motion planning framework can use information from the mega-cutting points 115 and a set of mega-cutting points 117 that provide the real-world coordinates of the target crop to plan the movement of the robot arm 22 and cutting tool 24 between two points in space without colliding with any part of the target crop. More specifically, operation step S11 may include positioning the blade portion 24b of the cutting tool 24 based on a cutting point mark, which is a position on the blade portion 24b of the cutting tool 24 where the mega cutting point 115 coincides.

[0176] In the preferred embodiment of the present invention described above, the agricultural feature detection step S4, in which specific agricultural features of the target crop are detected, is different from the component segmentation step S3. However, in another preferred embodiment of the present invention, the component segmentation step S3 may include identifying specific agricultural features of the target crop. For example, if the target crop is a grapevine, the component segmentation step S3 may include identifying buds of the grapevine when identifying different segments of the grapevine. For example, the component segmentation step S3 can be performed using an instance segmentation AI architecture 50 that identifies different segments of the grapevine, including the main trunk, each individual main branch, each individual short shoot, each individual branch, and each individual bud. In this case, the agricultural feature location 95 can be determined based on the results of the component segmentation step S3, such as an agricultural feature mask (bud mask) output by the instance segmentation AI architecture 50. Therefore, it is not necessary to provide a separate agricultural feature detection step S4.

[0177] In a preferred embodiment of the present invention, the agricultural feature locations 95 of the agricultural features detected in the agricultural feature detection step S4 are defined in two dimensions. For example, the agricultural feature locations 95 are defined by the x and y coordinates of the points of the bounding box 96 surrounding the agricultural feature. The agricultural feature projection step S12 includes generating three-dimensional agricultural features 130 using the agricultural feature projection module 1301. As shown in Figures 24A and 24B, the agricultural feature projection module 1301 receives input including a set of two-dimensional agricultural feature locations 95 generated in the agricultural feature detection step S4 and a corresponding disparity map 46 generated in the disparity estimation step S2. In Figures 24A and 24B, the set of two-dimensional agricultural feature locations 95 is shown on the feature image 94. For example, the input to the agricultural feature projection module 1301 may include agricultural feature locations 95 detected in the agricultural feature detection step S4 based on image L0 and a corresponding disparity map 46 generated in the disparity estimation step S2 based on images L0 and R0. In other words, both the agricultural feature locations 95 and the corresponding disparity maps 46 are generated based on images taken from the same viewpoint, for example, viewpoint 0 shown in Figures 5A and 5B.

[0178] In a preferred embodiment, the agricultural feature projection module 1301 outputs a three-dimensional agricultural feature 130. For example, in Figures 24A and 24B, the three-dimensional agricultural feature 130 is shown on a group of three-dimensional agricultural features 132. The agricultural feature projection module 1301 generates the three-dimensional agricultural feature 130 by slicing the agricultural feature locations (agricultural feature locations 95) from the disparity map 46, reprojecting the sliced ​​disparity using a known camera configuration of a camera (e.g., camera 20), and generating a three-dimensional agricultural feature 130 corresponding to an agricultural feature having a two-dimensional agricultural feature location 95. For example, pixels in a feature image 94 containing the two-dimensional agricultural feature location 95 can be identified, and the corresponding pixels in the disparity map 46 can be identified. The depth value of the corresponding pixel from the disparity map 46 can be used as the depth value of the two-dimensional agricultural feature having the agricultural feature location 95. In this way, a two-dimensional agricultural feature can be projected onto a three-dimensional agricultural feature 130 having X, Y, and Z coordinates.

[0179] In a preferred embodiment of the present invention, the agricultural feature projection step S12 includes generating a set of three-dimensional agricultural features 130 for each of a plurality of viewpoints (e.g., viewpoints 0 to 6) whose images were captured by the camera 20 in the imaging step S1. For example, using the agricultural feature projection module 1301, a first set of three-dimensional agricultural features 130 is generated using agricultural feature locations 95 from feature image 94-0 and a disparity map 46-0; a second set of three-dimensional agricultural features 130 is generated using agricultural feature locations 95 from feature image 94-1 and a disparity map 46-1; a third set of three-dimensional agricultural features 130 is generated using agricultural feature locations 95 from feature image 94-2 and a disparity map 46-2; and agricultural feature locations 9 from feature image 94-3 A fourth set of 3D agricultural features 130 can be generated using 5 and the disparity map 46-3; a fifth set of 3D agricultural features 130 can be generated using agricultural feature locations 95 from feature images 94-4 and the disparity map 46-4; a sixth set of 3D agricultural features 130 can be generated using agricultural feature locations 95 from feature images 94-5 and the disparity map 46-5; and a seventh set of 3D agricultural features 130 can be generated using agricultural feature locations 95 from feature images 94-6 and the disparity map 46-6.

[0180] When a set of 3D agricultural features 130 (for example, the first to seventh sets of 3D agricultural features 130) is generated in the agricultural feature projection step S12, the sets of 3D agricultural features 130 are combined / aligned with each other in the agricultural feature registration step S13 to form a set of mega agricultural features 134. As an example, Figures 25A and 25B show a set of 3D agricultural features 130 on a 3D agricultural feature group 132 corresponding to multiple viewpoints, and a set of mega agricultural features 134 on a mega agricultural feature group 136. The mega agricultural feature group 136 can be formed by merging the set of mega agricultural features 134 with the mega point cloud 116 generated in the point cloud registration step S6.

[0181] In a preferred embodiment, the agricultural feature registration module 1341 is used to combine / align a set of three-dimensional agricultural features 130 by determining one or more spatial transformations (e.g., scaling, rotation, and translation) that align the set of three-dimensional agricultural features 130. For example, similar to the point cloud registration step S6 and the cut point registration step S9, the agricultural feature registration step S13 may be performed based on one or more assumptions, including that the horizontal frame 16 is exactly horizontal and oriented correctly, and that the physical distance between each viewpoint (e.g., viewpoints 0-6) is a predetermined value. Based on one or more such assumptions, it may be sufficient to perform translation along the X-axis (the axis of the horizontal frame 16) to obtain a set of mega-agricultural features 134. In a preferred embodiment, a 4x4 transformation matrix can be used to transform individual sets of three-dimensional agricultural features 130 from one viewpoint to another, such that each element of the transformation matrix represents translation and rotation information. For example, to complete the agricultural feature registration step S13 and generate a set of mega-agricultural features 134, each of the three-dimensional agricultural features 130 can be sequentially transformed using a 4x4 transformation matrix.

[0182] The set of 3D agricultural features 130 is generated based on images taken from different viewpoints (e.g., viewpoints 0-6 in Figures 5A and 5B). Therefore, even after one or more spatial transformations are performed in the agricultural feature registration step S13 with the intention of aligning the set of 3D agricultural features 130, the set of 3D agricultural features 130 may not be perfectly aligned with one another. Accordingly, in order to identify 3D agricultural features 130 that belong to different sets of agricultural features but represent the same agricultural feature, i.e., 3D agricultural features 130 that represent the same agricultural feature but are still slightly misaligned with one another even after the set of 3D agricultural features 130 has been transformed, each of the 3D agricultural features 130 is assigned a search radius (e.g., approximately 4 cm). When a set of 3D agricultural features 130 is combined / aligned to generate a set of mega agricultural features 134, the search radius of the 3D agricultural features 130 is used to determine whether one or more other 3D agricultural features 130 from another set of 3D agricultural features 130 are located within the search radius of that 3D agricultural feature 130. If one or more other 3D agricultural features 130 are located within the search radius of that 3D agricultural feature 130, then that 3D agricultural feature 130 and the one or more other 3D agricultural features 130 are merged into a mega agricultural feature 134 included in the set of mega agricultural features 134.

[0183] In a preferred embodiment of the present invention, two or more 3D agricultural features 130 from different sets of 3D agricultural features must be merged to generate one mega-agricultural feature 134. For example, if, when a set of agricultural features 130 is merged / aligned, there are no other 3D agricultural features 130 from another set of 3D agricultural features 130 that are located within the search radius of a certain 3D agricultural feature 130, then the mega-agricultural feature 134 is not generated. As another example, three or more 3D agricultural features 130 from different sets of 3D agricultural features 130 are merged to generate one mega-agricultural feature 134. Alternatively, the mega-agricultural feature 134 may be generated based on a single 3D agricultural feature 130 even if, when a set of 3D agricultural features 130 is merged / aligned, there are no other 3D agricultural features 130 from another set of 3D agricultural features 130 that are located within the search radius of that 3D agricultural feature 130.

[0184] In a preferred embodiment of the present invention, the image captured in imaging step S1, the disparity map 46, the segmented image 51, the feature image 94, the point cloud 49, the mega point cloud 116, the cutting point image 106, the 3D cutting point cloud 112, the mega cutting point cloud 117, the 3D agricultural feature group 132, and the mega agricultural feature group 136, or a portion thereof, can be stored as a data structure for performing the various steps described above. However, one or more of the image captured in imaging step S1, the disparity map 46, the segmented image 51, the feature image 94, the point cloud 49, the mega point cloud 116, the cutting point image 106, the 3D cutting point cloud 112, the mega cutting point cloud 117, the 3D agricultural feature group 132, and the mega agricultural feature group 136, or a portion thereof, can also be displayed to the user, for example, on a display device 43 or via a user platform 45.

[0185] In a preferred embodiment of the present invention, the operation step S11 can be performed (for example, automatically) after the megacut points 115 have been generated (for example, after the megacut points 115 have been generated in the cut point registration step S9 and merged with the mega point cloud 116 generated in the point cloud registration step S6 to form the megacut point cloud 117). For example, the cutting system 1 can be set to a fully automatic mode in which the cutting system 1 automatically performs the operation step S11 after the megacut points 115 or a set of megacut points 115 have been generated. However, in another preferred embodiment of the present invention, the cutting system 1 can be set to a semi-automatic mode in which a proposed cut point or proposed cutting plane is displayed to the user before the operation step S11 is performed, as will be described in more detail below.

[0186] In a preferred embodiment of the present invention, in semi-automatic mode, before the operation step S11 is performed, the cut points (e.g., megacut points 115) can be displayed to the user as proposed cut points. For example, the set of megacut points 115 can be displayed as a set of proposed cut points by displaying to the user a megacut point group 117 that can be formed by merging the set of megacut points 115 with the megapoint group 116.

[0187] Figures 26A and 26B show an example of the user platform 45 when the cutting system is in semi-automatic mode. The display device 45b displays a mega-cut point group 117, which includes a set of mega-cut points 115, as a set of proposed cut points. The input device 45a of the user platform 45 allows the user to select one of the proposed cut points (one of the mega-cut points 115). For example, if the input device 45a and display device 45b of the user platform 45 are implemented as touchscreens, the user can select one of the proposed cut points by pressing a proposed cut point displayed on the display device 45b. Alternatively, one of the proposed cut points can be selected by moving a cursor using button 45a-1. Alternatively, one of the proposed cut points may already be selected when the display device 45b displays the mega-cut point group 117 to the user.

[0188] In a preferred embodiment, a portion of the display device 45b may indicate which of the proposed cutting points is currently selected. For example, a portion of the display device 45b may indicate "Cutting point 1 is being selected." In the example shown in Figures 26A and 26B, one of the proposed cutting points (the first mega cutting point 115-1) is selected using the input device 45a of the user platform 45, and the display device 45b displays "Cutting point 1 is being selected."

[0189] When one of the proposed cutting points is selected, the input device 45a allows the user to confirm the selected proposed cutting point by pressing the confirmation button 45a-2 (for example, to confirm that the proposed cutting point is properly positioned), delete the selected proposed cutting point by pressing the delete button 45a-4, or modify the selected proposed cutting point by pressing the modify button 45-3.

[0190] When the user chooses to modify a proposed cutting point by pressing the modify button 45-3, the input device 45a allows the user to move the proposed cutting point to a new desired position. For example, the user can move the proposed cutting point to a new desired position using button 45a-1. Alternatively, if the input device 45a and the display device 45b are implemented as a touchscreen, the user can move the proposed cutting point to a new desired position by pressing a point in the mega-cutting point group 117 displayed on the display device 45b that corresponds to the new desired position. Once the user has moved the proposed cutting point to a new desired position, the mega-cutting point group 117 displayed on the display device 45b is updated to show the proposed cutting point moved to the new desired position, allowing the user to visually confirm the new desired position or make further adjustments to it. After the user has moved the proposed cutting point to a new desired position, the user can confirm / OK the new desired position by pressing the confirmation button 45a-2. When the user confirms / OKs the new desired position by pressing the confirmation button 45a-2, the display device 45b may prompt the user to input one or more comments using the input device 45a explaining why the user decided to move the proposed cutting point to the new desired position.

[0191] In a preferred embodiment of the present invention, the user platform 45 may prompt the user to confirm, modify, or delete each of the proposed cutting points (e.g., each of the mega cutting points 115 displayed on the display device 45b) (using the confirmation button 45-a2). For example, as described above, one of the proposed cutting points (e.g., the first mega cutting point 115-1 shown in Figures 26A and 26B) may already be selected when the display device 45b displays the mega cutting point group 117 to the user. When the user confirms, modifies, or deletes one of the selected proposed cutting points (e.g., the first mega cutting point 115-1), the next proposed cutting point (e.g., the second mega cutting point 115-2 shown in Figures 26A and 26B) can be automatically selected for the user to confirm, modify, or delete. This process can be repeated until each of the proposed cutting points (e.g., each of the mega cutting points 115 displayed on the display device 45b) has been confirmed, modified, or deleted. Alternatively, the input device 45a of the user platform 45 may allow the user to select specific (specific megacut points 115) from among the proposed cut points to modify and / or delete, in which case the proposed cut points that are not modified and / or deleted are visually confirmed by the user by looking at the megacut point group 117 displayed on the display device 45b.

[0192] The input device 45a of the user platform 45 allows the user to confirm that each of the proposed cutting points displayed on the display device 45b is acceptable to the user after it has been checked (e.g., using confirmation buttons 45a-2), modified, deleted, or visually confirmed. For example, the user can confirm that each of the cutting points displayed on the display device 45b is acceptable to the user by pressing confirmation buttons 45-2 two or three times. Once the input device 45a of the user platform 45 receives confirmation that each of the cutting points displayed on the display device 45b is acceptable to the user, the cutting points (e.g., a set of mega cutting points 115) can be updated. The updated cutting points (e.g., an updated set of mega cutting points 115) can be used to perform the operation step S11 shown in Figure 4.

[0193] In a preferred embodiment of the present invention, one or more agricultural features detected using the object detection model 92 can be displayed to the user when the proposed cut point is displayed to the user. For example, as shown in Figures 26A and 26B, a set of mega agricultural features 134 can be displayed on the display device 45b when the proposed cut point is displayed to the user. The set of mega agricultural features 134 can be displayed on the display device 45b using dots or marks. By displaying the mega agricultural features 134 on the display device 45b using dots or marks, the user can more easily identify one or more agricultural features.

[0194] In the preferred embodiments of the present invention described above with respect to Figures 26A and 26B, suggested cutting points (e.g., mega cutting points 115) are displayed to the user using dots on the display device 45b. However, in another preferred embodiment, suggested cutting planes 118 can be displayed on the display device 45b in semi-automatic mode, as shown, for example, in Figures 27A and 27B. The set of suggested cutting planes 118 can be generated, for example, based on a set of mega cutting points 115 and mega cutting point angles corresponding to the set of mega cutting points 115. The suggested cutting planes 118 include suggested cutting points based on the mega cutting points 115 (points where the cutting plane 118 intersects with components of the crop) and suggested angles / orientations to which the blade portion 24b of the cutting tool 24 is directed when the cutting operation is performed at the suggested cutting points (mega cutting points 115).

[0195] Figures 27A and 27B show an example of the user platform 45 when the cutting system is in semi-automatic mode and the display device 45b displays a set of suggested cutting planes 118. The input device 45a of the user platform 45 allows the user to select one of the suggested cutting planes 118. For example, if the input device 45a and the display device 45b are implemented as a touchscreen, the user can select one of the suggested cutting planes 118 by pressing the suggested cutting plane displayed on the display device 45b. Alternatively, the user can use button 45a-1 to move the cursor and select one of the suggested cutting planes 118. Alternatively, one of the suggested cutting planes may already be selected when the display device 45b displays the suggested cutting planes 118.

[0196] In a preferred embodiment, a portion of the display device 45b can indicate which of the proposed cutting planes 118 is currently selected. For example, a portion of the display device 45b can indicate "Cutting plane 1 is being selected." In the example shown in Figures 27A and 27B, one of the proposed cutting planes (the first cutting plane 118-1 corresponding to the mega cutting point 115-1) is selected using the input device 45a of the user platform 45, and the display device 45b displays "Cutting plane 1 is being selected."

[0197] When one of the proposed cutting planes is selected, the input device 45a allows the user to confirm the selected proposed cutting plane by pressing the confirmation button 45a-2 (for example, to confirm that the proposed cutting plane is properly positioned and oriented), delete the selected proposed cutting plane by pressing the delete button 45a-4, or modify the selected proposed cutting plane by pressing the modify button 45-3.

[0198] If the user chooses to modify the proposed cross-section by pressing the modify button 45-3, the input device 45a allows the user to change the proposed cross-section to a new desired position and / or a new desired orientation. For example, the user can change the proposed cross-section to a new desired position and / or a new desired orientation using button 45a-1. Alternatively, if the input device 45a and the display device 45b are implemented as a touchscreen, the user can change the proposed cross-section to a new desired position and / or a new desired orientation by pressing a point in the mega-cross-section point group 117 displayed on the display device 45b that corresponds to the new desired position, and / or by manipulating the proposed cross-section (for example, rolling the cross-section by rotating it around the front-to-back axis, pitching the cross-section by rotating it around the left-to-right axis, or yawing the cross-section by rotating it around the vertical axis). When the user changes the proposed cutting plane to a new desired position and / or a new desired orientation, the mega-cutting point group 117 displayed on the display device 45b is updated to show the proposed cutting plane moved to the new desired position and / or a new desired orientation, allowing the user to visually confirm the new desired position and / or a new desired orientation, or to further adjust the new desired position and / or orientation. When the user adjusts the proposed cutting plane to a new desired position and / or a new desired orientation, the user can confirm / OK the new desired position and / or a new desired orientation by pressing the confirmation buttons 45a-2. When the user confirms / OKs the new desired position and / or a new desired orientation by pressing the confirmation buttons 45a-2, the display device 45b may prompt the user to enter one or more comments using the input device 45a explaining why the user decided to move the proposed cutting plane to a new desired position or a new desired orientation.

[0199] In a preferred embodiment of the present invention, the user platform 45 may prompt the user to confirm, modify, or delete each of the suggested cross-sections 118 (using confirmation buttons 45a-2). For example, when the display device 45b displays the suggested cross-sections to the user, one of the suggested cross-sections (e.g., the first cross-section 118-1 shown in Figures 27A and 27B) may already be selected. When the user confirms, modifies, or deletes one of the selected suggested cross-sections (e.g., the first cross-section 118-1 shown in Figures 27A and 27B), the next suggested cross-section (e.g., the second cross-section 118-2 shown in Figures 27A and 27B) can be automatically selected for the user to confirm, modify, or delete. This process can be repeated until each of the suggested cross-sections 118 displayed on the display device 45b has been confirmed, modified, or deleted. Alternatively, the input device 45a of the user platform 45 may allow the user to select a particular of the proposed cross-sections 118 to modify and / or delete, in which case the proposed cross-sections 118 that are not modified and / or deleted are visually confirmed by the user looking at the display device 45b.

[0200] The input device 45a of the user platform 45 allows the user to confirm, for example, that each of the proposed cutting surfaces displayed on the display device 45b is acceptable to the user after it has been checked (e.g., using confirmation button 45a-2), modified, deleted, or visually confirmed. For example, the user can confirm that each of the cutting surfaces is acceptable to the user by pressing confirmation button 45-2 two or three times. Once the input device 45a of the user platform 45 receives confirmation that each of the cutting surfaces 118 is acceptable to the user, it can update the cutting points (e.g., a set of mega cutting points 115) and the cutting angles (e.g., mega cutting point angles corresponding to the set of mega cutting points 115). The updated cutting points and cutting point angles can be used to perform the operation step S11 shown in Figure 4.

[0201] In a preferred embodiment of the present invention, the cutting system 1 can be switched from a semi-automatic mode to a fully automatic mode. For example, the cutting system 1 can be switched from a semi-automatic mode to a fully automatic mode based on the approval rate (user approval rate) of the proposed cutting point or proposed cutting surface, as will be described in more detail below.

[0202] In a preferred embodiment of the present invention, the cutting system 1 can record the rate at which suggested cutting points displayed to the user are confirmed (e.g., by confirmation using confirmation buttons 45a-2 or visual confirmation using a display device 45b), modified, and deleted. The approval rate of suggested cutting points (user approval rate) may include the rate at which suggested cutting points are confirmed without being modified or deleted. For example, if 10 suggested cutting points are displayed to the user and the user confirms 7 of the 10 suggested cutting points (i.e., confirms 7 suggested cutting points without modifying or deleting them), the cutting system 1 records an approval rate of 70%.

[0203] In a preferred embodiment of the present invention, the cutting system 1 can automatically switch from semi-automatic mode to fully automatic mode when the approval rate is equal to or greater than an approval rate threshold. For example, the cutting system 1 can automatically switch from semi-automatic mode to fully automatic mode when the approval rate is equal to or greater than an approval rate threshold of, for example, 70% or 80%.

[0204] In another preferred embodiment, the disconnection system 1 can automatically switch from semi-automatic mode to fully automatic mode when the approval rate is above an approval rate threshold and the user has reviewed (confirmed, modified, or deleted) a predetermined number of suggested disconnection points. For example, the disconnection system 1 can automatically switch from semi-automatic mode to fully automatic mode when the approval rate is above an approval rate threshold of, for example, 70% or 80% and the user has reviewed at least 20 suggested disconnection points.

[0205] In another preferred embodiment of the present invention, the user platform 45 can be shown to the user when the approval rate is above an approval rate threshold. Furthermore, the input device 45a of the user platform 45 can be used to allow the user to input a command to switch the cutting system 1 from semi-automatic mode to fully automatic mode. For example, as shown in Figures 26A and 26B, the user platform 45 can be shown to the user when the approval rate is above an approval rate threshold, and the input device 45a can receive a command from the user to switch the cutting system 1 from semi-automatic mode to fully automatic mode using buttons 45a-6.

[0206] In another preferred embodiment of the present invention, the buttons 45a-6 of the input device 45a can be used to receive a command from the user to switch the cutting system 1 from semi-automatic mode to fully automatic mode, regardless of whether the approval rate is above or below an approval rate threshold. In a preferred embodiment, the buttons 45a-6 of the input device 45a can also be used to switch the cutting system 1 from fully automatic mode to semi-automatic mode. In other words, the input device 45a (buttons 45a-6) can be used to switch between semi-automatic mode and fully automatic mode, i.e., from semi-automatic mode to fully automatic mode, and vice versa.

[0207] In another preferred embodiment, the user platform 45 may indicate to the user when the approval rate is equal to or greater than an approval rate threshold, and may receive a command from the user to switch the disconnection system 1 from semi-automatic mode to fully automatic mode only when the approval rate is equal to or greater than an approval rate threshold and the user has reviewed (confirmed, modified, or deleted) a predetermined number of suggestion disconnections. For example, the user platform 45 may not indicate when the approval rate is equal to or greater than an approval rate threshold, or it may not receive a command from the user to switch the disconnection system 1 from semi-automatic mode to fully automatic mode, until the approval rate is equal to or greater than an approval rate threshold of, for example, 70% or 80% and the user has reviewed at least 20 suggestion disconnections.

[0208] Similarly, in the preferred embodiments of the present invention described above, the cutting system 1 can record the rate at which suggested cutting surfaces displayed to the user are confirmed (e.g., confirmed by confirmation buttons 45a-2 or visually confirmed using the display device 45b), modified, and deleted. The approval rate of suggested cutting surfaces (user approval rate) may include the rate at which suggested cutting surfaces are confirmed without being modified or deleted. For example, if 10 suggested cutting surfaces are displayed to the user and the user confirms 7 of those 10 suggested cutting surfaces (i.e., confirms 7 of the suggested cutting surfaces without modifying or deleting them), the cutting system 1 records an approval rate of 70%.

[0209] In a preferred embodiment of the present invention, the cutting system 1 can automatically switch from semi-automatic mode to fully automatic mode when the approval rate of the proposed cutting surfaces is equal to or greater than an approval rate threshold. For example, the cutting system 1 can automatically switch from semi-automatic mode to fully automatic mode when the approval rate is equal to or greater than an approval rate threshold of, for example, 70% or 80%. In another preferred embodiment, the cutting system 1 can automatically switch from semi-automatic mode to fully automatic mode when the approval rate of the proposed cutting surfaces is equal to or greater than an approval rate threshold and the user has reviewed (confirmed, modified, or deleted) a predetermined number of proposed cutting surfaces. For example, the cutting system 1 can automatically switch from semi-automatic mode to fully automatic mode when the approval rate is equal to or greater than an approval rate threshold of, for example, 70% or 80% and the user has reviewed at least 20 proposed cutting surfaces.

[0210] In another preferred embodiment of the present invention, the user platform 45 can indicate to the user when the approval rate of the proposed cut surface is equal to or greater than an approval rate threshold. Furthermore, the input device 45a of the user platform 45 allows the user to input a command to switch the cutting system 1 from semi-automatic mode to fully automatic mode. For example, as shown in Figures 27A and 27B, the user platform 45 can indicate to the user when the approval rate is equal to or greater than an approval rate threshold, and can receive a command from the user to switch the cutting system 1 from semi-automatic mode to fully automatic mode using buttons 45a-6 on the input device 45a.

[0211] In another preferred embodiment, the user platform 45 may only indicate to the user that the approval rate of a proposed section is above the approval rate threshold and / or receive a command from the user to switch the cutting system 1 from semi-automatic mode to fully automatic mode, if the approval rate is above the approval rate threshold and the user has reviewed (confirmed, modified, or deleted) a predetermined number of proposed sections. For example, the user platform 45 may not indicate that the approval rate is above the approval rate threshold and / or receive a command from the user to switch the cutting system 1 from semi-automatic mode to fully automatic mode until the approval rate is above the approval rate threshold and the user has reviewed (confirmed, modified, or deleted) a predetermined number of proposed sections. For example, the user platform 45 may not indicate that the approval rate is above the approval rate threshold and / or receive a command from the user to switch the cutting system 1 from semi-automatic mode to fully automatic mode until the approval rate is above an approval rate threshold of, for example, 70% or 80% and the user has reviewed at least 20 proposed sections.

[0212] In the preferred embodiments of the present invention described above with respect to Figures 26A, 26B, 27A, and 27B, when the cutting system is set to a semi-automatic mode in which a suggested cutting point or suggested cutting plane is displayed to the user, the input device 45a of the user platform 45 may allow the user to change or switch the rules used to generate the suggested cutting point or suggested cutting plane. For example, as shown in Figures 26A and 26B, the input device 45a of the user platform 45 may include setting buttons 45a-8 that, when pressed, change the rules used to generate the suggested cutting point (e.g., change the rules for the cutting point generation step S18-3). For example, when pressed, the setting buttons 45a-8 may change the rules used to generate the suggested cutting point so that the suggested cutting point is generated between a second agricultural feature and a third agricultural feature on a particular segment / individual component of the target crop, rather than between a first agricultural feature and a second agricultural feature on a particular segment / individual component of the target crop. Similarly, in the examples shown in Figures 27A and 27B, the input device 45a of the user platform 45 may include setting buttons 45a-8 that, when pressed, can change the rules used to generate the suggested cross-section. For example, when pressed, the setting buttons 45a-8 can change the rules used to generate the suggested cross-section so that, instead of oriented the suggested cross-section 90 degrees to a particular segment / individual component of the target crop, it is oriented 75 degrees to a particular segment / individual component of the target crop.

[0213] In the preferred embodiments of the present invention described above with respect to Figures 26A, 26B, 27A, and 27B, the cutting system 1 can be set to a semi-automatic mode in which a suggested cutting point or suggested cutting plane is displayed to the user before the operation step S11 is performed. In another preferred embodiment of the present invention, the cutting system 1 can be set to a manual mode in which a new cutting point and / or new cutting plane can be created using a user platform, as will be described in more detail below.

[0214] Figures 28A and 28B show an example of the user platform 45 when the cutting system 1 is set to manual mode. In manual mode, the display device 45b can display crops. For example, the display device 45b can display a mega-agricultural feature group 136. As described above, the mega-agricultural feature group 136 can be formed by merging a set of mega-agricultural features 134 with a mega-point group 116. In manual mode, the input device 45a of the user platform 45 allows the user to create a new cutting point 119, for example, by pressing an add button 45-a5. When the add button 45-a5 is pressed, the user can specify the location of the new cutting point. For example, if the input device 45a and display device 45b of the user platform 45 are implemented as touchscreens, the user can specify the location of the new cutting point by pressing the touchscreen at the point corresponding to the location of the new cutting point. Alternatively, the user can specify the location of the new cutting point by moving the cursor using button 45a-1.

[0215] When a new cut point location is specified, the new cut point 119 is displayed on the display device 45b at the specified location. Furthermore, a portion of the display device 45b can indicate that the new cut point is currently selected. For example, a portion of the display device 45b can display "Cut point 1 selected". In Figures 28A and 28B, a new cut point (the first new cut point 119-1) has been created using the input device 45a of the user platform 45, and the display device 45b displays "Cut point 1 selected". When a new cut point is selected, the input device 45a allows, for example, the user to move / adjust the location of the new cut point. The user can move / adjust the location of the new cut point using button 45a-1. Alternatively, if the input device 45a and the display device 45b are implemented as a touchscreen, the user can move / adjust the location of the new cut point by pressing a point on the touchscreen that corresponds to the new desired location of the new cut point. When the user moves a new cutting point to a new desired position, the display on the display device 45b is updated to show the new cutting point moved to the new desired position, allowing the user to visually confirm the new desired position and make further adjustments. If the position of the new cutting point is acceptable to the user, they can press the confirmation button 45a-2 to confirm / OK the position of the new cutting point.

[0216] In a preferred embodiment of the present invention, the input device 45a of the user platform 45 allows the user to select one of the new cut points 119 after a new cut point has been created. For example, if the input device 45a and the display device 45b are implemented as a touchscreen, the user can select one of the new cut points by pressing a new cut point displayed on the display device 45b. Alternatively, the user can select one of the new cut points by moving a cursor using button 45a-1. When one of the new cut points is selected, the input device 45a allows the user to confirm the selected new cut point by pressing a confirmation button 45a-2 (for example, to confirm that the new cut point is properly positioned), delete the selected new cut point by pressing a delete button 45a-4, or modify the selected new cut point by pressing a modify button 45-3, in the same manner as described above with respect to Figures 26A and 26B.

[0217] In a preferred embodiment, the input device 45a of the user platform 45 allows the user to confirm that each of the new cut points 119 displayed on the display device 45b is acceptable to the user and that the user does not want to add any additional new cut points. For example, the user can confirm that each of the new cut points displayed on the display device 45b is acceptable to the user and that the user does not want to add any additional new cut points by pressing the confirmation button 45-2 two or three times. Once the input device 45a of the user platform 45 confirms that each of the new cut points displayed on the display device 45b is acceptable to the user and that the user does not want to add any additional new cut points, it can perform the operation step S11 shown in Figure 4 using the new cut points.

[0218] In a preferred embodiment of the present invention, one or more agricultural features detected using the object detection model 92 can be displayed to the user. For example, as shown in Figures 28A and 28B, a set of mega agricultural features 134 can be displayed on the display device 45b. For example, the set of mega agricultural features 134 can be displayed on the display device 45b using dots or marks, thereby making it easier for the user to identify one or more agricultural features.

[0219] In the preferred embodiments of the present invention described above with respect to Figures 28A and 28B, the user platform can be used to create a new cutting point when the cutting system 1 is set to manual mode. In another preferred embodiment of the present invention, as will be described in more detail below, the user platform can be used to create a new cutting plane when the cutting system 1 is set to manual mode. The new cutting plane 121 includes a new cutting point (the point where the new cutting plane 121 intersects with a component of the crop) and a new angle / orientation in which the blade portion 24b of the cutting tool 24 is directed when the cutting operation is performed at the new cutting point.

[0220] Figures 29A and 29B show an example of the user platform 45 when the cutting system is set to manual mode. The display device 45b displays the mega-agricultural feature group 136. As described above, the mega-agricultural feature group 136 can be formed by merging the set of mega-agricultural features 134 and the mega-point cloud 116. The input device 45a of the user platform 45 allows the user to create a new cutting plane 121, for example by pressing the add button 45-a5. By pressing the add button 45-a5, the user can specify the position and orientation of the new cutting plane. For example, if the input device 45a and the display device 45b are implemented as a touchscreen, the user can specify the position of the new cutting plane by pressing the touchscreen at a point corresponding to the position of the new cutting plane. Alternatively, the position of the new cutting plane can be specified by moving the cursor using button 45a-1. The initial angle / orientation of the new cutting plane can be a predetermined angle / orientation relative to a specific segment / individual component of the target crop, or it can be set using the input device 45a.

[0221] When a new cutting surface location is specified, the new cutting surface 121 is displayed on the display device 45b at the specified location and initial angle / orientation. Furthermore, a portion of the display device 45b can indicate that a new cutting surface is currently selected. For example, a portion of the display device 45b can indicate that "Cutting surface 1 is being selected." In Figures 29A and 29B, a new cutting surface (the first new cutting surface 121-1) has been created using the input device 45a of the user platform 45, and the display device 45b indicates that "Cutting surface 1 is being selected." The input device 45a allows the user to move / adjust the position and / or angle / orientation of the new cutting surface when a new cutting surface is selected. For example, the user can move / adjust the position and / or angle / orientation of the new cutting surface 121 using button 45a-1. Alternatively, if the input device 45a and the display device 45b are implemented as touchscreens, the user can move / adjust the position and / or angle / orientation of the new cutting surface by pressing the touchscreen at a point corresponding to the new desired position of the new cutting surface and manipulating the new cutting surface (for example, rolling the cutting surface by rotating it around its front-to-back axis, pitching it by rotating it around its left-to-right axis, and yawing it by rotating it around its vertical axis). When the user moves the new cutting surface to the new desired position and / or angle / orientation, the display on the display device 45b is updated to show the new cutting surface moved to the new desired position and angle / orientation, allowing the user to visually confirm the new desired position and angle / orientation, or to further adjust the new cutting surface. If the position and angle / orientation of the new cutting surface is acceptable to the user, the user can press the confirmation button 45a-2 to confirm / OK the position and angle / orientation of the new cutting surface.

[0222] In a preferred embodiment of the present invention, the input device 45a of the user platform 45 allows the user to select one of the new cutting surfaces 121 after a new cutting surface has been created. For example, if the input device 45a and the display device 45b are implemented as a touchscreen, the user can select one of the new cutting surfaces by pressing a new cutting surface displayed on the display device 45b. Alternatively, the user can select one of the new cutting surfaces by moving a cursor using button 45a-1. When one of the new cutting surfaces is selected, the input device 45a allows the user to confirm the selected new cutting surface by pressing confirmation button 45a-2 (for example, to confirm that the new cutting surface is properly positioned and oriented), delete the selected new cutting surface by pressing delete button 45a-4, or modify the selected new cutting surface by pressing modify button 45-3, in the same manner as described above with respect to Figures 27A and 27B.

[0223] In a preferred embodiment, the input device 45a of the user platform 45 allows the user to confirm that each of the new cross-sections 121 displayed on the display device 45b is acceptable to the user and that the user does not wish to add any additional new cross-sections. For example, the user can confirm that each of the new cross-sections displayed on the display device 45b is acceptable to the user and that the user does not wish to add any additional new cross-sections by pressing the confirmation button 45-2 two or three times. Once the input device 45a of the user platform 45 receives confirmation that each of the new cross-sections displayed on the display device 45b is acceptable to the user and that the user does not wish to add any additional new cross-sections, it can use the new cross-sections to perform the operation step S11 shown in Figure 4.

[0224] In a preferred embodiment of the present invention, one or more agricultural features detected using the object detection model 92 can be displayed to the user. For example, as shown in Figures 29A and 29B, a set of mega agricultural features 134 can be displayed on the display device 45b. For example, the set of mega agricultural features 134 can be displayed on the display device 45b using dots or marks, thereby making it easier for the user to identify one or more agricultural features.

[0225] In a preferred embodiment of the present invention, when the cutting system 1 is set to manual mode and a new cutting point and / or new cutting plane is created using the user platform 45, the user platform can display a suggested cutting point or suggested cutting plane after it has been used to create the new cutting point or new cutting plane.

[0226] In the examples shown in Figures 28A and 28B, if the user confirms using the input device 45a of the user platform 45 that each of the new cut points 119 displayed on the display device 45b is acceptable to the user and that the user does not want to add any additional new cut points (for example, if the user presses the confirmation button 45-2 two or three times), the display device 45b can display one or more suggested cut points (for example, automatically). For example, in Figures 28A and 28B, after a new cut point 119 (new cut point 119-1) is created using the user platform, the display device 45b can display a mega cut point 115 (mega cut point 115-1) as a suggested cut point. In this way, the user can create a new cut point 119 (new cut point 119-1) using the user platform without seeing or being affected by the suggested cut point (mega cut point 115-1), and after the new cut point 119 is created, the new cut point 119 can be visually compared with the suggested cut point. Alternatively, a command to display one or more suggested cutting points 119 can be entered using the input device 45a of the user platform 45. For example, the input device 45a of the user platform 45 may include suggestion buttons 45a-7 that can be used to enter a command to display one or more suggested cutting points. For example, in Figures 28A and 28B, the display device 45b can display mega cutting point 115 (mega cutting point 115-1) as a suggested cutting point when the suggestion button 45a-7 is pressed.

[0227] Similarly, in the examples shown in Figures 29A and 29B, when the user confirms using the input device 45a of the user platform 45 that each of the new cross-sections 121 displayed on the display device 45b is acceptable to the user and that the user does not want to add any additional new cross-sections (for example, when the user presses the confirmation button 45-2 two or three times), the display device 45b can display one or more suggested cross-sections 118 (for example, automatically). For example, in Figures 29A and 29B, after a new cross-section 121 (new cross-section 121-1) is created using the user platform, the display device 45b can display a suggested cross-section 118-1. In this way, the user can create a new cross-section 121 using the user platform without seeing or being influenced by the suggested cross-section 118-1, and after the new cross-section 121 (new cross-section 121-1) is created, the new cross-section 121 (new cross-section 121-1) can be visually compared with the suggested cross-section 118. Alternatively, a command to display one or more proposed cross-sections 118 can be entered using the input device 45a of the user platform 45. For example, the input device 45a of the user platform 45 may include suggestion buttons 45a-7 that can be used to enter a command to display one or more proposed cross-sections 118. For example, in Figures 29A and 29B, the display device 45b can display a proposed cross-section 118-1 when the suggestion button 45a-7 is pressed.

[0228] In a preferred embodiment of the present invention, the cutting system 1 can be switched from manual mode to fully automatic mode. For example, the cutting system 1 can be switched from manual mode to fully automatic mode based on the prediction rate of a new cutting point or a new cutting surface.

[0229] For example, in the preferred embodiment of the present invention described above, the cutting system 1 can record the percentage of new cutting points created by the user in manual mode that coincide with a proposed cutting point (e.g., mega cutting point 115). For example, if a new cutting point created by the user in manual mode is within a predetermined distance from a proposed cutting point (e.g., mega cutting point 115), it can be determined that the new cutting point created by the user in manual mode coincides with the proposed cutting point (e.g., mega cutting point 115). If 10 new cutting points are created by the user in manual mode, and 7 of the 10 new cutting points coincide with the proposed cutting point (e.g., mega cutting point 115), the cutting system 1 records a prediction rate of 70%.

[0230] In a preferred embodiment of the present invention, the cutting system 1 can automatically switch from manual mode to fully automatic mode when the prediction rate is equal to or greater than a prediction rate threshold. For example, the cutting system 1 can automatically switch from manual mode to fully automatic mode when the prediction rate is equal to or greater than a prediction rate threshold of, for example, 70% or 80%.

[0231] In another preferred embodiment, the cutting system 1 can automatically switch from manual mode to fully automatic mode when the prediction rate is greater than or equal to a prediction rate threshold and the user has created a predetermined number of new cutting points. For example, the cutting system 1 can automatically switch from manual mode to fully automatic mode when the prediction rate is greater than or equal to a prediction rate threshold of, for example, 70% or 80% and the user has created at least 20 new cutting points.

[0232] In another preferred embodiment of the present invention, the user platform 45 can indicate to the user when the prediction rate is above a prediction rate threshold. Furthermore, the input device 45a of the user platform 45 allows the user to input a command to switch the cutting system 1 from manual mode to fully automatic mode. For example, as shown in Figures 28A and 28B, the user platform 45 can indicate to the user when the prediction rate is above a prediction rate threshold, and can receive a command from the user to switch the cutting system 1 from manual mode to fully automatic mode using buttons 45a-6 on the input device 45a.

[0233] In another preferred embodiment of the present invention, the input device 45a can receive a command from the user to switch the cutting system 1 from manual mode to fully automatic mode using buttons 45a-6 of the input device 45a, regardless of whether the prediction rate is above or below a prediction rate threshold. In a preferred embodiment, buttons 45a-6 of the input device 45a can also be used to switch the cutting system 1 from fully automatic mode to manual mode. In other words, the input device 45a (buttons 45a-6) can be used to switch between manual mode and fully automatic mode, i.e., from manual mode to fully automatic mode, and vice versa.

[0234] In another preferred embodiment, the user platform 45 may indicate to the user that the prediction rate is above the prediction rate threshold and / or receive a command from the user to switch the cutting system 1 from manual mode to fully automatic mode only when the prediction rate is above the prediction rate threshold and the user has created a predetermined number of new cutting points. For example, the user platform 45 may not indicate that the prediction rate is above the prediction rate threshold and / or receive a command from the user to switch the cutting system 1 from manual mode to fully automatic mode until the prediction rate is above the prediction rate threshold and the user has created a predetermined number of new cutting points. For example, the user platform 45 may not indicate that the prediction rate is above the prediction rate and / or receive a command from the user (e.g., via buttons 45a-6) to switch the cutting system 1 from manual mode to fully automatic mode until the prediction rate is above a prediction rate threshold of, for example, 70% or 80% and the user has created at least 20 new cutting points.

[0235] Similarly, in the preferred embodiment of the present invention described above, the cutting system 1 can record a prediction rate that a new cutting surface 121 created by the user in manual mode will match the proposed cutting surface 118. For example, if a new cutting surface created by the user in manual mode includes a cutting point (a point where the new cutting surface intersects with a component of the crop) that is within a predetermined distance from the proposed cutting point of the proposed cutting surface 118, and the new cutting surface 121 includes an angle / orientation that is within a predetermined angle / degree from the proposed angle / orientation of the proposed cutting surface 118, then it can be determined that the new cutting surface 121 created by the user in manual mode will match the proposed cutting surface 118. If 10 new cutting surfaces are created by the user in manual mode, and 7 of the 10 new cutting surfaces match the proposed cutting surface, the cutting system 1 records a prediction rate of 70%.

[0236] In a preferred embodiment of the present invention, the cutting system 1 can automatically switch from manual mode to fully automatic mode when the prediction rate is equal to or greater than a prediction rate threshold. For example, the cutting system 1 can automatically switch from manual mode to fully automatic mode when the prediction rate is equal to or greater than a prediction rate threshold of, for example, 70% or 80%.

[0237] In another preferred embodiment, the cutting system 1 can automatically switch from manual mode to fully automatic mode when the prediction rate is above a prediction rate threshold and the user has created a predetermined number of new cut surfaces 121. For example, the cutting system 1 can automatically switch from manual mode to fully automatic mode when the prediction rate is above a prediction rate threshold of, for example, 70% or 80% and the user has created at least 20 new cut surfaces.

[0238] In another preferred embodiment of the present invention, the user platform 45 can be shown to the user when the prediction rate is above a prediction rate threshold. Furthermore, the input device 45a of the user platform 45 allows the user to input a command to switch the cutting system 1 from manual mode to fully automatic mode. For example, as shown in Figures 29A and 29B, the user platform 45 can be shown to the user when the prediction rate is above a prediction rate threshold, and the input device 45a-6 can be used to receive a command from the user to switch the cutting system 1 from manual mode to fully automatic mode.

[0239] In another preferred embodiment, the user platform 45 may indicate to the user that the prediction rate is above a prediction rate threshold and / or receive a command from the user to switch the cutting system 1 from manual mode to fully automatic mode only when the prediction rate is above a prediction rate threshold and the user has created a predetermined number of new cut surfaces 121. For example, the user platform 45 may not indicate that the prediction rate is above a prediction rate threshold and / or receive a command from the user to switch the cutting system 1 from manual mode to fully automatic mode until the prediction rate is above a prediction rate threshold and the user has created a predetermined number of new cut surfaces. For example, the user platform 45 may not indicate that the prediction rate is above a prediction rate threshold and / or receive a command from the user (e.g., via buttons 45a-6) until the prediction rate is above a prediction rate threshold of, for example, 70% or 80% and the user has created at least 20 new cut surfaces.

[0240] In the preferred embodiments of the present invention described above, the processor 45d of the user platform 45 may be configured or programmed to generate and control the information displayed on the display device 45b described above with respect to Figures 26-29, and to receive user input from the input device 45a. Furthermore, the processor 45d of the user platform 45 may be configured or programmed to generate commands for setting the cutting system to manual mode, semi-automatic mode, or fully automatic mode and to transmit them to the imaging electronic system 42 and / or base electronic system 34. However, in other preferred embodiments of the present invention, the display device 43 may similarly be used to display information and to receive input as described above with respect to Figures 26-29. For example, the processor of the display device 43 may be configured or programmed to generate and control the information described above with respect to Figures 26-29, and to receive user input from the input device of the display device 43, for example, using the touchscreen 43a of the display device 43. Furthermore, the processor of the display device 43 may be configured or programmed to generate commands for setting the cutting system to manual mode, semi-automatic mode, or fully automatic mode and to transmit them to the imaging electronic system 42 and / or base electronic system 34.

[0241] In a preferred embodiment of the present invention, the cutting system 1 can move to a waypoint located in front of the target crop (e.g., a grapevine) and perform a cutting operation, as described above. In fully automatic mode, the cutting system 1 can move to a first waypoint located in front of a first target crop, perform a first cutting operation at the first waypoint, and then automatically move to a second waypoint located in front of a second target crop and perform a second cutting operation at the second waypoint. In the semi-automatic and manual modes described above, the cutting system 1 can move to a first waypoint located in front of a first target crop based on a first input received by the input device 45a of the user platform 45 and perform a first cutting operation at the first waypoint. The cutting system can also move to a second waypoint located in front of a second target crop and perform a second cutting operation at the second waypoint based on a second input received by the input device 45a of the user platform 45.

[0242] In a preferred embodiment, as shown in Figure 30A, the cutting system 1 moves to a first waypoint located in front of a first target crop in step S30A-1, images the first target crop (e.g., using a camera 20) in step S30A-2, receives a first input (e.g., an input to confirm, modify, or delete one or more cutting points or cutting surfaces) from the input device 45a of the user platform 45 in step S30A-3, and performs a first cutting operation based on the first input in step S30A-4. However, the cutting system 1 is not limited to this process. For example, as shown in Figure 30B, if the first target crop has been previously imaged in step S30B-1 (e.g., the cutting system 1 has previously imaged the first target crop), the first input can be received by the input device 45a of the user platform 45 in step S30B-2 before the cutting system 1 moves to the first waypoint in step S30B-3 and performs the first cutting operation in step S30B-4. For example, if the cutting system 1 has previously moved through the field and imaged a first target crop and a second target crop, the first and second inputs may be received by the input device 45a of the user platform 45 before the cutting system 1 is moved to the first waypoint to perform the first cutting operation, or before it is moved to the second waypoint to perform the second cutting operation.

[0243] As described above, the processor and memory elements of the imaging electronic system 42 may be configured or programmed to control one or more devices, including the camera 20, the robot arm 22, the robot arm mount assembly 23, and the cutting tool 24, and may also be configured or programmed to process image data obtained by the camera 20. In a preferred embodiment of the present invention, the processor and memory elements of the imaging electronic system 42 may be configured or programmed to perform the functions described above, including a disparity estimation step S2, a component splitting step S3, an agricultural feature detection step S4, a point cloud generation step S5, a point cloud registration step S6, a cut point generation step S7, a cut point projection step S8, a cut point registration step S9, a mega registration step S10, an operation step S11, an agricultural feature projection step S12, and an agricultural feature registration step S13. In other words, the processor and memory elements of the imaging electronic system 42 can be defined, configured, or programmed to function as components including the AI ​​disparity estimation model 44, instance segmentation AI architecture 50, object detection model 92, point cloud generation module 491, point cloud registration module 1161, cutpoint generation module 104, cutpoint projection module 110, cutpoint registration module 1151, trace module 120, agricultural feature projection module 1301, and agricultural feature registration module 1341.

[0244] In the preferred embodiments of the present invention described above, the target crop is a grapevine. However, preferred embodiments of the present invention are also applicable to other target crops such as fruit trees and flowering plants such as rose bushes.

[0245] It should be understood that the foregoing description is merely illustrative of the present invention. Various alternatives and modifications can be devised by those skilled in the art without departing from the present invention. Accordingly, the present invention is intended to encompass all such alternatives, modifications, and variations that fall within the scope of the appended claims.

Claims

1. An input section that receives one or more inputs from the user, The display and A processor operably connected to the input unit and the display, A user platform equipped with, The processor is configured or programmed to control the display to show suggested cutting points for crops, The processor is configured or programmed to verify that the proposed breakpoint is properly positioned, delete the proposed breakpoint, or modify the proposed breakpoint based on one or more inputs received by the input unit. User platform.

2. The processor is configured or programmed to move the proposed disconnect point to a new desired position based on one or more inputs received by the input unit. The user platform according to claim 1.

3. The processor is configured or programmed to control the display to show the proposed break point after the proposed break point has been moved to the new desired position. The processor is configured or programmed to confirm, based on the input received by the input unit, that the proposed disconnection point is appropriately positioned at the new desired location. The user platform according to claim 2.

4. The processor is configured or programmed to receive user comments via the input unit regarding the reason why the proposed disconnection point was moved to the new desired position. The user platform according to claim 3.

5. The processor is configured or programmed to control the display to display a plurality of proposed breakpoints, including the proposed breakpoint. For each of the plurality of proposed breakpoints, the processor is configured or programmed to verify that the proposed breakpoint is properly located, delete the proposed breakpoint, or modify the proposed breakpoint based on one or more inputs received by the input unit. The user platform according to claim 1.

6. The processor is configured or programmed to confirm, based on the input received by the input unit, that each of the plurality of proposed disconnections is acceptable to the user. The user platform according to claim 5.

7. The processor is configured or programmed to control the display to show one or more agricultural characteristics of the crop. The one or more agricultural features of the crop are detected using an object detection model. The user platform according to claim 1.

8. The processor is configured or programmed to control the display to display a proposed cutting plane including the proposed cutting point and the proposed cutting angle or cutting direction. The user platform according to claim 1.

9. The processor is configured or programmed to modify the proposed cutting plane based on one or more inputs received by the input unit, and to change the proposed cutting plane to a new desired position and / or a new desired cutting angle or cutting orientation. The user platform according to claim 8.

10. The processor is configured or programmed to switch the cutting system, which includes the user platform, from a semi-automatic mode in which a plurality of suggested cutting points, including the suggested cutting point, are displayed on the display before the cutting system performs the operation steps, to a fully automatic mode in which the cutting system automatically performs the operation steps. The switching of the cutting system from the semi-automatic mode to the fully automatic mode is performed based on the user's approval rate for the multiple proposed cutting points. The user platform according to claim 1.

11. The processor is configured or programmed to switch the disconnection system from the semi-automatic mode to the fully automatic mode when the user's approval rate is equal to or greater than an approval rate threshold. The user platform according to claim 10.

12. The processor is configured or programmed to switch the cutting system from the semi-automatic mode to the fully automatic mode when the user's approval rate is equal to or greater than an approval rate threshold, and a predetermined number of proposed cutting points have been reviewed by the user via the user platform. The user platform according to claim 10.

13. The processor is configured or programmed to perform control to display a message indicating that the user's approval rate is equal to or greater than an approval rate threshold. The processor is configured or programmed to switch the cutting system from the semi-automatic mode to the fully automatic mode based on the input received by the input unit. The user platform according to claim 10.

14. The processor is configured or programmed to switch the cutting system, which includes the user platform, between a semi-automatic mode in which a plurality of suggested cutting points, including the suggested cutting point, are displayed on the display before the cutting system performs the operation steps, and a fully automatic mode in which the cutting system automatically performs the operation steps. The processor is configured or programmed to switch the cutting system between the semi-automatic mode and the fully automatic mode based on the input received by the input unit. The user platform according to claim 1.

15. The processor is configured or programmed to modify one or more rules used to generate the proposed breakpoints based on the input received by the input unit. The user platform according to claim 1.

16. An input section that receives input from the user, The display and A processor operably connected to the input unit and the display, A user platform equipped with, The processor is configured or programmed to control the display to show agricultural products, The processor is configured or programmed to create a new cutting point or new cutting surface of the crop based on one or more inputs received by the input unit. The processor is configured or programmed to control the display to show the new cutting point or the new cutting surface. User platform.

17. The processor is configured or programmed to control the display to show one or more agricultural characteristics of the crop. The one or more agricultural features of the crop are detected using an object detection model. The user platform according to claim 16.

18. The processor is configured or programmed to control the display to show a proposed cutting point or proposed cutting plane. The user platform according to claim 16.

19. The processor is configured or programmed to switch the cutting system, which includes the user platform, from a manual mode in which the user platform is used to create the new cutting point or new cutting surface before the cutting system performs the operation steps, to a fully automatic mode in which the cutting system automatically performs the operation steps, where Creating the new cutting point or new cutting surface of the crop includes creating a plurality of new cutting points or new cutting surfaces of the crop. Displaying the new cutting point or new cutting plane on the user platform includes displaying the plurality of new cutting points or new cutting planes. The processor is configured or programmed to switch the cutting system from the manual mode to the fully automatic mode based on the prediction rate of the plurality of new cutting points or new cutting surfaces. The prediction rate includes the rate at which the plurality of new cutting points or new cutting surfaces coincide with the plurality of proposed cutting points or proposed cutting surfaces. The user platform according to claim 16.

20. Using a processor to generate proposed cutting points for crops, The proposed cutting point is to be displayed on the user platform, The user platform receives at least one of the following inputs: an input to confirm that the proposed breakpoint is properly positioned, an input to delete the proposed breakpoint, and an input to modify the proposed breakpoint. A method that includes this.

Images