Cutting position determination program, cutting position determination device, and harvesting device

Abstract

A control unit (63) recognizes, through image recognition, a harvest object (7), a pedicel (6a) connected to the harvest object, and a main stem (6a) connected to the pedicel. The control unit (63) determines a cutting position of the pedicel when the pedicel is included in a cutting region (Ra) for cutting the pedicel in the recognized image, and an overlapping ratio of the main stem with respect to the cutting region is less than an overlapping threshold.

Description

Cutting point determination program, cutting point determination device, and harvesting device CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Japanese Application No. 2024-75892 filed on May 8, 2024, the contents of which are incorporated herein by reference.

[0002] The present disclosure relates to a cutting point determination program, a cutting point determination device, and a harvesting device.

[0003] In the agricultural field, various tasks have traditionally been performed by humans using agricultural work support devices such as cultivators, but in recent years, various technologies have been introduced into the agricultural field as companies and others have entered the field. Various technologies have been proposed to solve problems when harvesting crops (see, for example, Patent Document 1).

[0004] The technology described in Patent Document 1 allows the harvest target to be harvested in a short time by performing the search process for the target harvest target using an image processor and the harvesting process of the target harvest target using a harvesting robot in parallel.

[0005] Japanese Patent Application Laid-Open No. 2019-097447

[0006] Even when the technology described in Patent Document 1 is applied, there is a risk of deep damage to the harvest object or the main stem when harvesting the harvest object. An object of the present disclosure is to provide a cutting point determination program, a cutting point determination device, and a harvesting device that can accurately determine the cutting point of the harvest object without causing deep damage to the main stem.

[0007] According to one aspect of the present disclosure, an image of a harvest object, a stalk connected to the harvest object, and a main stem connected to the stalk is recognized, and the cutting location of the stalk is determined when the recognized image shows that the stalk is included in a cutting area for cutting the stalk and the overlap ratio of the main stem to the cutting area is less than an overlap threshold. This allows the cutting location of the harvest object to be accurately determined without causing deep damage to the harvest object or the main stem.

[0008] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. The drawings are as follows: Figure 1 is a schematic diagram showing the configuration of a harvesting device in one embodiment; Figure 2 is a schematic diagram showing the growth environment of a harvest object in one embodiment; Figure 3 is an enlarged view of a harvest object in one embodiment; Figure 4 is a schematic block diagram of the configuration of a harvesting device for one embodiment; Figure 5 is a functional block diagram of a cutting point determination device in one embodiment; Figure 6 is a first flowchart that generally explains the operation in one embodiment; Figure 7 is a second flowchart that generally explains the operation in one embodiment; Figure 8 is a first explanatory diagram of the interference check operation in one embodiment; Figure 9 is a second explanatory diagram of the interference check operation in one embodiment; Figure 10 is a third flowchart that generally explains the operation in one embodiment; Figure 11 is a diagram showing the interference situation between the cutting area and the main stem before the optimization process and explaining the orthogonality between the cutting area and the stalk, as shown in one embodiment; and Figure 12 is a diagram showing the interference situation between the cutting area and the main stem after the optimization process and explaining the orthogonality between the cutting area and the stalk, as shown in one embodiment.

[0009] An embodiment will be described below with reference to the drawings. Figure 2 shows a partial configuration of a harvesting system 10 installed in a farm field, and illustrates a cultivation area for harvest objects 7, such as bunches of crops. The harvesting system 10 is a system that employs, for example, a high-wire system to harvest the harvest objects 7 as bunches. While the harvest objects 7 will be described as cherry tomatoes, the system can also be applied to other crops and agricultural products, such as large tomatoes.

[0010] For ease of explanation, the following description will be given using three-dimensional X, Y, and Z directions as shown in Figure 2. In the X, Y, and Z directions shown in Figure 2, the Y direction indicates a direction extending vertically from a horizontal plane, and the XZ direction indicates a horizontal plane. As shown in Figure 2, in a farm field, soil 2 containing culture soil is arranged in a predetermined linear area, for example, along the X direction, and seedlings of harvest targets 7 are grown in the soil 2 at intervals in the soil 2 arranged in the predetermined area. In recent years, nutrient solution has been used instead of the soil 2. In the high-wire method, an extension wire 4 is fixed in a straight line above a row of seedlings in a greenhouse, and seedlings are grown by pulling main stems 6 in the Y direction using attracting wires 5 suspended from the extension wire 4.

[0011] At this time, the locks of the attracting wires 5 are fixed to the main stems 6 arranged in a row at a distance from each other, the main stems 6 are pulled up in a straight line above the soil 2, and with the main stems 6 hanging from the attracting wires 5, the locks of the attracting wires 5 are moved up in accordance with the growth of the main stems 6.

[0012] The harvest object 7 becomes ready for harvest when it ripens into the main stem 6. Furthermore, as the harvest object 7 is further harvested, when the tip of the main stem 6 reaches the extension wire 4, the height of the harvest object 7 can be adjusted by sliding the attracting wire 5 in accordance with the growth of the main stem 6. In this harvesting system 10, the height of the grown harvest object 7 is adjusted to be located within a predetermined height range H.

[0013] This allows the main stem 6 to be pulled up, adjusting the height of the harvestable object 7. Figure 3 shows in detail how the object 7 becomes the main stem 6. At this time, the main stem 6 extends obliquely upward from the ground toward the ceiling. The line of sight shown in Figure 3 indicates the line of sight of the hand camera 54 (corresponding to the imaging unit, see below).

[0014] The harvested product 7 bears fruit hanging from the main stem 6 via a stalk 6a (also called a lateral branch). The cherry tomatoes that become the harvested product 7 are clusters of several to a dozen individual fruits 7a, and the fruits 7a grow in order from the stalk 6a at the base of the main stem 6.

[0015] First, the basic system configuration will be explained, along with the basic flow of harvesting. Figure 1 shows an example of the external configuration of this harvesting system 10. As shown in Figure 1, the harvesting system 10 is provided for harvesting harvest objects 7 within a predetermined height range H. The harvesting system 10 is equipped with a harvesting device 50, which travels while ensuring safety in the surrounding area and harvests the harvest objects 7.

[0016] A robot arm 51 is attached to the harvesting device 50. The robot arm 51 is configured by connecting multiple arms 511 to 514. In the following description, when the term "robot arm 51" is used, it refers to at least a part of the drive unit of the harvesting device 50, and may refer to any one or more arms (e.g., 514) of the multiple arms 511 to 514 that make up the robot arm 51.

[0017] An end effector 52 is attached to an arm 514 at the tip of the robot arm 51. A harvesting unit 53 is configured at the tip of the end effector 52. Although not shown in detail, the harvesting unit 53 includes a cutting member 53a, a clamping member, and a guard member. Hereinafter, the movable area of ​​the cutting member 53a assumed by the control unit 63 in the analysis will be referred to as the cutting area Ra. The cutting member 53a is configured to cut the object to be harvested 7 at a cutting point P (see, for example, FIG. 12 ) and to clamp the object to be harvested 7 in a clamping area located on the object to be harvested 7 side of the cutting point P. The guard member is provided to prevent damage to obstacles such as the main stem 6 when the cutting member 53a of the harvesting unit 53 is inserted up to the cutting point P. The clamping member and guard member may be provided as needed.

[0018] The hand camera 54 is attached to the end effector 52 via a stay, and the imaging area in which images are captured by the hand camera 54 moves as the end effector 52 moves. The hand camera 54 is configured with a normal camera and is a device that can capture images of target objects in three-dimensional space and perform image recognition. The hand camera 54 may be a stereo camera, a ToF (Time of Flight) camera, a structured light scanner, or the like.

[0019] A traveling unit 14 is provided at the bottom of the harvesting device 50. The traveling unit 14 has a pair of drive wheels 16 and a plurality of driven wheels 17 in the Z direction shown in the figure, and travels automatically or manually. The drive wheels 16 are rotated by a drive source such as a traveling motor 60 (see FIG. 4 ) via a reducer, for example. The driven wheels 17 rotate as the traveling unit 14 travels due to the drive wheels 16, and support and stabilize the overall travel of the harvesting device 50.

[0020] When the traveling unit 14 travels autonomously, it uses a traveling camera 57, an obstacle detection unit 552, GPS, QR Code (registered trademark), RFID, etc., and uses SLAM technology to recognize its own position. See FIG. 4 for the traveling camera 57 and the obstacle detection unit 552. SLAM stands for Simultaneous Localization and Mapping.

[0021] The configuration of the traveling unit 14 is generally the same as that of an automated guided vehicle, but the traveling unit 14 of this embodiment is configured to travel along a hot water pipe (not shown). An automated guided vehicle is also called an AGV (Automated Guided Vehicle). This application is not limited to traveling on hot water pipes, but can also be applied to traveling on ordinary rails, paved roads, etc. A disinfection unit 18 is provided on the traveling unit 14.

[0022] Although not shown, the target field has multiple cultivation lanes with soil 2 shown in Figure 2 lined up at intervals in the Z direction, and hot water pipes are arranged in parallel in each cultivation lane, and the traveling unit 14 travels along the hot water pipes. A work passage is provided at one end of the cultivation lane. The traveling unit 14 automatically travels on the passage when heading from the storage area to the cultivation lane or when moving between cultivation lanes.

[0023] Next, the hardware configuration of the harvesting device 50 will be described with reference to Fig. 4. As shown in Fig. 4, the harvesting device 50 is equipped with a hand camera LED 72 to facilitate image capture by the hand camera 54. The harvesting device 50 also includes a left look-ahead camera 56, a traveling camera 57, a right look-ahead camera 58, a traveling camera LED 71, and a look-ahead camera LED 73 as a traveling monitoring unit.

[0024] In addition to the robot arm 51 described above, the harvesting device 50 also includes a robot controller 64 and a hand actuator 59. The harvesting device 50 also includes a travel motor 60 for driving the travel unit 14. The harvesting device 50 also includes a safety monitoring unit 55 for safety monitoring. Although not shown in FIG. 1 , the harvesting device 50 also includes a container for transporting the harvested objects 7 after harvesting.

[0025] The harvesting device 50 also includes a control unit 63 for overall control of the entire system, and the control unit 63 includes a robot controller 64. The control unit 63 corresponds to a cutting point determination device. The robot controller 64 may be configured as the "control unit" of the present disclosure. A teaching pendant 65 is connected to the control unit 63. The harvesting device 50 also includes a wireless LAN router 66 connected to the control unit 63, and is configured to be able to link with an external cloud server 67 via the wireless LAN router 66.

[0026] The configuration will be described in detail below. The teaching pendant 65 is connected to the robot controller 64. A setting support application program that supports the user in setting the movements of the robot arm 51 and the hand actuator 59 is installed on the teaching pendant 65, and the teaching pendant 65 is provided to input various operations and display the status of the harvesting device 50 on a screen. The teaching pendant 65 accepts various operations by the user and displays various information.

[0027] 1, the left-side looking-ahead camera 56 and the right-side looking-ahead camera 58 are attached to the robot arm 51. The control unit 63 roughly recognizes the harvest target 7 through the left-side looking-ahead camera 56 and the right-side looking-ahead camera 58. When the control unit 63 roughly recognizes the harvest target 7, it instructs the robot controller 64 to operate, and the robot controller 64 moves the robot arm 51 close to the cherry tomatoes that will be the harvest target 7.

[0028] The robot controller 64 operates and controls the hand actuator 59 via the robot arm 51. This allows the harvesting unit 53 to move in three dimensions, up and down, left and right, and front and back, and the harvesting direction of the harvesting unit 53 can be controlled to any position in the three dimensions. The robot arm 51 also allows the harvesting unit 53 attached to its tip to rotate around an axis in any desired three-dimensional direction. The hand actuator 59 is controlled by the robot controller 64, and operates and controls the harvesting unit 53 to harvest and hold the harvest target 7.

[0029] The harvesting unit 53 cuts the fruit stalk 6a and clamps a portion of the fruit stalk 6a to hold the bunch of harvest objects 7 so as not to drop the fruit 7a. The robot controller 64 drives the travel motor 60 to rotate the drive wheels 16, thereby moving the traveling unit 14. The robot controller 64 can operate and control the robot arm 51 to place the harvested harvest objects 7 in a collection container or the like (not shown).

[0030] <Regarding the safety monitoring unit 55> A safety monitoring unit 55 is provided to ensure that the harvesting device 50 travels safely within the field and safely harvests the harvest target objects 7. The safety monitoring unit 55 is preferably configured independent of the robot controller 64, the harvesting unit 53, and the travel motor 60 of the travel unit 14. The safety monitoring unit 55 includes an obstacle detection unit 552, a travel path detection unit 553, a robot posture detection unit 554, and a travel monitoring unit 555. The obstacle detection unit 552 detects an obstacle and stops the harvesting device 50. The travel path detection unit 553 detects the hot water pipe. The robot posture detection unit 554 detects the posture of the harvesting device 50. The travel monitoring unit 555 monitors the travel status of the travel unit 14. The sensors and switches that make up the safety monitoring unit 55 are preferably configured using sensors and switches that have obtained predetermined safety certification.

[0031] 1, the obstacle detection units 552 are provided at the front and rear of the traveling unit 14. The obstacle detection units 552 are disposed in a dispersed manner at the front and rear of the traveling direction of the harvesting device 50. The obstacle detection units 552 are provided to detect obstacles present around the harvesting device 50.

[0032] The obstacle detection unit 552 is configured with, for example, a laser sensor that detects obstacles present in the surroundings, a so-called LiDAR, an ultrasonic sensor, etc. The obstacles detected by the obstacle detection unit 552 may be, for example, cultivation equipment, support members, people, various types of equipment, etc. present in the farm field.

[0033] 1, the travel path detection units 553 are provided at the front and rear of the traveling unit 14. The travel path detection units 553 are also disposed separately at the front and rear of the harvesting device 50 in the traveling direction. The travel path detection units 553 are provided to ensure safe traveling along the hot water pipe. The travel monitoring unit 555 is equipped with a travel monitoring encoder and detects the number of rotations of the drive wheels 16 of the traveling motor 60 or the number of rotations of the axle, and monitors the traveling speed, traveling distance, and traveling conditions of the traveling unit 14.

[0034] The robot posture detection unit 554 monitors and controls the robot arm 51 to prevent unintended movements, and safely controls the operation of the robot arm 51 using images captured by the left look-ahead camera 56 and the right look-ahead camera 58. The robot posture detection unit 554 may also safely control the operation using images captured by the hand camera 54.

[0035] The control unit 63 includes a processor such as an image processor, memory such as volatile memory or nonvolatile memory, I / O, etc. (none of which are shown), as well as a robot controller 64. The robot controller 64 can be integrated with the control unit 63 and used as the "control unit" of the present disclosure. The memory is used as a non-transient tangible recording medium. The hand camera 54, left-lookahead camera 56, and right-lookahead camera 58 are connected to the control unit 63. Images captured by the hand camera 54, left-lookahead camera 56, and right-lookahead camera 58 are stored in the memory of the control unit 63.

[0036] The control unit 63 has the functions of an image recognition unit 63a and a cutting location determination unit 63b, as shown in Figure 5. The image recognition unit 63a performs image recognition of the harvest object 7, the stalk 6a connected to the harvest object 7, and the main stem 6 connected to the stalk 6a. The cutting location determination unit 63b determines a cutting location P for the stalk 6a when the recognized image includes the stalk 6a in a cutting region Ra for cutting the stalk 6a and the overlapping ratio of the main stem 6 with the cutting region Ra is less than an overlap threshold.

[0037] The control unit 63 performs image recognition of the harvest target 7 using images captured by the hand camera 54 stored in memory by the image processor executing a program stored in memory. The control unit 63 performs image recognition of the condition of the left side of the bunch of harvest target 7 using images captured by the left look-ahead camera 56 stored in memory. The control unit 63 performs image recognition of the condition of the right side of the bunch of harvest target 7 using images captured by the right look-ahead camera 58.

[0038] The image processor of the control unit 63 acquires images including the height range H described above using the hand camera 54, and constructs a three-dimensional image according to the images captured by the hand camera 54. The control unit 63 recognizes the three-dimensional position, shape, and color of the harvest target 7, and acquires three-dimensional position information, shape information, and color information of the harvest target 7, as well as three-dimensional position information, shape information, and color information of the main stem 6 and stalk 6a.

[0039] The control unit 63 uses this information to perform clustering processing, which recognizes the harvest objects 7, the stalk 6a, and the main stem 6 through image recognition and separates them from the background. This processing is performed using a support vector machine (SVM). This makes it possible to distinguish the three-dimensional areas in which the harvest objects 7, the stalk 6a, and the main stem 6 exist. Specifically, the control unit 63 obtains voxelized image data by performing three-dimensional image recognition on the image captured by the hand camera 54.

[0040] The control unit 63 uses clustering technology to extract features of the harvest object 7, the stalk 6a, and the main stem 6 from this collection of image data, and divides the three-dimensional area into parts to distinguish them. The stalk 6a physically connects the main stem 6 and the harvest object 7, so the three-dimensional area in which the stalk 6a is located can be detected and divided into parts to connect the main stem 6 and the harvest object 7.

[0041] The hand camera 54 is provided on the robot arm 51. Therefore, the control unit 63 can control the line of sight and the imaging area when the hand camera 54 captures an image by operating the robot arm 51 via the robot controller 64.

[0042] The harvesting unit 53 is also provided on the robot arm 51. The control unit 63 can operate the harvesting unit 53 by operating the robot arm 51 and the hand actuator 59 via the robot controller 64 in accordance with information on the recognized harvest object 7, main stem 6, and stalk 6a.

[0043] The robot controller 64 can change the XYZ direction position and three-dimensional angle of the harvesting unit 53 by operating and controlling the hand actuator 59 via the robot arm 51. By operating the hand actuator 59, the harvesting unit 53 can clamp the stalk 6a on which the harvest target 7 has borne fruit and cut the cutting point P, thereby harvesting the harvest target 7 along with the bunch.

[0044] <Control Overview> Next, a detailed example of a control method for the robot arm 51 when harvesting the harvest target 7 through image recognition will be described. The control unit 63 rotates the travel motor 60 via the robot controller 64 to cause the travel unit 14 to travel and reach the vicinity of the harvest target 7. Then, as shown in Figure 6, the control unit 63 acquires information about the cherry tomato clusters that will become the harvest target 7 from the hand camera 54 in S11, and also acquires an image of the surrounding area. This image is a two-dimensional image captured by the hand camera 54. Next, the control unit 63 executes a calculation process to prevent cutting of the main stem 6 in S12, determines a cutting location P in S13, controls the harvesting unit 53 in S14, and then harvests in S15.

[0045] 7 shows an outline of the calculation process for preventing cutting of the main stem 6 in S12. The control unit 63 performs image recognition of the harvest object 7, the stalk 6a connected to the harvest object 7, and the main stem 6 connected to the stalk 6a (function of the image recognition unit 63a). The control unit 63 sets the cutting area Ra of the harvesting unit 53 within the image acquired in S22. Because the cutting area Ra of the harvesting unit 53 and the imaging area of ​​the hand camera 54 change in conjunction with the operation of the robot arm 51, the cutting area Ra of the cutting member 53a of the harvesting unit 53 can be uniquely set within the image of the imaging area of ​​the hand camera 54.

[0046] Furthermore, because the cutting member 53a of the harvesting unit 53 is a so-called scissors, its cutting range is long in the horizontal direction and short in the vertical direction, which is perpendicular to the long direction. Therefore, the cutting area Ra assumed for this cutting member 53a is set to a horizontally long rectangle as shown in Figure 8. The set cutting area Ra is an area used when performing the interference determination process in the first interference check described below, and is a predetermined first rectangular area (e.g., 57 x 15 pixels) that is set relatively wide.

[0047] In S23, the control unit 63 determines whether the cutting area Ra of the harvesting unit 53 includes the stalk 6a. At this time, the control unit 63 maps the main stem 6 and the stalk 6a, which are distinguished in three-dimensional regions by the clustering process described above, onto an image captured from the viewpoint of the hand camera 54, resulting in a two-dimensional area representing the two-dimensional existence area of ​​the main stem 6 and the stalk 6a. The control unit 63 then determines whether the two-dimensional existence area of ​​the stalk 6a is included in the cutting area Ra of the harvesting unit 53. If the stalk 6a is not included in the cutting area Ra, the control unit 63 changes the search location and acquires the image again in S24. The control unit 63 then returns to S21 to recognize the image and repeats the setting of the cutting area Ra in S22. That is, the control unit 63 operates the hand camera 54 via the robot arm 51 to search for multiple candidate points. In this case, it is advisable to look into the stalk 6a at a predetermined distance and angle to determine the structure of the stalk 6a.

[0048] If the control unit 63 determines in S23 that the stalk 6a is included in the cutting area Ra, it determines in S25 whether the ratio of the area of ​​the main stem 6 to the cutting area Ra is less than the area threshold. This represents the first interference check process, and the area threshold indicates a predetermined ratio threshold. If the harvest object 7 is a cherry tomato, the area threshold for the area ratio should be set to a value in the range of 5 to 25%, and more preferably to a value in the range of 10 to 20%. It is more preferable to set the area threshold to 20%.

[0049] Comparing cherry tomatoes 7 and other crops, the damage caused by the cutting member 53a when the harvest object 7 or main stem 6 comes into contact with them varies depending on the strength of the harvest object 7 and the stalk 6a. When delicate crops are harvested, the area threshold is preferably set to a value between 0 and 5%, a stricter threshold than the above. Therefore, it is desirable to determine the area threshold by parameterizing factors such as whether the crop variety is acceptable for contact with the harvest object 7 or main stem 6 by the cutting member 53a and / or the strength of the main stem 6. The area threshold parameters may also be determined based on the material, size, dimensions, and shape of the cutting member 53a. It is desirable to set the area threshold so as not to inflict deep damage on the harvest object 7 or main stem 6. For example, Figure 8 shows an example of a setting in which the ratio of the main stem 6 area to the cutting area Ra is less than the specified area threshold.

[0050] If the control unit 63 determines in S25 that the ratio of the area of ​​the main stem 6 to the cutting area Ra is equal to or greater than the area threshold, it rejects the interference check in S26. In this case, the control unit 63 assumes that the cutting angle of the cutting member 53a will be changed, changes the cutting area Ra in S27, and returns to S25. The control unit 63 then searches for a cutting area Ra that satisfies the conditions of S25. For example, in the example of FIG. 8, an area that is elongated horizontally is set as the cutting area Ra, but the angle of this cutting area Ra may be rotated to elongate it diagonally or vertically. In this way, the control unit 63 may change the cutting area Ra of the cutting member 53a and search for a cutting area Ra that satisfies the conditions of S25.

[0051] If the conditions of S25 are met, the control unit 63 reduces the cutting area Ra to its central region in S28 and determines in S29 whether the overlap ratio of the main stem 6 with the central region of the cutting area Ra is less than the overlap threshold. This process is the second interference check process. In this second interference check process, the control unit 63 measures the overlap ratio for a predetermined second rectangular region (e.g., 15 x 15 pixels) obtained by reducing the initial cutting area Ra. Figure 9 shows the second rectangular region of the cutting area Ra at this time. If the overlap ratio of the main stem 6 with the second rectangular region of the cutting area Ra is large, interference is more likely to occur. Conversely, if the overlap ratio is small, it becomes more difficult to find the stalk 6a at the base of the main stem 6. Therefore, it is desirable to set the overlap threshold taking these trade-offs into consideration. As with the first interference check described above, the overlap threshold for the main stem 6 can be adjusted according to the parameters of the main stem 6, the harvest object 7, and the cutting member 53a of the harvesting unit 53.

[0052] In S29, the control unit 63 sets the overlap threshold for the overlap rate to zero percent, and rejects interference with the central region of the cutting area Ra of the harvesting unit 53, thereby minimizing cutting of the main stem 6. In this case, by rejecting the interference check in S26, the control unit 63 can prevent the cutting member 53a from being inserted into the cutting position P where there is a risk of cutting the main stem 6.

[0053] In S29, the control unit 63 determines whether the overlap ratio of the main stem 6 with the central region of the cutting region Ra is less than the overlap threshold. If the result is equal to or greater than the overlap threshold, the control unit 63 determines in S26 that the interference check is NG. The control unit 63 then changes the cutting region Ra and returns to S25 to search again for a cutting region Ra that satisfies the conditions in S25 and S29.

[0054] 7, if the control unit 63 is unable to find a cutting area Ra that satisfies both the conditions of S25 and S29 even after changing the cutting area Ra a predetermined number of times in S27, the control unit 63 may change the search location. In this case, the control unit 63 may operate the robot arm 51 to change the search location, acquire an image, and then repeat the process from S21. If the control unit 63 is able to find a cutting area Ra that satisfies the conditions of S25 and S29, it ends the calculation process to prevent cutting of the main stem 6.

[0055] <Process for determining cutting point P> The control unit 63 executes the process for determining cutting point P shown in Fig. 10. As shown in S31, the control unit 63 calculates a main stem interference score A by scoring the degree of interference between the cutting area Ra and the main stem 6. As shown in S32, the control unit 63 calculates a fruit stalk orthogonality score B by scoring the orthogonality between the cutting area Ra and the fruit stalk 6a. As shown in S33 to S35, the control unit 63 determines the cutting point P based on the calculation results of the main stem interference score A and the fruit stalk orthogonality score B.

[0056] Details will be explained. In S31, the control unit 63 scores the overlapping area between the main stem 6 and the cutting area Ra of the cutting member 53a, obtaining the main stem interference score A. In this case, it is preferable to apply a relatively horizontally long first rectangular area to the cutting area Ra. The range of the main stem interference score A here is a first predetermined range from the minimum value to the maximum value (e.g., 0 to 20), and the main stem interference score A should be reduced if the overlapping area is large, and increased if the overlapping area is small. Therefore, the control unit 63 sets the main stem interference score A to the maximum value if there is no overlap, and to the minimum value if there is overlap. It is desirable to set the range of values ​​according to the purpose and use.

[0057] In S32, the control unit 63 calculates the fruit stalk orthogonality score B by scoring the angle by which the extension direction of the fruit stalk 6a deviates from the normal direction to the longitudinal direction of the cutting region Ra. The range of the fruit stalk orthogonality score B here is a second predetermined range from the minimum value to the maximum value (e.g., zero to 20), with the maximum value being taken when the extension direction coincides with the normal direction to the longitudinal direction of the cutting region Ra, and the minimum value being taken when the extension direction is perpendicular to the normal direction. As described above, it is desirable to set the range according to the purpose and application. For example, the fruit stalk orthogonality score B can be calculated by: Fruit stalk orthogonality score B = Maximum value - Maximum value x (abs (normal angle - angle of target candidate) / 90), where abs indicates an absolute value.

[0058] In S33, the control unit 63 weights the scores calculated in S31 and S32. The control unit 63 then determines the cutting point P based on the total score calculated by weighting each of the main stem interference score A and the stalk orthogonality score B (the function of the cutting point determination unit 63b). The weighting is performed so as to optimize the parameters based on the position and posture of the cutting member 53a of the harvesting unit 53.

[0059] The parameters are the three-dimensional coordinate position and orientation when the cutting member 53a is inserted into the fruit stalk 6a, and the three-dimensional coordinate position and orientation when cutting along the fruit stalk 6a toward the main stem 6. The control unit 63 weights the main stem interference score A and the fruit stalk orthogonality score B so that the parameters are optimized in steps S31 and S32. If the weighted values ​​of the main stem interference score A and the fruit stalk orthogonality score B when controlling the insertion of the cutting member 53a and cutting toward the main stem 6 are wa and wb, respectively, the total score can be calculated by (wa x A + wb x B).

[0060] When inserting the cutting member 53a, it is recommended that the weighting value wa of the main stem interference score A be greater than the weighting value wb of the stalk orthogonality score B. Furthermore, when the cutting member 53a is moved close to the main stem 6 to cut, it is recommended that the weighting value wa be the same as the weighting value wb or within a predetermined range. Furthermore, if the main stem 6 or the harvest object 7 is flexible and strong, and even if the cutting member 53a of the harvesting unit 53 comes into contact with the main stem 6, deep scratches will not be caused and the growth of the main stem 6 or the harvest object 7 will be minimally affected, the importance of the main stem interference score A will be low.

[0061] Conversely, if the main stem 6 is not strong enough and the cutting member 53a of the harvesting unit 53 makes contact with the main stem 6 or the harvest object 7, causing deep scratches and adversely affecting the main stem 6 or the harvest object 7, the importance of the main stem interference score A becomes higher. It is desirable to determine the weighting values ​​wa and wb of the main stem interference score A and the pedicel orthogonality score B according to the purpose and use.

[0062] The control unit 63 calculates the total score in S34, and determines the cutting point P that satisfies the conditions for the best total score in S35. The control unit 63 then returns the process to S14 in Fig. 6 to control the harvesting unit 53, and harvests the harvest target 7 in S15. This allows the harvest target 7 to be harvested while being cut at an appropriate cutting point P, thereby improving the harvest success rate.

[0063] 11 and 12 show examples of the results of the optimization of the cutting area Ra before and after the optimization process. As shown in FIG. 11, before the optimization process, the cutting member 53a can be inserted at an angle perpendicular to the extension direction of the stalk 6a, but the cutting area Ra overlaps the main stem 6 to a large extent, which may cause deep damage to the main stem 6. In contrast, as shown in FIG. 12, after the optimization process, the cutting area Ra does not overlap the main stem 6, and the cutting member 53a can be inserted at an angle roughly perpendicular to the stalk 6a. This allows the stalk 6a to be reliably cut without causing deep damage to the harvest object 7 or the main stem 6, and the cutting point P can be cut at an appropriate angle.

[0064] <Background and Issues of the Present Embodiment, and Summary of the Present Embodiment> For example, if an algorithm is applied that checks for interference if the overlapping area is less than a predetermined percentage, even if the main stem 6 is near the center of the cutting member 53a of the harvesting unit 53, there is a risk of cutting the main stem 6. Also, a point where the cutting member 53a is close to a direction perpendicular to the stalk 6a, even if there is some interference with the main stem 6, and a point where there is no interference but at an angle significantly different from the perpendicular direction, may be determined as candidates for the cutting point P. In this case, if an algorithm is adopted that sets the former as the cutting point P, there is a risk of a higher risk of cutting failure compared to when the latter "cutting point P without interference" is selected as a candidate.

[0065] In contrast, according to this embodiment, the harvest object 7, stalk 6a, and main stem 6 are image-recognized, and the cutting point P of the stalk 6a is determined when the recognized image includes the stalk 6a in a cutting area Ra for cutting the stalk 6a and the overlapping ratio of the main stem 6 with the cutting area Ra is less than an overlap threshold. This allows the cutting point P of the harvest object 7 to be accurately determined without causing deep damage to the harvest object 7 or main stem 6.

[0066] In this embodiment, the control unit 63 performs area determination by applying a first rectangular area that is slightly larger to determine the area between the cutting area Ra of the harvesting unit 53 and the main stem 6 (S25). Then, the control unit 63 performs overlap determination by applying a central second rectangular area that is slightly smaller than the cutting area Ra of the harvesting unit 53 to the overlap area with the main stem 6 (S29). As a result, the cutting point P of the harvest object 7 can be determined without causing deep damage to the main stem 6 or the harvest object 7. This increases the harvesting success rate.

[0067] (Other Embodiments) The present invention is not limited to the above-described embodiments, and the following modifications or extensions are possible, for example. Cherry tomatoes have been exemplified as the harvest object 7 that bears bunches of fruit 7a, but the present invention is not limited to this, and other crops and agricultural products such as large tomatoes, medium-sized tomatoes, and eggplants can also be used as the harvest object 7. The same applies when other crops and agricultural products are used as the harvest object 7.

[0068] Although the embodiment in which the processor that executes the processing according to the present application is provided in the control unit 63 has been described, the present invention is not limited to this, and the processing according to the present application may be executed by a processor provided in the cloud server 67. Furthermore, if a processor is provided in the hand camera 54, the processor of the hand camera 54 may be caused to execute the processing according to the present application. Furthermore, if processors are provided in at least two or more elements of the control unit 63, the cloud server 67, and the hand camera 54, the processing according to the present application may be shared and executed by these processors. Furthermore, processors may be configured in other configurations and executed or shared processing may be executed by the processors.

[0069] The techniques described in this disclosure may be implemented by a special purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the techniques described in this disclosure may be implemented by a special purpose computer configured with a processor comprising one or more dedicated hardware logic circuits. Alternatively, the techniques described in this disclosure may be implemented by one or more special purpose computers configured with a processor comprising one or more hardware logic circuits in combination with a processor and memory programmed to perform one or more functions. Furthermore, a computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium.

[0070] Although the present disclosure has been described based on the above-described embodiment, it is understood that the present disclosure is not limited to the embodiment or the structure described in the embodiment. The present disclosure also encompasses various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more than one element, or less than one element, are also within the scope and spirit of the present disclosure.

Claims

1. A cutting point determination program that causes a processor to perform image recognition of a harvest object (7), a fruit stalk (6a) connected to the harvest object, and a main stem (6) connected to the fruit stalk, and determines a cutting point (P) for the fruit stalk when the recognized image shows that the fruit stalk is included in a cutting area (Ra) for cutting the fruit stalk and the overlapping ratio of the main stem to the cutting area is less than an overlap threshold.

2. A cutting point determination program as described in claim 1, which causes the processor to determine the cutting point when the cutting area includes the stalk, the overlap ratio of the main stem to the cutting area is less than the overlap threshold, and the ratio of the area of ​​the main stem to the cutting area is less than the area threshold.

3. A cutting point determination program as described in claim 1 or 2, which causes the processor to calculate a main stem interference score by scoring the degree of interference between the cutting area and the main stem, calculate a fruit stalk orthogonality score by scoring the orthogonality between the cutting area and the fruit stalk, and determine the cutting point based on the calculation results of the main stem interference score and the fruit stalk orthogonality score.

4. A cutting point determination program according to claim 3, which causes the processor to determine the cutting point based on a total score calculated by weighting each of the main stem interference score and the pedicel orthogonality score.

5. A cutting location determination device comprising: an image recognition unit (63a) that performs image recognition of a harvest object (7), a fruit stalk (6a) connected to the harvest object, and a main stem (6) connected to the fruit stalk; and a cutting location determination unit (63b) that determines the cutting location of the fruit stalk when the recognized image includes the fruit stalk in a cutting area for cutting the fruit stalk and the overlapping ratio of the main stem to the cutting area is less than an overlap threshold.

6. A harvesting device that harvests a harvest object (7) by cutting a stalk (6a) connecting the main stem (6) to the harvest object, comprising: a robot arm (51) connected to a harvesting unit for harvesting the harvest object; an imaging unit (54) provided on the robot arm; and a control unit (63, 64) that controls the attitude of the robot arm, wherein the control unit comprises: an image recognition unit (63a) that performs image recognition of the harvest object (7), the stalk (6a) connected to the harvest object, and the main stem (6) connected to the stalk; and a cutting point determination unit (63b) that determines the cutting point of the stalk when the stalk is included in the recognized image in a cutting area for cutting the stalk and the overlapping ratio of the main stem to the cutting area is less than an overlap threshold.

Images