Harvesting robot, control method and control program for harvesting robot, and harvesting system

Abstract

To provide a harvesting robot that improves accuracy of estimating a position of an obstacle object to be an obstacle to harvesting to enable prevention of collision between the harvesting robot and the obstacle object.SOLUTION: A harvesting robot for harvesting crops grown on deciduous tree comprises: a storage unit that stores defoliation period three-dimensional position data including a position of at least part of a tree from which its leaves have fallen; a picking unit for harvesting a crop; a manipulator for moving the picking unit; a sensor unit for acquiring data on the position of the crop; and a processing unit that at the time of crop harvesting, detects the harvesting target crop on the basis of the data on the position acquired by the sensor unit, estimates at least a position of an obstacle object to be an obstacle to movement of the picking unit on the basis of the defoliation period three-dimensional position data, and, when the harvesting target crop is detected, controls the manipulator so that the picking unit moves to a position at which harvesting the detected crop is possible while inhibiting at least part of the harvesting robot from colliding with the obstacle object on which the estimation is performed.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The embodiments disclosed in this specification relate to a harvesting robot, a control method and control program for the harvesting robot, and a harvesting system for harvesting crops that grow on deciduous trees. [Background technology]

[0002] In recent years, harvesting robots that harvest crops such as fruits that grow on trees have become known. For example, Patent Documents 1 and 2 describe a control technology in which a harvesting robot analyzes captured images to estimate the three-dimensional positions of the crops to be harvested and obstacles that hinder harvesting, and harvests the crops while avoiding contact between the end effector of the harvesting robot and the obstacles. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-195335 A [Patent Document 2] JP 2010-207118 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the techniques described in Patent Documents 1 and 2 are methods for estimating the position of obstacles based on images taken during harvesting, but obstacles such as tree branches hidden by leaves are not visible in the images, making it impossible to accurately estimate the positions of obstacles. This has led to a problem in that components of the harvesting robot, such as the end effector and manipulator, collide with obstacles whose positions could not be estimated during the harvesting operation of the harvesting robot.

[0005] The harvesting robot, the control method and control program for the harvesting robot, and the harvesting system disclosed in this specification aim to improve the accuracy of estimating the position of obstacles that hinder harvesting, and to prevent collisions between the harvesting robot and the obstacles. [Means for solving the problem]

[0006] The disclosed harvesting robot is a harvesting robot that harvests crops that grow on deciduous trees, and is equipped with a memory unit that stores three-dimensional position data during the deciduous season including the position of at least a portion of the tree that has shed its leaves, a picking unit that harvests the crop, a manipulator that moves the picking unit, a sensor unit that acquires data regarding the position of the crop, and a processing unit that detects the crop to be harvested based on the position data acquired by the sensor unit when the crop is harvested, estimates at least the position of an obstacle that will hinder the movement of the picking unit based on the three-dimensional position data during the deciduous season, and when a crop to be harvested is detected, controls the manipulator so that the picking unit moves to a harvestable position of the detected crop without collision between the estimated obstacle and at least a part of the harvesting robot.

[0007] In addition, in the disclosed harvesting robot, when a crop to be harvested is detected, it is preferable that the processing unit acquires three-dimensional position data at the time of harvesting including the position of at least part of an obstacle based on position data acquired by the sensor unit, and estimates at least the position of the obstacle that is hindering the movement of the picking unit based on the three-dimensional position data during the deciduous period and the three-dimensional position data at the time of harvesting.

[0008] In addition, in the disclosed harvesting robot, it is preferable that the sensor unit has an imaging unit that photographs a subject and outputs the photographed image data of the subject, and a measurement unit that measures the distance to surrounding objects and outputs distance data indicating the measured distance, and that the processing unit calculates three-dimensional position data corresponding to one or more points in the photographed image data based on the distance data corresponding to the output one or more points, and acquires the calculated three-dimensional position data as three-dimensional position data at the time of harvest.

[0009] In addition, in the disclosed harvesting robot, it is preferable that the three-dimensional position data during the defoliation period is three-dimensional position data corresponding to one or more points, calculated based on image data captured by a capture unit during the defoliation period immediately prior to the harvesting of the crop, and distance data corresponding to one or more points within the captured image data output by a measurement unit at the time of capturing the image.

[0010] In addition, in the disclosed harvesting robot, it is preferable that the processing unit estimates, from the captured image data, an image area indicating the tree branches and an image area indicating the member supporting the tree branches as obstacle image areas, and acquires, from the calculated three-dimensional position data, three-dimensional position data within the obstacle image areas as three-dimensional position data at the time of harvesting.

[0011] In addition, the disclosed harvesting robot is preferably equipped with a cart that moves within a field in which trees are planted, and the processing unit preferably controls the movement of the cart when a crop to be harvested is detected so that the picking unit can move to a harvestable position for the detected crop without colliding with an estimated obstacle and at least a part of the harvesting robot.

[0012] In addition, in the disclosed harvesting robot, it is preferable that the processing unit estimates the position and shape of the obstacle based on the three-dimensional position data during the leaf-shedding period.

[0013] The disclosed control method is a control method for a harvesting robot that harvests crops that grow on deciduous trees, and is equipped with a memory unit that stores three-dimensional position data during the deciduous season including the position of at least a portion of the trees that have shed their leaves, a picking unit that harvests the crops, a manipulator that moves the picking unit, a sensor unit that acquires data regarding the position of the crops, and a processing unit, wherein the processing unit detects the crop to be harvested based on the position data acquired by the sensor unit when harvesting the crop, estimates at least the position of an obstacle that will hinder the movement of the picking unit based on the three-dimensional position data during the deciduous season, and, when the crop to be harvested is detected, controls the manipulator so that the picking unit moves to a harvestable position of the detected crop without colliding with the estimated obstacle and at least a part of the harvesting robot.

[0014] The disclosed control program is a control program for a harvesting robot that harvests crops that grow on deciduous trees, and is equipped with a memory unit that stores three-dimensional position data during the deciduous season including the position of at least a portion of the trees that have shed their leaves, a picking unit that harvests the crops, a manipulator that moves the picking unit, a sensor unit that acquires data regarding the position of the crops, and a processing unit, and causes the processing unit to detect a crop to be harvested based on the position data acquired by the sensor unit when harvesting the crop, estimate at least the position of an obstacle that will hinder the movement of the picking unit based on the three-dimensional position data during the deciduous season, and, when a crop to be harvested is detected, control the manipulator so that the picking unit moves to a harvestable position of the detected crop without collision between the estimated obstacle and at least a part of the harvesting robot.

[0015] The disclosed harvesting system has a harvesting robot that harvests crops that grow on deciduous trees and a server device that communicates with the harvesting robot, wherein the server device comprises a memory unit that stores three-dimensional position data during the deciduous season including the position of at least a portion of the trees that have shed their leaves, and a transmitting unit that transmits the three-dimensional position data during the deciduous season to the harvesting robot, and the harvesting robot comprises a receiving unit that receives the three-dimensional position data during the deciduous season transmitted from the server device, a picking unit that harvests the crop, a manipulator that moves the picking unit, a sensor unit that acquires data regarding the position of the crop, and a processing unit that detects a crop to be harvested based on the position data acquired by the sensor unit when harvesting the crop, estimates at least the position of an obstacle that will hinder the movement of the picking unit based on the received three-dimensional position data during the deciduous season, and when a crop to be harvested is detected, controls the manipulator so that the picking unit moves to a harvestable position of the detected crop without colliding with the estimated obstacle and at least a portion of the harvesting robot. Effect of the Invention

[0016] The disclosed harvesting robot, harvesting robot control method and control program, and harvesting system make it possible to improve the accuracy of estimating the position of obstacles that hinder harvesting, and to prevent collisions between the harvesting robot and the obstacles. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a harvesting system. [Diagram 2] FIG. 1A is a perspective view showing an example of the appearance of a harvesting robot, and FIG. 1B is a perspective view showing an example of the appearance of an end effector system. [Diagram 3] FIG. 1 is a diagram illustrating an example of a schematic configuration of a harvesting robot. [Figure 4] FIG. 2 illustrates an example of a schematic configuration of a server device. [Diagram 5] FIG. 11 is a diagram illustrating an example of an operational flow of a process for extracting obstacle data during the deciduous season. [Figure 6]FIG. 1(a) is a diagram showing an example of image data captured in a farm field where deciduous trees are planted, and FIG. 1(b) is a schematic diagram showing an example of depth data measured when the image data was captured. [Figure 7] FIG. 13A is a diagram showing an example of estimated image data indicating an obstacle image region, and FIG. 13B is a schematic diagram showing an example of obstacle data during the leaf-shedding season. [Figure 8] FIG. 11 is a diagram showing an example of an operational flow of a harvesting process. [Figure 9] FIG. 13 is a diagram illustrating an example of an operation flow of a harvesting obstacle data extraction process. [Figure 10] (a) is a diagram showing an example of captured image data of a crop to be harvested at harvest time, (b) is a schematic diagram showing an example of depth data measured when the captured image data was captured, (c) is a diagram showing an example of estimated image data showing an obstacle image area, and (d) is a schematic diagram showing an example of obstacle data during harvesting. [Figure 11] FIG. 11 is a diagram showing an example of an operation flow of a harvest control process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Various embodiments of the present invention will be described below with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions described in the claims and their equivalents.

[0019] (Overview of Harvesting System 1) 1 is a diagram showing an example of a schematic configuration of a harvesting system 1. The harvesting system 1 of this embodiment includes a harvesting robot 2 and a server device 3.

[0020] The harvesting robot 2 has various functions for harvesting crops that grow on deciduous trees. The deciduous trees are grown in a specified field. The crops that grow on the deciduous trees include cherries, peaches, plums, akebia japonica, blueberries, raspberries, pears, apples, persimmons, grapes, plums, and apricots. Hereinafter, "deciduous trees" may be simply referred to as "trees." The specified field is a field suitable for growing trees, such as an open field, a vinyl greenhouse field, or a pipe greenhouse field.

[0021] The server device 3 has a function for managing three-dimensional position data of trees with shed leaves. The three-dimensional position data is data indicating three-dimensional coordinates corresponding to points on the surface of an object such as a tree according to a predetermined position coordinate system. Note that the objects indicated by the three-dimensional position data may include not only "trees" but also objects that obstruct the movement of the harvesting robot 2, the end effector system 22, and the manipulator system 23 (for example, "various members supporting the branches (supports (pipes, stakes, etc.), wires, tapes, and / or strings, etc.)").

[0022] The predetermined position coordinate system is a geographic coordinate system such as the Japanese Geodetic System 2011, which is indicated by latitude, longitude, and altitude. The predetermined position coordinate system may be a plane rectangular coordinate system or a projected coordinate system of the UTM (Universal Transverse Mercator) coordinate system, which corresponds to a predetermined area including the field in which the trees are planted. The predetermined position coordinate system may also be a coordinate system that has an origin at a predetermined point in the field in which the trees are planted, an x-axis parallel to a predetermined direction on a predetermined horizontal plane, a y-axis on the horizontal plane that is perpendicular to the x-axis, and a z-axis in the vertical direction. The predetermined position coordinate system may also be a world coordinate system that defines the entire three-dimensional virtual space corresponding to the field in which the trees are planted.

[0023] The three-dimensional coordinate points indicated by the three-dimensional position data may be points on a part of the surface of a tree. Also, the three-dimensional coordinate points indicated by the three-dimensional position data may be points on the surface of multiple trees, or may be points on a part of the surface of each of the multiple trees. By including multiple three-dimensional coordinate points in the three-dimensional position data, the shape (volume) of the corresponding tree is indicated, and the shape of an obstacle such as a tree can be estimated by generating a surface model described later. Hereinafter, the "three-dimensional position data of a tree that has lost its leaves" may be referred to as "three-dimensional position data in the deciduous season". When a tree loses its leaves in multiple periods, the "deciduous season" of the "three-dimensional position data in the deciduous season" is the period immediately before the harvest of the crops that have borne fruit on the tree, among the multiple periods in which the tree loses its leaves. For example, a tree that is in the deciduous season from December to March of the following year goes through the deciduous season every year since planting. In such a case, if the harvesting time by the harvesting system 1 of this embodiment is June, the "deciduous period" of the "deciduous period three-dimensional position data" is the deciduous period from December immediately before June to March (in other words, the "deciduous period three-dimensional position data" of the "deciduous period" one year or more before the harvesting time is not used at the harvesting time). The "deciduous period" of the "deciduous period three-dimensional position data" may be a period when all the trees planted in the field have not lost their leaves. For example, the deciduous period may be a period when at least some of the leaves of one or more trees planted in the field have lost their leaves. In addition, the "defoliation" of a tree means "natural defoliation depending on various factors such as temperature and / or sunshine hours" or "leaves being removed from the branches and trunks of the tree by humans, and / or branches with leaves being removed from the branches and trunks of the tree by humans."

[0024] In FIG. 1, one server device 3 is illustrated as a component of the harvesting system 1, but the server device 3 may be a collection of multiple computers, each of which is a physically separate entity. In this case, each of the multiple computers may have the same functions, or may have the functions of the single server device 3 in a distributed manner. In the harvesting system 1, an information processing device such as a personal computer (PC), a notebook PC, and / or a tablet terminal, which has the same functions as the server device 3, may be used in place of or together with the server device 3.

[0025] (Harvesting Robot 2) Figure 2(a) is an oblique view showing an example of the appearance of the harvesting robot 2, Figure 2(b) is an oblique view showing an example of the appearance of the end effector system 22 equipped on the harvesting robot 2, and Figure 3 is a diagram showing an example of the general configuration of the harvesting robot 2.

[0026] As shown in FIGS. 2( a ) and 3 , the harvesting robot 2 includes a carriage 21 , an end effector system 22 , a manipulator system 23 , wheels 24 , and a support member 25 .

[0027] The carriage 21 has a control function for controlling the end effector system 22 and the manipulator system 23, a movement function for moving inside and outside the field, etc. As shown in Figures 2(a) and 3, the carriage 21 has a control device 211 and a movement device 212, thereby realizing the control function, the movement function, etc.

[0028] The control device 211 executes processing for making the harvesting robot 2 realize various functions, and outputs control signals to the moving device 212, as well as the sensor device 221, the driving device 222, the irradiation device 224, and the driving device 251, which will be described later. The control device 211 uses the three-dimensional position data during the deciduous season managed by the server device 3 to control the manipulator system 23 so that at least a part of the harvesting robot 2 does not collide with trees. In this way, the three-dimensional position data during the deciduous season is used by the control device 211 as obstacle data indicating obstacles to the harvesting robot 2. Note that although the "obstacle" is a "tree" (not including "leaves"), the "obstacle" may include an object that impedes the movement of the harvesting robot 2, the end effector system 22, and the manipulator system 23 (for example, "various members that support the branches (posts (pipes, stakes, etc.), wires, tapes, and / or strings, etc.)"). Hereinafter, all or a part of the three-dimensional position data during the deciduous season may be referred to as obstacle data during the deciduous season. The control device 211 includes a communication unit 211a, a storage unit 211b, and a processing unit 211c.

[0029] The communication unit 211a includes a communication interface circuit that performs wireless communication with an access point of a wireless LAN (Local Area Network) (not shown) based on a wireless communication method of a predetermined communication standard. The predetermined communication standard is, for example, the IEEE (The Institute of Electrical and Electronics Engineers, Inc.) 802.11 standard. The communication unit 211a receives various data and the like transmitted from the server device 3 via wireless communication. The communication unit 211a may transmit various data to the server device 3 via wireless communication.

[0030] The communication unit 211a may establish a wireless communication link with a base station (not shown) via a channel assigned by the base station, and communicate with the base station. The wireless communication link is, for example, a wireless communication link conforming to the LTE (Long Term Evolution) standard, a wireless communication link of a fourth generation mobile communication system (LTE-Advanced, 4G), or a wireless communication link of a mobile communication system such as a fifth generation mobile communication system (5G) standard.

[0031] The communication unit 211a may also have an interface circuit for performing short-distance wireless communication according to a communication method such as Bluetooth (registered trademark), receive radio waves corresponding to various data from the server device 3, and transmit radio waves corresponding to the various data to the server device 3. The communication unit 211a may also have a transmission circuit for transmitting various signals corresponding to the various data by infrared communication or the like, and a reception circuit for receiving various signals corresponding to the various data by infrared communication or the like. The communication unit 211a may also have a communication interface circuit for a wired LAN.

[0032] In this way, by being equipped with the communication unit 211a, the harvesting robot 2 is able to transmit various data to the server device 3 and receive various data from the server device 3 via the communication unit 211a.

[0033] The harvesting robot 2 may be provided with an input / output device that detachably holds a portable storage medium instead of or in addition to the communication unit 211a. When a portable storage medium storing data managed by the server device 3 is attached to the input / output device provided in the harvesting robot 2, the input / output device acquires the data stored in the portable storage medium and passes it to the processing unit 211c described below. This allows the harvesting robot 2 to receive data from the server device 3 via the portable storage medium. The input / output device also stores data in the portable storage medium in response to an instruction from the processing unit 211c. This allows the harvesting robot 2 to provide data to the server device 3 via the portable storage medium.

[0034] The storage unit 211b is, for example, a semiconductor memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 211b stores an operating system program, a driver program, an application program, data, and the like used in the processing in the processing unit 211c. The driver programs stored in the storage unit 211b are a communication device driver program for controlling the communication unit 211a, a sensor device driver program for controlling the sensor device 221, and a lighting device driver program for controlling the irradiation device 224. The application programs stored in the storage unit 211b are a harvesting obstacle data extraction process for extracting harvesting obstacle data (described later) based on sensor data from the sensor device 221, a defoliation obstacle data extraction process for extracting defoliation obstacle data based on sensor data from the sensor device 221, and a harvesting control process for controlling the end effector system 22 and the manipulator system 23 and controlling the movement of the moving device 212 during harvesting, and the like, and the like, for causing the processing unit 211c to execute such processes. Furthermore, the storage unit 211b may temporarily store data relating to a predetermined process.

[0035] The processing unit 211c is a processing device that loads the operating system program, driver program, and application program stored in the storage unit 211b into memory and executes instructions included in the loaded programs. The processing unit 211c is, for example, an electronic circuit such as a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP), or a combination of various electronic circuits. In FIG. 3, the processing unit 211c is illustrated as a single component, but the processing unit 211c may be a collection of multiple physically separate processors.

[0036] The processing unit 211c executes various commands included in the control program to realize various functions. An example of the functions realized by the processing unit 211c will be described in detail later.

[0037] The moving device 212 includes a steering mechanism including a drive motor that rotates the wheels 24 via a drive transmission mechanism (not shown), and a motor that changes the steering angle of the wheels 24. The moving device 212 controls the wheels 24 in accordance with a movement control signal from the processing unit 211c so that the carriage 21 moves along a specified movement path to a specified position inside or outside the field.

[0038] The end effector system 22 has a harvesting function for harvesting one or more types of crops. As shown in Fig. 2(b) and Fig. 3, the end effector system 22 has a sensor device 221, a driving device 222, a picking device 223, and an irradiation device 224 to realize the harvesting function.

[0039] The sensor device 221 has a function of acquiring data on the position of the crop, and includes, for example, an image capturing unit 221a for capturing an image of a subject and outputting captured image data of the subject, and a measurement unit 221b for measuring the distance to objects around the harvest robot 2. For example, an RGB-Depth sensor such as the "Intel RealSense Depth Camera D435i ("Intel RealSense" is a registered trademark)" may be used as the sensor device 221. The sensor device 221 is an example of a sensor unit.

[0040] The photographing unit 221a has an optical lens and an image sensor. The optical lens focuses a light beam from a subject on the image sensor's image sensor. The image sensor is a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and outputs an image of the subject image focused on the image sensor. The photographing unit 221a creates still image data in a predetermined file format from an image generated by the image sensor as photographed image data and outputs the created still image data. The photographing unit 221a may output the photographed image data in association with the acquisition time of the photographed image data. The photographing unit 221a may create moving image data in a predetermined file format from images continuously generated by the image sensor at predetermined intervals and output the captured image data.

[0041] The measuring unit 221b is, for example, a pair of infrared sensors. In the example shown in FIG. 2(b), a pair of a first lens 221b1 and a second lens 221b2 is provided in the sensor device 221. Each of the first lens 221b1 and the second lens 221b2 forms an image of infrared rays from an object in a space substantially the same as the shooting target space corresponding to the shooting image data acquired by the shooting unit 221a on a light receiving element. The measuring unit 221b outputs an infrared image corresponding to each of the first lens 221b1 and the second lens 221b2 based on an electrical signal from the light receiving element, measures distances corresponding to a plurality of points on the surface of the object from the output infrared image using a known active stereo method, and outputs depth data (distance data) corresponding to each of the plurality of points. The measuring unit 221b may output the depth data in association with the acquisition time of the depth data. The measuring unit 221b may create and output time-series depth data in a predetermined file format from the continuously output infrared images at predetermined intervals. The infrared (IR) projector 221c has a function of projecting infrared rays in order to improve the sensitivity of the measuring unit 221b to receiving infrared rays from an object.

[0042] The sensor device 221 supplies sensor data including the photographed image data output from the photographing unit 221a and the depth data output from the measuring unit 221b to the processing unit 211c. The sensor data (photographed image data and depth data) is an example of data related to the position of the crop. Only one of the photographed image data and the depth data may be used as the data related to the position of the crop.

[0043] The sensor device 221 is not limited to the above examples, and may be, for example, various sensor devices that output distance data corresponding to multiple points on the surface of an object (a pulse-echo ultrasonic distance sensor, or a laser scanner such as LiDAR (Light Detection and Ranging) or a laser range finder).

[0044] The driving device 222 is a driving motor or the like for operating the harvesting device 223 described later. The harvesting device 223 has a harvesting function of gripping a crop to be harvested and plucking the gripped crop from the branch. The harvesting device 223 shown in FIG. 2(b) has a gripping unit that grips the stalk when the crop to be harvested is a crop (such as cherries) that is attached to the branch via the stalk. The harvesting device 223 is not limited to the example shown in FIG. 2(b), and may have a configuration suitable for harvesting each crop to be harvested. The harvesting device 223 performs a harvesting operation suitable for each crop to be harvested (an operation of gripping a crop to be harvested and plucking the gripped crop from the branch) to harvest the crop. The harvesting device 223 is an example of a picking unit.

[0045] The irradiation device 224 is, for example, an LED lighting device, and irradiates light in response to an operation instruction from an operator who operates the harvesting robot 2 or a control instruction from the processing unit 211c.

[0046] The manipulator system 23 has a function of moving the picking device 223 of the end effector system 22 to a position where the crop can be harvested, etc. The manipulator system 23 shown in Fig. 2(a) is, for example, a vertical articulated manipulator having joints.

[0047] The support member 25 is installed on the upper surface of the dolly 21 and supports the manipulator system 23. The support member 25 may be configured to be movable relative to the dolly 21. The support member 25 includes a drive unit 251. The drive unit 251 is a drive motor or the like for rotating the manipulator system 23 with respect to the support member 25. For example, the drive unit 251 is rotatably connected to the tip of the manipulator system 23 and rotates the manipulator system 23 about two predetermined axes (for example, an axis in the y direction in FIG. 2(a) and an axis in a direction perpendicular to the axis in the y direction). This allows for the realization of movements similar to "shoulder rotation" and "arm raising" when the drive unit 251 and the manipulator system 23 are considered to be a human "shoulder" and "arm".

[0048] The manipulator system 23 may include a drive motor for rotating the end effector system 22 relative to the manipulator system 23. For example, the drive motor is rotatably connected to the tip of the manipulator system 23, and rotates the end effector system 22 about a predetermined axis of the harvesting robot 2 (for example, an axis in the extension direction of the manipulator system 23 and an axis in a direction perpendicular to the axis). This realizes movements similar to "wrist twisting motion" and "wrist bending motion" when the end effector system 22 and the manipulator system 23 are considered to be a human "hand" and "arm".

[0049] The movement device 212 moves the carriage 21 in accordance with the movement control signal from the processing unit 211c so as to be located within a range where the crop can be harvested by the end effector system 22. Then, the drive device 251 is controlled in accordance with the movement control signal from the processing unit 211c so as to move the end effector system 22 to a position where the crop can be harvested.

[0050] (Server device 3) FIG. 4 is a diagram illustrating an example of a schematic configuration of the server device 3. As shown in FIG.

[0051] The server device 3 has a data management function for storing and managing defoliation season obstacle data, and a transmission function for transmitting defoliation season obstacle data to the harvesting robot 2. The server device 3 also has a receiving function for receiving defoliation season obstacle data transmitted from an external device. The external device may be the harvesting robot 2. To this end, the server device 3 includes a server communication unit 31, a server storage unit 32, and a server processing unit 33.

[0052] The server communication unit 31 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination of these. The server device 3 can transmit various data, such as obstacle data during the leaf fall season, to the harvesting robot 2 via the server communication unit 31. The server device 3 can receive various data from an external device via the server communication unit 31. The server communication unit 31 may have an interface circuit for short-distance wireless communication according to a communication method such as Bluetooth (registered trademark), and may transmit radio waves to the harvesting robot 2 and receive radio waves from the harvesting robot 2. The server communication unit 31 may have a transmission circuit for transmitting various signals corresponding to various data by infrared communication or the like, and may have a reception circuit for receiving various signals corresponding to various data by infrared communication or the like. The server communication unit 31 may also have a communication interface circuit for a wired LAN. This allows the server device 3 to transmit and receive various data to and from the harvesting robot 2 via the server communication unit 31. The server device 3 may have an input / output device that detachably holds a portable storage medium instead of or in addition to the server communication unit 31. In this case, the input / output device acquires various data such as the defoliation season obstacle data stored in the portable storage medium, and supplies the acquired various data to the storage processing unit 332. This allows the server device 3 to acquire various data such as the defoliation season obstacle data from the external device via the portable storage medium.

[0053] The server storage unit 32 is, for example, a semiconductor memory device such as a ROM or a RAM. The server storage unit 32 may be, for example, a magnetic disk, an optical disk, or any other storage device capable of storing data. The server storage unit 32 stores an operating system program, a driver program, an application program, data, and the like used in the processing in the server processing unit 33. The computer program stored in the server storage unit 32 may be installed in the server storage unit 32 from a computer-readable portable storage medium such as a CD-ROM or a DVD-ROM using a known setup program or the like. The data stored in the server storage unit 32 includes various data such as defoliation obstacle data acquired from an external device. The server storage unit 32 may temporarily store data related to a predetermined processing.

[0054] The server processing unit 33 is a processing device that loads the operating system program, the driver program, and the application program stored in the server storage unit 32 into memory and executes instructions included in the loaded programs. The server processing unit 33 is, for example, an electronic circuit such as a CPU, an MPU, a DSP, or a combination of various electronic circuits. In Fig. 4, the server processing unit 33 is illustrated as a single component, but the server processing unit 33 may be a collection of multiple physically separate processors.

[0055] The server processing unit 33 executes various commands included in an application program (control program) stored in the server storage unit 32, and thereby functions as a server receiving unit 331, a storage processing unit 332, and a server transmitting unit 333.

[0056] The server receiving unit 331 receives various data, such as the defoliation obstacle data, transmitted from an external device via the server communication unit 31. The storage processing unit 332 stores the various data received by the server receiving unit 331 in the server memory unit 32. The server transmitting unit 333 transmits various data, such as the defoliation obstacle data, stored in the server memory unit 32 to the harvesting robot 2 via the server communication unit 31 in response to a request from the harvesting robot 2 or in response to an operation by the operator.

[0057] (Deaf season obstacle data extraction processing) An example of the deciduous season obstacle data extraction process will be described below with reference to Figs. 5 to 7. The deciduous season obstacle data extraction process is executed using sensor data acquired by the sensor device 221 of the harvest robot 2. For example, when the harvest robot 2 is moving through a field in which the trees of a crop to be harvested are planted during the season in which the trees have lost their leaves, the sensor device 221 of the harvest robot 2 acquires sensor data at a predetermined timing. The deciduous season obstacle data extraction process may be executed by an external device different from the harvest robot 2. For example, the external device may have the same functions as the sensor device 221 and the processing unit 211c of the harvest robot 2, and the deciduous season obstacle data extraction process may be executed by the external device during or after the operator moves using sensor data acquired while moving with the external device.

[0058] FIG. 5 is a diagram showing an example of an operational flow of the leaf-fall season obstacle data extraction process executed by the processing unit 211c.

[0059] First, in step S101, the processing unit 211c acquires sensor data output from the sensor device 221. The sensor device 221 may output sensor data at predetermined time intervals (for example, every second). In this case, the processing unit 211c acquires the sensor data output from the sensor device 221 every time the sensor data is output from the sensor device 221.

[0060] The sensor data includes the photographed image data output from the photographing unit 221a and the depth data output from the measuring unit 221b. FIG. 6(a) is a diagram showing an example of photographed image data photographed in a field where defoliated trees are planted. The photographed image data is, for example, data for displaying a rectangular image composed of a plurality of pixels, and RGB data for expressing a color is associated with each pixel. FIG. 6(b) is a schematic diagram showing an example of depth data (distance data) output when photographing the photographed image data shown in FIG. 6(a). The depth data (distance data) is, for example, a set of numerical data indicating the distance from the sensor device 221 corresponding to each of a plurality of pixels constituting a substantially identical rectangular shape to the photographed image data photographed at the same time. The resolution of the depth data may be the same as that of the photographed image data or may be different from that of the photographed image data. Hereinafter, the depth data measured during the period when the trees are defoliated may be referred to as defoliation period depth data.

[0061] Next, in step S102, the processing unit 211c acquires the captured image data included in the sensor data (acquired in step S101).

[0062] Next, in step S103, the processing unit 211c estimates an obstacle image area in the captured image data (obtained in step S102). An example of the obstacle image area estimation process executed by the processing unit 211c will be described below.

[0063] The processing unit 211c executes a process of estimating an obstacle image area using a learning model generated by deep learning based on, for example, U-Net (Convolutional Networks for Biomedical Image Segmentation).

[0064] U-Net is one of the methods for recognizing objects for each pixel constituting the photographed image data, and the processing unit 211c estimates an image area showing a "branch" in the photographed image data as an obstacle image area. The processing unit 211c may estimate, together with the image area showing the "branch", an image area showing various members (supports (pipes, stakes, etc.), wires, tapes, and / or strings, etc.) supporting the branch in the photographed image data as an obstacle image area. In this way, the processing unit 211c estimates, as an obstacle image area, an image area showing an obstacle that impedes the movement of the end effector system 22 and / or the manipulator system 23 during harvesting.

[0065] Fig. 7(a) is a diagram showing an example of estimated image data showing an obstacle image area. The estimated image data shown in Fig. 7(a) is output from the output layer of the trained neural network when the captured image data input to the input layer of the trained neural network is the captured image data shown in Fig. 6(a). The estimated image data is image data in which pixels showing obstacles and pixels showing non-obstacles are displayed in different ways.

[0066] The process of generating the learning model is executed by the processing unit 211c of the harvesting robot 2, the server processing unit 33 of the server device 3, or a processing unit of an external device.

[0067] For example, a plurality of pieces of teacher data are created by an operator or manager of the harvesting robot 2. Then, deep learning is performed using the plurality of pieces of teacher data, thereby generating or updating a learning model in which the weights of each neuron in the neural network are learned.

[0068] Next, the process returns to step S104, and the processing unit 211c determines whether or not an obstacle image area has been estimated in step S103 (whether or not an obstacle image area exists in the estimated image data output in step S103).

[0069] When an obstacle image area has not been estimated in step S103 (step S104-No), the processing unit 211c advances the leaf-fall season obstacle data extraction process to step S108.

[0070] If an obstacle image area is estimated in step S103 (step S104-Yes), in step S105, the processing unit 211c acquires defoliation depth data included in the sensor data (acquired in step S101).

[0071] Next, in step S106, the processing unit 211c calculates 3D point cloud data in the deciduous season based on the deciduous season depth data (acquired in step S105). The deciduous season 3D point cloud data is a set of 3D position data corresponding to each pixel of the deciduous season depth data.

[0072] The calculation process of 3D point cloud data during the deciduous season is a process of calculating 3D point cloud data during the deciduous season by simultaneously converting the position and depth data of each pixel of the deciduous season depth data into 3D position data according to a specified position coordinate system (for example, a geographic coordinate system such as the Japanese Geodetic System 2011, which is indicated by latitude, longitude, and altitude).

[0073] As a calculation process for the three-dimensional point cloud data during the deciduous season, for example, the processing unit 211c may execute a process of calculating the three-dimensional point cloud data during the deciduous season based on a known self-position estimation and environmental map creation method such as SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion).

[0074] When the calculation process of the three-dimensional point cloud data in the deciduous season based on the self-location estimation and environmental map creation technique is executed, the depth data included in the sensor data may not be used. In this case, the processing unit 211c may calculate the relative positions in the virtual three-dimensional space for each of the shooting positions of the two pieces of photographed image data and the positions corresponding to the multiple feature points, using only multiple pieces of photographed image data with different shooting positions, according to SLAM, SfM, or the like, to calculate the three-dimensional point cloud data in the deciduous season. When the processing unit 211c executes the calculation process of the three-dimensional point cloud data in the deciduous season based on the self-location estimation and environmental map creation technique, it may execute a known calibration process by passing again a point that has been passed since starting to acquire sensor data (for details of the calibration process, refer to Japanese Patent No. 6525148, etc.).

[0075] Next, returning to step S107 of the defoliation obstacle data extraction process shown in FIG. 5, the processing unit 211c extracts defoliation obstacle data from the calculated defoliation obstacle three-dimensional point cloud data. For example, the processing unit 211c superimposes the defoliation obstacle data and the estimated image data so that the measurement target space of the defoliation obstacle depth data (acquired in step S105) and the shooting target space of the estimated image data (estimated in step S103) are substantially the same. Next, the processing unit 211c extracts pixels that are superimposed on the obstacle image area in the estimated image data from among the pixels of the defoliation obstacle depth data. The processing unit 211c extracts the defoliation obstacle three-dimensional point cloud data corresponding to the extracted pixels as defoliation obstacle data. FIG. 7(b) is a schematic diagram showing an example of defoliation obstacle data extracted based on the defoliation obstacle depth data shown in FIG. 6(b).

[0076] Next, in step S108, the processing unit 211c judges whether or not the acquisition of sensor data from the sensor device 221 has been completed. If it is judged that new sensor data is to be acquired from the sensor device 221 (step S108-No), the processing unit 211c returns the defoliation season obstacle data extraction process to step S101. If it is judged that the acquisition of sensor data from the sensor device 221 has been completed (step S108-Yes), the processing unit 211c ends the defoliation season obstacle data extraction process.

[0077] When the leaf-fall season obstacle data extraction process is completed, the processing unit 211c transmits the leaf-fall season obstacle data to the server device 3 via the communication unit 211a. The server receiving unit 331 of the server device 3 receives the leaf-fall season obstacle data via the server communication unit 31, and the storage processing unit 332 stores the received leaf-fall season obstacle data in the server storage unit 32. The processing unit 211c may store the leaf-fall season obstacle data in the storage unit 211b without transmitting the leaf-fall season obstacle data to the server device 3.

[0078] (Harvesting) When the crops are ready to be harvested, the harvest robot 2 harvests them. An example of the harvesting process executed by the processing unit 211c of the harvest robot 2 will be described below with reference to FIG. 8. Note that before the harvesting process is executed, the server transmission unit 333 of the server device 3 transmits the defoliation season obstacle data stored in the server storage unit 32 to the harvest robot 2 via the server communication unit 31. The processing unit 211c of the harvest robot 2 receives the defoliation season obstacle data via the communication unit 211a and stores the received defoliation season obstacle data in the storage unit 211b. Note that the processing unit 211c does not need to receive the defoliation season obstacle data from the server device 3 when the defoliation season obstacle data is stored in the storage unit 211b.

[0079] First, in step S201, the processing unit 211c acquires sensor data output from the sensor device 221. The sensor device 221 may output sensor data at predetermined time intervals (for example, every second). In this case, the processing unit 211c acquires the sensor data output from the sensor device 221 every time the sensor data is output from the sensor device 221.

[0080] Next, in step S202, the processing unit 211c acquires the captured image data included in the sensor data (acquired in step S201).

[0081] Next, in step S203, the processing unit 211c detects the presence or absence of crops by determining whether or not the captured image data (acquired in step S202) includes crops to be harvested. An example of the crop detection process executed by the processing unit 211c will be described below. Note that the case where the crop to be detected is "cherries" will be described as an example.

[0082] First, the processing unit 211c instructs the sensor device 221 to acquire sensor data. Note that the processing unit 211c may instruct the sensor device 221 to acquire sensor data while the irradiation device 224 is irradiating light.

[0083] Next, the processing unit 211c estimates a candidate image area corresponding to a cherry fruit based on the captured image data included in the sensor data acquired from the sensor device 221. For example, the processing unit 211c executes a process of determining whether or not the candidate image area corresponds to a cherry fruit, using a learning model generated by machine learning based on a known SVM (Support Vector Machine).

[0084] If the processing unit 211c estimates that the candidate image area corresponding to the cherry fruit is not present in the image data (step S203-No), the processing unit 211c advances the harvesting process to step S209. If the processing unit 211c estimates that the candidate image area corresponding to the cherry fruit is present in the image data (step S203-Yes), the processing unit 211c further estimates an image area corresponding to a peduncle connected to the cherry fruit in the candidate image area by using a predetermined learning model, and calculates the position of the crop (step S204).

[0085] For example, in step S204, the processing unit 211c outputs three-dimensional position data of the stalk estimated from the candidate image area (three-dimensional position data in the camera coordinate system) as the position of the crop. For example, the processing unit 211c calculates three-dimensional coordinates (x, y, z) in the camera coordinate system corresponding to one or more pixels corresponding to the stalk based on one or more pixels in the image area corresponding to the stalk and the depth value indicated in the depth data. Then, the processing unit 211c outputs three-dimensional position data indicating the calculated three-dimensional coordinates (x, y, z) as the position of the crop. In this way, the processing unit 211c makes it possible to detect the crop to be harvested based on the sensor data.

[0086] In addition, as a method for determining the crop to be harvested, a learning model generated by machine learning based on a CNN (Convolutional Neural Network) such as SSD (Single Shot Multibox Detector) or YOLO (You Only Look Once) may be used to determine the crop to be harvested. The object recognition method using SSD is described in W. Liu, D. Anguelov, D. Erhan, C. Szegedy, and SE Reed, “SSD: Single Shot Multibox Detector”, December 29, 2016, [online],<https: / / arxiv.org. / pdf / 1512.02325.pdf> Please refer to, etc.

[0087] Next, in step S205, the processing unit 211c executes a harvesting obstacle data extraction process, the details of which will be described later.

[0088] Next, in step S206, the processing unit 211c acquires the leaf-shedding season obstacle data stored in the storage unit 211b.

[0089] Next, in step S207, the processing unit 211c converts the three-dimensional coordinates (X, Y, Z) in a predetermined position coordinate system indicated by the acquired defoliation obstacle data into three-dimensional coordinates (x, y, z) in a camera coordinate system, for example, using the camera parameters of the measurement unit 221b, thereby aligning the position of the defoliation obstacle data (three-dimensional coordinates (X, Y, Z) in the predetermined position coordinate system) with the shooting target space (three-dimensional space in the camera coordinate system) within a predetermined field of view in the current shooting direction of the sensor device 221.

[0090] In addition, when defoliation season obstacle data and shooting time obstacle data are generated based on the self-position estimation and environmental map creation techniques, a three-dimensional model (e.g., a known surface model) may be generated based on each obstacle data, and the position of the defoliation season obstacle data may be aligned with the shooting target space (three-dimensional space in the camera coordinate system) within a specified field of view angle in the current shooting direction of the sensor device 221 by rotating and / or moving the defoliation season obstacle data so that the two three-dimensional models approximately match.

[0091] Next, in step S208, the processing unit 211c executes a harvesting control process for controlling the end effector system 22, the manipulator system 23, and the moving device 212 based on the defoliation season obstacle data after the alignment process and the harvesting obstacle data extracted in step S205. The harvesting control process will be described in detail later.

[0092] Next, in step S209, the processing unit 211c determines whether or not to harvest another crop. For example, the processing unit 211c may determine to further harvest another crop in response to a harvest instruction from the operator. In this case, the processing unit 211c may determine to end the harvest of the other crop in response to an end instruction from the operator.

[0093] If it is determined that other crops are to be harvested (step S209-Yes), the processing unit 211c returns the harvesting process to step S201, and if it is determined that the harvesting of other crops is to be ended (step S209-No), the processing unit 211c ends the harvesting process.

[0094] (Obstacle data extraction processing during harvesting) An example of the harvest obstacle data extraction process will be described below with reference to Figures 9 and 10. The harvest obstacle data extraction process is executed in step S205 of the harvest process using sensor data acquired by the sensor device 221 of the harvest robot 2. For example, during harvesting, when the harvest robot 2 moves through a farm field where trees bearing crops to be harvested are planted, the sensor device 221 of the harvest robot 2 acquires sensor data at a predetermined timing.

[0095] FIG. 9 is a diagram showing an example of an operational flow of the harvesting obstacle data extraction process executed by the processing unit 211c.

[0096] First, in step S301, the processing unit 211c acquires the captured image data included in the sensor data (acquired in step S201). The process of step S301 is the same as the process of step S102. Fig. 10(a) is a diagram showing an example of captured image data of a crop to be harvested at harvest time. Fig. 10(b) is a schematic diagram showing an example of harvest depth data measured when the captured image data shown in Fig. 10(a) was captured.

[0097] Next, in step S302, the processing unit 211c estimates an obstacle image area in the captured image data (acquired in step S301). The process of step S302 is similar to the process of step S103. Fig. 10(c) is a diagram showing an example of estimated image data indicating an obstacle image area estimated from the captured image data shown in Fig. 10(a).

[0098] Next, in step S303, the processing unit 211c determines whether or not an obstacle image area has been estimated in step S302. The process of step S303 is similar to the process of step S104.

[0099] When the obstacle image area has not been estimated in step S302 (step S303-No), the processing unit 211c ends the harvesting obstacle data extraction process.

[0100] If the obstacle image area is estimated in step S302 (step S303-Yes), in step S304, the processing unit 211c acquires harvest depth data included in the sensor data (acquired in step S201). Note that the process of step S304 is the same as the process of step S105.

[0101] Next, in step S305, the processing unit 211c calculates three-dimensional point cloud data at harvest based on the depth data at harvest (acquired in step S304). For example, the three-dimensional point cloud data at harvest is three-dimensional position data indicating three-dimensional coordinates (x, y, z) of the camera coordinate system corresponding to each pixel, calculated from each pixel of the depth data at harvest and the depth data of each pixel.

[0102] Then, in step S306, the processing unit 211c extracts harvest obstacle data from the calculated harvest three-dimensional point cloud data, and ends the harvest obstacle data extraction process. The process of step S306 is the same as the process of step S107. FIG. 10(d) is a schematic diagram showing an example of harvest obstacle data extracted based on the harvest depth data shown in FIG. 10(b). The harvest obstacle data is an example of harvest three-dimensional position data. The harvest obstacle data extracted by the harvest obstacle data extraction process is harvest three-dimensional point cloud data corresponding to a tree that does not include "leaves". The "leaves" of a tree are objects that do not hinder the movement of the harvest robot 2, the end effector system 22, and the manipulator system 23, and the processing unit 211c makes it possible to generate harvest obstacle data that does not include three-dimensional point cloud data corresponding to "leaves".

[0103] (Harvest Control Treatment) An example of the harvest control process will be described below with reference to Fig. 11. The harvest control process is executed by the processing unit 211c in step S208 of the harvest process.

[0104] First, in step S401, the processing unit 211c determines whether or not at least a part of the harvesting robot 2 will collide with an obstacle when the end effector system 22 is moved from its current position to a position where a harvesting operation can be performed on a crop to be harvested. The crop to be harvested is the crop corresponding to the position output in step S204. At least a part of the harvesting robot 2 is, for example, a part of the end effector system 22 and the manipulator system 23.

[0105] First, the processing unit 211c estimates at least one of the position of the obstacle and the shape of the obstacle based on at least one of the obstacle data during defoliation and the obstacle data during harvest. For example, the processing unit 211c generates a known surface model based on each of the three-dimensional position data of the obstacle data during defoliation and the obstacle data during harvest. The surface model may be represented by a predetermined three-dimensional object (e.g., a cylindrical object) data, or may be represented by polygon data in which the surface is represented by polygons. In this way, the surface model is generated, and the position and shape of the obstacle are estimated.

[0106] Next, the processing unit 211c calculates the movement paths of the end effector system 22 and the manipulator system 23 accompanying the movement of the end effector system 22. Then, the processing unit 211c determines whether or not the generated surface model intersects with the calculated movement path. If the processing unit 211c determines that the generated surface model intersects with the calculated movement path, it determines that at least a part of the harvesting robot 2 will collide with an obstacle. If the processing unit 211c determines that the generated surface model does not intersect with the calculated movement path, it determines that at least a part of the harvesting robot 2 will not collide with an obstacle. The processing unit 211c may determine whether or not the calculated movement path intersects with at least one of the deciduous season obstacle data and the harvesting obstacle data. In this case, if the processing unit 211c determines that at least one of the deciduous season obstacle data and the harvesting obstacle data intersects with the calculated movement path, it determines that at least a part of the harvesting robot 2 will collide with an obstacle, and if it determines that at least one of the deciduous season obstacle data and the harvesting obstacle data does not intersect with the calculated movement path, it determines that at least a part of the harvesting robot 2 will not collide with an obstacle.

[0107] If it is determined that at least part of the harvesting robot 2 will collide with an obstacle (step S401-Yes), in step S402, the processing unit 211c notifies at least one of the moving device 212, the driving device 251, the manipulator system 23, and the driving device 222 of an avoidance control instruction, and returns processing to step S401.

[0108] For example, when the processing unit 211c determines that there is an action that does not cause the harvesting robot 2 to collide with an obstacle among a plurality of actions (movement paths) of the end effector system 22 and the manipulator system 23 for moving the manipulator system 23, the processing unit 211c controls at least one of the driving device 251, the manipulator system 23, and the driving device 222 according to the control information of the action. Also, if at least a part of the harvesting robot 2 is sure to collide with an obstacle when the end effector system 22 is moved from the current position of the harvesting robot 2 to a position where a picking action is possible, the processing unit 211c notifies the moving device 212 of a movement instruction and moves the cart 21 to change the position of the harvesting robot 2.

[0109] If it is determined that at least a part of the harvesting robot 2 will not collide with an obstacle (step S401-No), in step S403, the processing unit 211c notifies at least one of the driving device 251, the manipulator system 23, and the driving device 222 of a harvesting control instruction, and ends the harvesting control process. The harvesting control process enables the manipulator system 23 to move to a position where it can pick a crop, and the picking device 223 to perform a predetermined picking operation.

[0110] It should be noted that the harvest control process may be executed using only the defoliation obstacle data, without using the harvest obstacle data.

[0111] As described above in detail, the harvesting system 1 of this embodiment uses sensor data acquired when the trees of the crop to be harvested have lost their leaves to determine the presence or absence of obstacles that may impede harvesting. This makes it possible for the harvesting system 1 of this embodiment to prevent accidents in which the obstacle is hidden by the leaves of the trees and goes undetected, resulting in the harvesting robot 2 colliding with the obstacle.

[0112] (Variation 1) The defoliation stage obstacle data extraction process (FIG. 5) may include a data addition step of adding flower bud position data indicating the position of the flower bud to the three-dimensional point cloud data at harvest. For example, the data addition step is executed when it is determined that the acquisition of sensor data from the sensor device 221 has ended (step S108-Yes). The data addition step is, for example, a process in which a learning model (for example, a model based on a known SVM) generated by machine learning determines whether or not the position of the flower bud exists in the captured image data, and when it is determined that the position of the flower bud exists, a label indicating the position of the flower bud is added to the three-dimensional point cloud data at harvest corresponding to the position. The data addition step may be an operation process in which an operator adds flower bud position data regarding the position of the flower bud to the three-dimensional point cloud data at harvest.

[0113] When the flower bud position data is added to the harvest time 3D point cloud data, for example, in step S203 of the harvest process, it may be determined whether the captured image data includes a crop to be harvested depending on whether the captured image data includes the flower bud position data. Also, in step S204 of the harvest process, the flower bud position data may be output as the position of the crop.

[0114] In this way, by adding flower bud position data to the 3D point cloud data at the time of harvest, it is possible to estimate the position of the crop even in cases where it is not possible to determine whether or not the crop is present in the captured image data.

[0115] (Variation 2) The "defoliation period" in the "defoliation period three-dimensional position data" used in the harvesting system 1 of this embodiment may be a "defoliation period" one year or more before the harvest time. [Explanation of symbols]

[0116] 1. Harvesting System 2. Harvesting robot 21 Trolley 211 Control device 211a Communications Department 211b Storage section 211c Processing section 212 Mobile Device 22 End effector system 221 Sensor Device 221a Photography Department 221b Measuring section 222 Drive unit 223 Picking device 224 Irradiation device 23 Manipulator System 24 wheels 25 Support member 251 Drive unit 3. Server equipment 31 Server Communication Department 32 Server memory section 33 Server Processing Unit 331 Server Reception Unit 332 Memory Processing Unit 333 Server Processing Unit

Claims

1. A harvesting robot that harvests crops that grow on deciduous trees, a storage unit that stores three-dimensional position data during the leaf-shedding season, the data including the position of at least a portion of the tree from which leaves have fallen; A harvesting unit for harvesting the crop; A manipulator that moves the picking unit; A sensor unit for acquiring data relating to the position of the crop; Detecting a crop to be harvested based on the position data acquired by the sensor unit at the time of harvesting the crop; estimating at least a position of an obstacle that impedes the movement of the picking unit based on the leaf-fall stage three-dimensional position data; When the harvest target crop is detected, the picking unit controls the manipulator so as to move to a harvestable position of the detected crop without causing collision between the estimated obstacle and at least a part of the harvesting robot. A processing section; A harvesting robot comprising:

2. The processing unit includes: When the harvest target crop is detected, harvest time three-dimensional position data including a position of at least a part of the obstacle is acquired based on the position data acquired by the sensor unit; The harvesting robot according to claim 1 , wherein at least a position of an obstacle that impedes movement of the picking unit is estimated based on the three-dimensional position data during leaf fall and the three-dimensional position data during harvest.

3. the sensor unit has an image capturing unit that captures an image of a subject and outputs captured image data of the subject, and a measurement unit that measures distances to surrounding objects and outputs distance data indicating the measured distances; The processing unit includes: calculating three-dimensional position data corresponding to one or more points in the captured image data based on the output distance data corresponding to the one or more points; The harvesting robot according to claim 2 , wherein the calculated three-dimensional position data is acquired as the three-dimensional position data at the time of harvesting.

4. The harvesting robot of claim 3, wherein the three-dimensional position data during the defoliation period is three-dimensional position data corresponding to one or more points, calculated based on the image data captured by the photographing unit during the defoliation period immediately prior to the harvesting of the crop, and distance data corresponding to one or more points within the captured image data output by the measuring unit at the time of capturing the image.

5. The processing unit includes: From the captured image data, an image area showing a branch of the tree and an image area showing a member supporting the branch of the tree are estimated as obstacle image areas; The harvesting robot according to claim 3 or 4, wherein three-dimensional position data within the obstacle image area among the calculated three-dimensional position data is acquired as the harvesting time three-dimensional position data.

6. A cart that moves within a field in which the trees are planted is provided, The harvesting robot according to any one of claims 1 to 5, wherein, when the crop to be harvested is detected, the processing unit controls the movement of the cart so that the picking unit can move to a harvestable position of the detected crop without colliding with the estimated obstacle and at least a part of the harvesting robot.

7. The harvesting robot according to any one of claims 1 to 6, wherein the processing unit estimates a position and a shape of the obstacle based on the three-dimensional position data during the leaf-shedding season.

8. A method for controlling a harvesting robot that harvests crops that grow on deciduous trees, the harvesting robot comprising: a memory unit that stores three-dimensional position data during a deciduous season that includes a position of at least a portion of the tree from which leaves have fallen; a picking unit that harvests the crops; a manipulator that moves the picking unit; a sensor unit that acquires data on the position of the crops; and a processing unit, The processing unit includes: Detecting a crop to be harvested based on the position data acquired by the sensor unit at the time of harvesting the crop; estimating at least a position of an obstacle that impedes the movement of the picking unit based on the leaf-fall stage three-dimensional position data; When the harvest target crop is detected, the picking unit controls the manipulator so as to move to a harvestable position of the detected crop without causing collision between the estimated obstacle and at least a part of the harvesting robot. The control method includes:

9. A control program for a harvesting robot that harvests crops that grow on deciduous trees, the harvesting robot comprising: a memory unit that stores three-dimensional position data during the deciduous season, the data including the position of at least a portion of the tree from which leaves have fallen; a picking unit that harvests the crops; a manipulator that moves the picking unit; a sensor unit that acquires data regarding the position of the crops; and a processing unit, The processing unit includes: Detecting a crop to be harvested based on the position data acquired by the sensor unit at the time of harvesting the crop; estimating at least a position of an obstacle that impedes the movement of the picking unit based on the leaf-fall stage three-dimensional position data; When the harvest target crop is detected, the picking unit controls the manipulator so as to move to a harvestable position of the detected crop without causing collision between the estimated obstacle and at least a part of the harvesting robot. A control program for executing the above operations.

10. A harvesting system having a harvesting robot that harvests crops that grow on deciduous trees and a server device that communicates with the harvesting robot, The server device includes: a storage unit that stores three-dimensional position data during the leaf-shedding season, the data including the position of at least a portion of the tree from which leaves have fallen; A transmitter that transmits the three-dimensional position data during the leaf-shedding period to the harvesting robot; Equipped with The harvesting robot is a receiving unit for receiving the leaf-shedding season three-dimensional position data transmitted from the server device; A harvesting unit for harvesting the crop; A manipulator that moves the picking unit; A sensor unit for acquiring data relating to the position of the crop; Detecting a crop to be harvested based on the position data acquired by the sensor unit at the time of harvesting the crop; estimating at least a position of an obstacle that impedes movement of the picking unit based on the received three-dimensional position data during the leaf fall period; When the harvest target crop is detected, the picking unit controls the manipulator so as to move to a harvestable position of the detected crop without causing collision between the estimated obstacle and at least a part of the harvesting robot. A processing section; A harvesting system comprising:

Images