Harvesting system, harvesting method, and program

Abstract

To increase a harvesting amount of fruits SF by an automatic harvesting device 5.SOLUTION: A plant product harvesting system Sys includes: an imaging device control unit 213 for acquiring image information DG showing an image GG of an imaged cultivation area from an imaging device 54 for imaging a cultivation area in which fruits SF of strawberries S; an object recognition unit 22 for recognizing an object Obj included in the image GG shown by the image information DG; a harvest decision unit 23 for deciding whether or not to harvest the fruit SF by an automatic harvesting device 5 when an obstacle OB which may hinder the harvesting of the fruits SF by the automatic harvesting device 5 is present in the object Obj recognized by the object recognizing unit 22; and a manipulator control unit 212 which controls the automatic harvesting device 5 to harvest the fruits SF when the harvest decision unit 23 decides to harvest the fruits SF.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a harvesting system, a harvesting method, and a program. [Background technology]

[0002] In recent years, technology related to harvesting devices for harvesting fruits and vegetables, fruits, and other vegetables (hereinafter sometimes referred to as "harvesting objects") has become known. For example, Patent Document 1 discloses a technology that determines the presence or absence of an obstacle that will hinder the harvesting of fruits and vegetables based on an image capture result of an area including the fruits and vegetables, and controls the harvesting device so that the fruits and vegetables are harvested by the harvesting device if no obstacle is present, and the harvesting device does not harvest the fruits and vegetables if an obstacle is present. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-176087 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, according to conventional techniques, if an obstacle is present, the harvester will not harvest the fruits and vegetables, which may result in a large amount of fruits and vegetables remaining unharvested by the harvester.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and one of the problems to be solved is to increase the harvest yield of fruits and vegetables by a harvesting device compared to conventional techniques. [Means for solving the problem]

[0006] In order to solve the above problems, the harvesting system of the present invention is characterized in that it comprises an acquisition unit that acquires imaging information indicating an image of a cultivation area in which harvest objects are grown from an imaging device that images the cultivation area, a recognition unit that recognizes objects contained in the image indicated by the imaging information, a decision unit that decides whether or not to harvest the harvest object by the harvesting device when an obstacle that hinders the harvesting of the harvest object by the harvesting device is present among the objects recognized by the recognition unit, and an apparatus control unit that controls the harvesting device to harvest the harvest object when the decision unit decides to harvest the harvest object.

[0007] In addition, the harvesting method of the present invention is characterized in that it acquires imaging information indicating an image of a cultivation area in which harvest objects are grown from an imaging device that images the cultivation area, recognizes objects contained in the image indicated by the imaging information, and if an obstacle that hinders a harvesting device from harvesting the harvest objects by the harvesting device is present among the recognized objects, determines whether or not to harvest the harvest objects by the harvesting device, and if it is determined that the harvest objects should be harvested, controls the harvesting device to harvest the harvest objects.

[0008] In addition, the program of the present invention is characterized in that it causes a processor to function as an acquisition unit that acquires imaging information indicating an image of a cultivation area in which harvest objects are grown from an imaging device that images the cultivation area, a recognition unit that recognizes objects contained in the image indicated by the imaging information, a decision unit that decides whether or not to harvest the harvest object using the harvesting device when an obstacle that hinders the harvesting of the harvest object by the harvesting device is present among the objects recognized by the recognition unit, and an apparatus control unit that controls the harvesting device to harvest the harvest object when the decision unit decides to harvest the harvest object. Effect of the Invention

[0009] According to the present invention, the yield of harvested objects by a harvesting device can be increased as compared to the conventional techniques. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a plant production and harvesting system Sys according to an embodiment of the present invention. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of the internal environment of an agricultural greenhouse 8. [Diagram 3] FIG. 2 is an explanatory diagram showing an example of the internal environment of an agricultural greenhouse 8. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of an automatic harvesting device 5. [Diagram 5] 1 is a block diagram showing an example of the configuration of a plant harvest managing server 1. FIG. [Figure 6] FIG. 2 is a block diagram showing an example of the configuration of an administrator terminal device 9. [Figure 7] FIG. 4 is a diagram showing an example of the data configuration of a detected object information DB. [Figure 8] 11 is a diagram showing an example of the data structure of detected fruit information DF. FIG. [Figure 9] 13 is a flowchart showing an example of a harvesting process. [Figure 10] 13 is a flowchart showing an example of a harvest readiness determination process. [Figure 11] 13 is a flowchart showing an example of a fruit status determination process. [Figure 12] 13 is a flowchart showing an example of a stem state determination process. [Figure 13] 13 is a flowchart illustrating an example of an obstacle analysis process. [Figure 14] 13 is a flowchart illustrating an example of an obstacle analysis process. [Figure 15] 13 is a flowchart illustrating an example of an obstacle analysis process. [Figure 16] FIG. 11 is an explanatory diagram showing an example of the state of fruit SF. [Figure 17] FIG. 11 is an explanatory diagram showing an example of the state of fruit SF. [Figure 18] FIG. 11 is an explanatory diagram showing an example of the state of fruit SF. [Figure 19] FIG. 11 is an explanatory diagram showing an example of the state of fruit SF. [Figure 20] It is a flowchart showing an example of the harvesting process according to Modification 3 of the present invention. [Figure 21] It is a block diagram showing an example of the configuration of the automatic harvesting device 5B according to Modification 4 of the present invention. [Figure 22] It is a block diagram showing an example of the configuration of the plant harvesting management server 1B according to Modification 4 of the present invention. [Figure 23] It is a block diagram showing an example of the configuration of the plant production harvesting system Sys-C according to Modification 5 of the present invention.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. Note that, in each figure, the dimensions and scales of each part are appropriately different from the actual ones. Further, the following-described embodiments are preferred specific examples of the present invention, and thus various technically preferable limitations are imposed. However, the scope of the present invention is not limited to these embodiments unless otherwise specifically stated to limit the present invention in the following description.

[0012] <A. Embodiment> Hereinafter, an embodiment of the present invention will be described.

[0013] <A.1. Outline of the Plant Production Harvesting System> Hereinafter, the outline of the plant production harvesting system Sys will be described with reference to FIGS. 1 to 6.

[0014] <A.1.1. Overall Outline of the Plant Production Harvesting System> FIG. 1 is a block diagram showing an example of the outline of the configuration of the plant production harvesting system Sys.

[0015] As illustrated in FIG. 1, the plant production harvesting system Sys includes a plant harvesting management server 1, an agricultural house 8, and an administrator terminal device 9. Note that, in the present embodiment, the plant production harvesting system Sys is an example of a "harvesting system".

[0016] The agricultural greenhouse 8 is a facility for cultivating plants, such as a plastic greenhouse or a glass greenhouse. A part or all of the outer walls, such as the side walls and roof, of the agricultural greenhouse 8 are made of a light-transmitting material, such as plastic, vinyl, or glass. The vinyl may be, for example, an agricultural polyolefin film (also called an agricultural PO film), or a film made of a fluororesin.

[0017] In this embodiment, as an example, it is assumed that a plant to be cultivated and harvested by the plant production and harvesting system Sys is cultivated in the agricultural greenhouse 8. That is, in this embodiment, as an example, it is assumed that the environment for cultivating a plant to be cultivated and harvested by the plant production and harvesting system Sys (hereinafter, referred to as "plant cultivation environment") is the internal environment of the agricultural greenhouse 8 (i.e., the indoor environment). However, the present invention is not limited to such an embodiment. For example, the plant cultivation environment may be a farmland provided outdoors (i.e., an outdoor environment).

[0018] In addition, in this embodiment, as an example, it is assumed that the plant cultivated in the agricultural greenhouse 8 is a strawberry. That is, in this embodiment, as an example, it is assumed that the fruit or vegetable of the plant to be harvested by the plant production and harvesting system Sys is a strawberry. However, the present invention is not limited to such an embodiment. Fruits and vegetables of plants to be harvested by the plant production and harvesting system Sys may be fruits other than strawberries, such as apples or melons, or edible parts of vegetables, such as eggplant or corn, or ornamental parts of ornamental plants, such as rose flowers or carnation flowers. In the following, the fruits and vegetables of the plants to be harvested by the plant production and harvesting system Sys may be referred to as "harvest target OT."

[0019] In addition, in the present embodiment, as an example, it is assumed that the variety of strawberries cultivated in the agricultural greenhouse 8 is "Amaou" (registered trademark). However, the present invention is not limited to such a mode. The variety of strawberries cultivated in the agricultural greenhouse 8 may be "Tochiotome" (registered trademark) or other varieties.

[0020] The plant harvest management server 1 can communicate with the agricultural greenhouse 8 via the network NW. The plant harvest management server 1 controls various devices provided in the agricultural greenhouse 8.

[0021] The administrator terminal device 9 can communicate with the plant harvest management server 1 via the network NW. The user of the administrator terminal device 9 supplies a harvest start signal for instructing the start of the harvest of the harvest target object OT in the agricultural greenhouse 8 from the administrator terminal device 9 to the plant harvest management server 1. The plant harvest management server 1 that has received the harvest start signal controls various devices provided in the agricultural greenhouse 8 so that the harvest of the harvest target object OT is started in the agricultural greenhouse 8.

[0022] <A.1.2. Overview of Agricultural Greenhouse> Hereinafter, with reference to FIGS. 2 to 3, the overview of the agricultural greenhouse 8 will be described.

[0023] FIGS. 2 and 3 are explanatory diagrams for explaining an example of the internal environment of the agricultural greenhouse 8. Among these, FIG. 2 is a view of the internal environment of the agricultural greenhouse 8 as seen from the horizontal direction. FIG. 3 is a plan view of the internal environment of the agricultural greenhouse 8 as seen from above.

[0024] In the following, the vertically upward direction is referred to as the Z1 direction, the horizontal direction orthogonal to the Z1 direction is referred to as the X1 direction, and the horizontal direction orthogonal to the Z1 direction and the X1 direction is referred to as the Y1 direction. Also, in the following, the Z1 direction and the Z2 direction opposite to the Z1 direction are collectively referred to as the Z-axis direction, the X1 direction and the X2 direction opposite to the X1 direction are collectively referred to as the X-axis direction, and the Y1 direction and the Y2 direction opposite to the Y1 direction are collectively referred to as the Y-axis direction.

[0025] 2, in this embodiment, strawberries S are grown on a plant cultivation shelf PL inside an agricultural greenhouse 8. The strawberries S have fruits SF, leaves SL, and stems SK. In the following, among the fruits SF of the strawberry S, the fruits SF that are mature enough to be harvested are referred to as mature fruits SFA. In the following, among the fruits SF of the strawberry S, the fruits SF that are not mature enough to be harvested are referred to as immature fruits SFB.

[0026] As illustrated in FIG. 2, in this embodiment, an automatic harvesting device 5 for harvesting fruits SF of strawberries S is disposed inside an agricultural greenhouse 8. Although details will be described later, the automatic harvesting device 5 includes a control device 51 for controlling each part of the automatic harvesting device 5, a traveling device 52 for changing the position and posture of the automatic harvesting device 5 by self-propelling the automatic harvesting device 5, a manipulator 53 for harvesting the fruits SF of the strawberries S by cutting the fruits SF from the stems SK, an imaging device 54 for imaging an area including the strawberries S, a detection device 55 equipped with various sensors, and a communication device 56 for communicating with an external device outside the automatic harvesting device 5. Under the control of the control device 51 based on commands from the plant harvest management server 1, the automatic harvesting device 5 moves to the vicinity of the strawberries S by the traveling device 52, and harvests the fruits SF of the strawberries S using the manipulator 53. In this embodiment, the automatic harvesting device 5 is an example of a "harvesting device."

[0027] In this embodiment, it is assumed that a plurality of plant cultivation shelves PL are provided inside the agricultural greenhouse 8. In this embodiment, it is assumed that a plurality of strawberries S are cultivated on each of the plurality of plant cultivation shelves PL. Specifically, as illustrated in Fig. 3, this embodiment assumes a case in which four plant cultivation shelves PL(1) to PL(4) are provided inside an agricultural greenhouse 8. In this embodiment, as an example, it is assumed that each plant cultivation shelf PL is provided to extend in the X-axis direction, and the four plant cultivation shelves PL(1) to PL(4) are provided to be aligned in the Y-axis direction.

[0028] In this embodiment, the automatic harvesting device 5 moves on a path RT provided inside the agricultural greenhouse 8. Here, the path RT is a path preset to pass near each of a plurality of plant cultivation shelves PL, starting from a start position PS and ending at an end position PE. In this embodiment, it is assumed that the path RT includes a path provided along the plant cultivation shelf PL near the Y1 side of each plant cultivation shelf PL and a path provided along the plant cultivation shelf PL near the Y2 side of each plant cultivation shelf PL.

[0029] In this embodiment, the automatic harvesting device 5 stops at predetermined intervals on the path RT, and at the stopped position, harvests the fruit SF of the strawberry S using the manipulator 53. Specifically, in this embodiment, as illustrated in FIG. 3, when the automatic harvesting device 5 moves near the Y1 side of the plant cultivation shelf PL in the path RT, it stops at predetermined intervals, and at the stopped position, extends the manipulator 53 in the Y2 direction to harvest the fruit SF of the strawberry S cultivated on the plant cultivation shelf PL. Also, in this embodiment, as illustrated in FIG. 3, when the automatic harvesting device 5 moves near the Y2 side of the plant cultivation shelf PL in the path RT, it stops at predetermined intervals, and at the stopped position, extends the manipulator 53 in the Y1 direction to harvest the fruit SF of the strawberry S cultivated on the plant cultivation shelf PL.

[0030] <A.1.3. Outline of Automatic Harvesting Device> Hereinafter, with reference to FIG. 4, the outline of the automatic harvesting device 5 will be described.

[0031] FIG. 4 is a functional block diagram showing an example of the outline of the configuration of the automatic harvesting device 5.

[0032] As illustrated in FIG. 4, the automatic harvesting device 5 includes a control device 51, a traveling device 52, a manipulator 53, an imaging device 54, a detection device 55, and a communication device 56, as described above.

[0033] The control device 51 is configured to include, for example, a processor, and controls each part of the automatic harvesting device 5. The processor provided in the control device 51 is configured to include, for example, one or more central processing units (CPUs). However, the processor provided in the control device 51 may be configured to include hardware such as a graphics processing unit (GPU), a digital signal processor (DSP), or a field programmable gate array (FPGA) in addition to the one or more CPUs, or in place of some or all of the one or more CPUs.

[0034] The traveling device 52 includes a moving mechanism including a plurality of tires or legs, and a drive mechanism such as a motor for driving the moving mechanism. The traveling device 52 changes the position and posture of the automatic harvesting device 5 by driving the moving mechanism with the drive mechanism under the control of the control device 51.

[0035] The manipulator 53 includes a multi-joint arm robot and an end effector attached to the tip of the arm robot. Under the control of the control device 51, the manipulator 53 operates the arm robot and the end effector to cut the fruit SF of the strawberry S from the stem SK and harvest the fruit SF.

[0036] The imaging device 54 captures an image of an area including strawberries S grown on the plant cultivation shelf PL under the control of the control device 51. Specifically, when the automatic harvesting device 5 is stopped on the route RT, the imaging device 54 captures an image of an area on the plant cultivation shelf PL where strawberries S having fruits SF that the automatic harvesting device 5 plans to harvest using the manipulator 53 are grown. When the imaging device 54 captures an image of the area on the plant cultivation shelf PL where the strawberries S are grown, it generates imaging information DG indicating a captured image GG obtained as a result of the imaging.

[0037] In addition, in the present embodiment, the area on the plant cultivation shelf PL where the strawberry S is cultivated is an example of the "cultivation area". Further, in the present embodiment, the area on the plant cultivation shelf PL where the strawberry S is cultivated includes, in addition to the area located in the Z1 direction when viewed from the plant cultivation shelf PL, the area where the strawberry S being cultivated on the plant cultivation shelf PL can exist.

[0038] The detection device 55 includes an obstacle detection sensor such as an infrared sensor and an ultrasonic sensor. When the obstacle sensor detects the presence of an obstacle in the moving direction of the automatic harvesting device 5, the control device 51 stops the movement of the automatic harvesting device 5. Further, the detection device 55 includes a position detection sensor for detecting the position of the automatic harvesting device 5, such as a GPS (Global Positioning System) receiver, an acceleration sensor, and an angular velocity sensor. The control device 51 moves on the path RT based on the detection result of the position detection sensor.

[0039] Note that, in the present embodiment, it is assumed that the detection device 55 includes an obstacle detection sensor and a position detection sensor, but the present invention is not limited to such a mode. The detection device 55 may include only one of the obstacle detection sensor and the position detection sensor, or may include a sensor other than the obstacle detection sensor and the position detection sensor. Further, in the present embodiment, it is assumed that the automatic harvesting device 5 includes the detection device 55, but the present invention is not limited to such a mode. The automatic harvesting device 5 may be configured without including the detection device 55.

[0040] The communication device 56 is hardware for communicating with external devices such as the plant harvest management server 1 and the administrator terminal device 9 existing outside the automatic harvesting device 5 via, for example, the network NW.

[0041] <A.1.4. Overview of the Administrator Terminal Device> Hereinafter, the overview of the plant harvest management server 1 will be described with reference to FIG. 5.

[0042] FIG. 5 is a functional block diagram showing an example of an outline of the configuration of the plant harvest managing server 1. As shown in FIG.

[0043] As illustrated in FIG. 5, the plant harvest management server 1 includes a control device 2, a storage device 3, and a communication device 4.

[0044] The communication device 4 is, for example, hardware for communicating with an external device, such as an automatic harvesting device 5, that exists outside the plant harvest managing server 1, via the network NW.

[0045] The storage device 3 is configured to include, for example, one or both of a volatile memory such as a RAM (Random Access Memory) that functions as a working area for the control device 2, and a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory) that stores various information.

[0046] 5, the storage device 3 stores a detected object information DB, detected fruit information DF, and a control program PG-S for the plant harvest management server 1. The detected object information DB and the detected fruit information DF will be described later.

[0047] The control device 2 is configured to include, for example, a processor, and controls each part of the plant harvest management server 1. The processor provided in the control device 2 is configured to include, for example, one or more CPUs. However, the processor provided in the control device 2 may be configured to include hardware such as a GPU, DSP, or FPGA in addition to the one or more CPUs, or in place of some or all of the one or more CPUs.

[0048] The processor provided in the control device 2 executes the control program PG-S of the plant harvest management server 1 stored in the memory device 3, and operates in accordance with the control program PG-S, thereby operating as a harvesting device control unit 21, an object recognition unit 22, and a harvest determination unit 23, and controlling each part of the plant harvest management server 1.

[0049] As illustrated in FIG. 5, the harvesting device control unit 21 includes a traveling device control unit 211, a manipulator control unit 212, and an imaging device control unit 213.

[0050] The traveling device control unit 211 controls the traveling device 52 provided in the automatic harvesting device 5. Specifically, the traveling device control unit 211 controls the traveling device 52 via the control device 51 by supplying a control command to the control device 51 to control the traveling device 52 so that the automatic harvesting device 5 moves, stops, or the like.

[0051] The manipulator control unit 212 controls the manipulator 53 provided in the automatic harvesting device 5. Specifically, the manipulator control unit 212 controls the manipulator 53 via the control device 51 by supplying a control command to the control device 51 to control the manipulator 53 so that the manipulator 53 harvests the fruits SF of the strawberries S.

[0052] In this embodiment, the manipulator control unit 212 that controls the manipulator 53 is an example of the "device control unit."

[0053] The imaging device control unit 213 controls the imaging device 54 provided in the automatic harvesting device 5. Specifically, the imaging device control unit 213 controls the imaging device 54 via the control device 51 by supplying a control command to the control device 51 to control the imaging device 54 so that the imaging device 54 captures an image of the cultivation area. In addition, the imaging device control unit 213 controls the imaging device 54 via the control device 51 by supplying a control command to the control device 51 to control the imaging device 54 so that the imaging device 54 generates imaging information DG indicating an image GG obtained as a result of the imaging device 54 capturing an image of the cultivation area.

[0054] Then, the imaging device control unit 213 acquires the imaging information DG generated by the imaging device 54. Specifically, the imaging device control unit 213 controls the imaging device 54 via the control device 51 by supplying a control command to the control device 51 to control the imaging device 54 so as to transmit the imaging information DG generated by the imaging device 54 to the plant harvest management server 1 via the communication device 56.

[0055] In this embodiment, the imaging device control section 213 that acquires the imaging information DG indicating the captured image GG is an example of an "acquisition section."

[0056] In this embodiment, as described above, the harvesting device control unit 21 controls the traveling device 52, the manipulator 53, and the imaging device 54 via the control device 51, but the present invention is not limited to this embodiment. The harvesting device control unit 21 may directly control some or all of the traveling device 52, the manipulator 53, and the imaging device 54 without using the control device 51.

[0057] The object recognition unit 22 recognizes the type of each of one or more objects Obj included in the captured image GG indicated by the imaging information DG acquired by the imaging device control unit 213.

[0058] Specifically, the object recognition unit 22 identifies the type of the object Obj included in the captured image GG by using a learning model (not shown) that has learned the relationship between some or all of the shape, color, and size of the object Obj included in the captured image GG and the type of the object Obj. Note that in this embodiment, it is assumed that the learning model is stored in the storage device 3. More specifically, in this embodiment, as an example, it is assumed that the object recognition unit 22 classifies each object Obj included in the captured image GG into either the fruit SF of the strawberry S, the leaves SL of the strawberry S, the stems SK of the strawberry S, equipment OSB such as a plant cultivation shelf PL, or an unidentifiable object that the object recognition unit 22 cannot identify.

[0059] In this embodiment, it is assumed that the object recognition unit 22 identifies the type of the object Obj by using a learning model, but the present invention is not limited to this aspect. The object recognition unit 22 may recognize the type of the object Obj by analyzing a part or all of the shape, color, and size of the object Obj.

[0060] In this embodiment, the object recognition section 22 that recognizes the object Obj included in the captured image GG indicated by the imaging information DG is an example of a "recognition section."

[0061] 5, the harvest decision unit 23 includes a fruit state determination unit 231, a stalk state determination unit 232, and an obstacle analysis unit 233. In this embodiment, the harvest decision unit 23 is an example of a "decision unit."

[0062] The fruit status determination unit 231 executes a fruit status determination process. Here, the fruit status determination process is a process for determining whether a fruit SF included in a captured image GG indicated by the imaging information DG is a ripe fruit SFA or an unripe fruit SFB.

[0063] In this embodiment, as described above, the strawberry S fruit SF may be referred to as a "harvest target object OT." In addition, hereinafter, among the objects Obj, objects Obj other than the harvest target object OT may be referred to as a "non-target object ON."

[0064] The stem status determination unit 232 executes a stem status determination process. Here, the stem status determination process is a process for identifying a stem SK corresponding to a fruit SF in the captured image GG. In other words, the stem status determination process is a process for identifying a stem SK that has a fruit SF at its tip in the captured image GG as the stem SK that corresponds to the fruit SF. In this embodiment, the automatic harvesting device 5 harvests the fruits SF by cutting the stems SK corresponding to the fruits SF.

[0065] The obstacle analysis unit 233 executes an obstacle analysis process. Here, the obstacle analysis process is a process of determining whether or not an obstacle OB exists among the objects Obj included in the captured image GG indicated by the imaging information DG, and analyzing the type of the obstacle OB if the obstacle OB exists.

[0066] Here, an obstacle OB refers to one or more objects Obj included in the captured image GG that hinders the automatic harvesting device 5 from harvesting the strawberries S, which are the harvest target objects OT, when the fruits SF are harvested by the automatic harvesting device 5.

[0067] Specifically, the obstacle OB is an object Obj that exists in the vicinity of the harvest target object in the captured image GG, among one or more objects Obj included in the captured image GG. Here, the vicinity of the harvest target object is an area through which the manipulator 53 may pass when the manipulator 53 approaches the fruit SF to harvest the fruit SF. In this embodiment, as an example, it is assumed that the harvest target vicinity region is a region consisting of a region in front of the fruit vicinity region and a region in front of the stem vicinity region in the captured image GG. Among these, the fruit vicinity region is a region in the vicinity of the fruit SF which is the harvest target OT. Also, the stem vicinity region is a region in the vicinity of the stem SK which corresponds to the fruit SF which is the harvest target OT. In the following description, an object Obj that hinders the harvesting of the fruit SF may be referred to as an obstacle OB corresponding to the fruit SF.

[0068] In this embodiment, even if an obstacle OB corresponding to a fruit SF is present, it is assumed that there are both cases in which it is impossible for the automatic harvesting device 5 to harvest the fruit SF, and cases in which it is possible for the automatic harvesting device 5 to harvest the fruit SF. In other words, in this embodiment, it is assumed that even if an obstacle OB corresponding to a fruit SF is present, there are cases in which it is possible for the automatic harvesting device 5 to harvest the fruit SF.

[0069] In the following, among the obstacles OB, an obstacle OB that makes it impossible to harvest the fruit SF corresponding to the obstacle OB due to the presence of the obstacle OB is referred to as a harvest-impeding obstacle OB1. Also, in the following, among the obstacles OB, an obstacle OB that makes it possible to harvest the fruit SF corresponding to the obstacle OB even when the obstacle OB exists is referred to as a harvest-permitting obstacle OB2.

[0070] In this embodiment, the plant production and harvesting system Sys does not harvest the fruit SF if the obstacle OB corresponding to the fruit SF is a harvest-impeding obstacle OB1. Hereinafter, when a harvest-impeding obstacle OB1 exists corresponding to the fruit SF and the fruit SF is not harvested, the fruit SF is referred to as a non-harvestable object OT1.

[0071] Furthermore, in this embodiment, the plant production and harvesting system Sys harvests the fruit SF when the obstacle OB corresponding to the fruit SF is a harvesting-permitted obstacle OB2. In the following, when a harvesting-permitted obstacle OB2 exists corresponding to the fruit SF, but the fruit SF can be harvested, the fruit SF will be referred to as a harvestable object OT2. In the following, even when no obstacle OB exists corresponding to the fruit SF, and the fruit SF can be harvested, the fruit SF will be referred to as a harvestable object OT2.

[0072] The obstacle analysis unit 233 judges whether or not the object Obj is an obstacle OB based on the position of the object Obj in the captured image GG and the position of the fruit SF in the captured image GG. Further, the obstacle analysis unit 233 judges whether the obstacle OB is a harvesting-impairing obstacle OB1 or a harvesting-permitting obstacle OB2 based on the characteristics of the obstacle OB. That is, the obstacle analysis unit 233 judges whether the fruit SF is an unharvestable object OT1 or a harvestable object OT2 based on the characteristics of the obstacle OB corresponding to the fruit SF. Here, the characteristics of the obstacle OB may be the type of the obstacle OB. Furthermore, the characteristics of the obstacle OB may be whether the fruit SF can be harvested in the future when the obstacle OB is a fruit SF.

[0073] <A.1.5. Overview of the Administrator Terminal Device> Hereinafter, with reference to FIG. 6, the overview of the administrator terminal device 9 will be described.

[0074] FIG. 6 is a functional block diagram showing an example of the outline of the configuration of the administrator terminal device 9.

[0075] As illustrated in FIG. 6, the administrator terminal device 9 includes a control device 91, a display device 92, an operation device 93, a storage device 94, and a communication device 95.

[0076] The control device 91 is configured to include, for example, a processor and controls each part of the administrator terminal device 9. The processor provided in the control device 91 is configured to include, for example, one or more CPUs. However, the processor provided in the control device 91 may be configured to include hardware such as a GPU, a DSP, or an FPGA in addition to or instead of some or all of the one or more CPUs.

[0077] The display device 92 displays various information related to the agricultural house 8. The operation device 93 receives various operations by the user of the administrator terminal device 9. The storage device 94 stores various information such as the control program of the administrator terminal device 9. The communication device 95 is hardware for communicating with external devices such as the plant harvesting management server 1 and the automatic harvesting device 5 existing outside the administrator terminal device 9 via, for example, the network NW.

[0078] When the storage device 94 receives an operation indicating the start of the harvesting process by the user of the administrator terminal device 9, the control device 91 generates a harvesting start signal and controls the communication device 95 so that the harvesting start signal is transmitted to the plant harvesting management server 1.

[0079] <A.2. Overview of Various Data in the Plant Production and Harvesting System> Hereinafter, an overview of various data stored in the plant production and harvesting system Sys will be described with reference to FIGS.

[0080] FIG. 7 is a diagram showing an example of the data configuration of the detected object information DB.

[0081] Here, the detected object information DB is information generated by the object recognition unit 22 when the imaging device control unit 213 acquires the imaging information DG, and is information regarding one or more objects Obj included in the captured image GG indicated by the imaging information DG acquired by the imaging device control unit 213. Note that if an object Obj is not included in the captured image GG indicated by the imaging information DG acquired by the imaging device control unit 213, the detected object information DB is not generated. Furthermore, the detected object information DB generated when the imaging device control unit 213 acquires the imaging information DG is discarded when the imaging device control unit 213 acquires the imaging information DG next time. That is, in this embodiment, the detected object information DB is generated every time the imaging device control unit 213 acquires the imaging information DG.

[0082] 7, the detected object information DB has a number of records that correspond one-to-one to one or more objects Obj included in the captured image GG. Each record in the detected object information DB has an object ID, object region information DBR, and object type information DBS.

[0083] The object ID is information for identifying each object Obj from among one or more objects Obj included in the captured image GG.

[0084] The object region information DBR is information indicating the region in the captured image GG where the object Obj exists. In this embodiment, the object region information DBR includes object position information DBR1 and object size information DBR2. Here, the object position information DBR1 is information indicating the position in the captured image GG where the object Obj exists. Moreover, the object size information DBR2 is information indicating the size of the object Obj in the captured image GG. However, the present invention is not limited to such an embodiment. The object region information DBR may be, for example, information indicating a set of pixels in the captured image GG where the object Obj exists.

[0085] FIG. 8 is a diagram showing an example of the data configuration of the detected fruit information DF.

[0086] Here, the detected fruit information DF is information generated when the imaging device control unit 213 acquires the imaging information DG, and is information regarding one or more fruits SF among one or more objects Obj contained in the captured image GG indicated by the imaging information DG acquired by the imaging device control unit 213. Note that no detected fruit information DF is generated if a fruit SF is not included in one or more objects Obj included in the captured image GG indicated by the imaging information DG acquired by the imaging device control unit 213. Furthermore, the detected fruit information DF generated when the imaging device control unit 213 acquires the imaging information DG is discarded when the imaging device control unit 213 next acquires the imaging information DG. That is, in this embodiment, the detected fruit information DF is generated every time the imaging device control unit 213 acquires the imaging information DG.

[0087] 8, the detected fruit information DF has a plurality of records corresponding one-to-one to one or more fruits SF included in the captured image GG. Each record of the detected fruit information DF has an object ID, a fruit ID, fruit ripeness information DFS, fruit stalk information DFK, surrounding obstacle information DFB, and harvest readiness information DFF.

[0088] The fruit ID is information for identifying each fruit SF from one or more fruits SF included in the captured image GG. In the following, it is assumed that the number of fruits SF included in the captured image GG is "M pieces." Here, the value M is a natural number equal to or greater than 1. In the following, the mth fruit SF among the fruits SF included in the captured image GG is referred to as fruit SF[m]. In this embodiment, for convenience of explanation, the detected fruit information DF includes a fruit ID, but the present invention is not limited to this embodiment. The detected fruit information DF may be configured without including a fruit ID.

[0089] The fruit maturity information DFS is information indicating whether the fruit SF is ripe or not. Specifically, the fruit maturity information DFS is information indicating whether the fruit SF is a ripe fruit SFA or an unripe fruit SFB. More specifically, the fruit maturity information DFS indicates "1" when the fruit SF is a ripe fruit SFA, and indicates "0" when the fruit SF is an unripe fruit SFB. In the following, the fruit ripeness information DFS corresponding to the fruit SF[m] may be referred to as fruit ripeness information DFS[m].

[0090] The fruit status determination unit 231 determines whether or not the fruit SF[m] included in the captured image GG is ripe, using a learning model (not shown) that has learned the relationship between some or all of the shape, color, and size of the fruit SF included in the captured image GG and whether or not the fruit SF is ripe. Then, the fruit status determination unit 231 generates fruit ripeness information DFS[m] based on the determination result of the ripeness of the fruit SF[m]. Note that in this embodiment, it is assumed that the learning model is stored in the storage device 3.

[0091] The fruit stalk information DFK is information indicating a stem SK corresponding to a fruit SF. Specifically, if a stem SK corresponding to a fruit SF exists, the fruit stalk information DFK indicates the object ID of the stem SK, and if no stem SK corresponding to a fruit SF exists, the fruit stalk information DFK is set to "Null." In the following, the fruit stalk information DFK corresponding to the fruit SF[m] may be referred to as fruit stalk information DFK[m].

[0092] The stem status determination unit 232 identifies the stem SK corresponding to the fruit SF[m] included in the captured image GG using a learning model (not shown) that has learned the relationship between the shape of the fruit SF included in the captured image GG and the shape of the stem SK included in the captured image GG, and the relationship between whether the fruit SF and the stem SK correspond to each other. Then, the stem status determination unit 232 generates fruit and stalk information DFK[m] based on the identification result of the stem SK corresponding to the fruit SF[m]. Note that in this embodiment, it is assumed that the learning model is stored in the storage device 3.

[0093] The surrounding obstacle information DFB is information indicating an obstacle OB corresponding to a fruit SF. Specifically, when an obstacle OB corresponding to a fruit SF exists, the surrounding obstacle information DFB indicates the object ID of the obstacle OB, and when no obstacle OB corresponding to a fruit SF exists, the surrounding obstacle information DFB is set to "Null." In the following description, the surrounding obstacle information DFB corresponding to the fruit SF[m] may be referred to as surrounding obstacle information DFB[m].

[0094] The obstacle analysis unit 233 identifies an obstacle OB corresponding to a fruit SF[m] included in the captured image GG using a learning model (not shown) that has learned the relationship between the position, shape, and size of the fruit SF included in the captured image GG, the position, shape, and size of the object Obj included in the captured image GG, and whether the object Obj is an obstacle OB corresponding to the fruit SF. Then, the obstacle analysis unit 233 generates surrounding obstacle information DFB[m] based on the identification result of the obstacle OB corresponding to the fruit SF[m]. Note that in this embodiment, it is assumed that the learning model is stored in the storage device 3.

[0095] The harvestability information DFF is information indicating whether the fruit SF is harvestable. Specifically, the harvestability information DFF indicates "1" when the fruit SF is a harvestable object OT2 that can be harvested, and indicates "0" when the fruit SF is a non-harvestable object OT1 that cannot be harvested. Note that hereinafter, the information indicating whether the fruit SF[m] is harvestable may be referred to as the harvestability information DFF[m].

[0096] <A.3. Operation of the Plant Harvest Management Server> Hereinafter, with reference to FIGS. 9 to 15, an overview of the operation of the plant harvest management server 1 will be described.

[0097] <A.3.1. Harvesting Process> FIG. 9 is a diagram showing an example of an overview of the operation of the plant harvest management server 1 when the harvesting process is executed.

[0098] Here, the harvesting process is a process of controlling the automatic harvesting device 5 to move from the start position PS to the end position PE along the route RT, and periodically stopping the automatic harvesting device 5 on the route RT, and harvesting the fruit SF of the strawberry S on the plant cultivation shelf PL with respect to the automatic harvesting device 5 stopped on the route RT. The harvesting process is started when the plant harvest management server 1 is supplied with a harvest start signal from the administrator terminal device 9.

[0099] As illustrated in FIG. 9, when the harvesting process is started, the traveling device control unit 211 controls the traveling device 52 so that the automatic harvesting device 5 moves along the route RT (S101).

[0100] Next, the traveling device control unit 211 determines whether the automatic harvesting device 5 has reached the end position PE of the route RT (S103).

[0101] If the result of the determination in step S103 is affirmative, the control device 2 ends the harvesting process. If the result of the determination in step S103 is negative, the imaging device control unit 213 controls the imaging device 54 to image the cultivation area (the area on the plant cultivation shelf PL where strawberry S is cultivated), thereby obtaining imaging information DG indicating the captured image GG from the imaging device 54 (S105).

[0102] Next, the control device 2 executes a harvestability determination process (S107).

[0103] Here, the harvestability determination process is a process of identifying the fruits SF (harvestable target objects OT2) of the strawberries S that can be harvested among one or more objects Obj included in the captured image GG acquired by the imaging device control unit 213 in step S105. Note that the harvestability determination process will be described later with reference to FIG. 10.

[0104] Next, based on the identification result in step S107, the control device 2 determines whether there is a fruit SF (harvestable target object OT2) that can be harvested and is a pre-harvest fruit SF among one or more objects Obj included in the captured image GG acquired by the imaging device control unit 213 in step S105 (S109).

[0105] If the result of the determination in step S109 is negative, that is, if the captured image GG does not include the fruit SF, and if there is no fruit SF that can be harvested and is a pre-harvest fruit SF among the one or more fruit SFs included in the captured image GG, the control device 2 advances the process to step S101 to move the automatic harvesting device 5 along the path RT.

[0106] If the result of the determination in step S109 is positive, the manipulator control unit 212 controls the manipulator 53 to harvest the unharvested fruit SF[m] among the pre-harvest harvestable fruit SFs (harvestable target objects OT2) included in the captured image GG (S111), and advances the process to step S109.

[0107] <A.3.2. Harvestability Determination Process> FIG. 10 is a diagram showing an example of an outline of the operation of the plant harvest managing server 1 when the harvest feasibility determination process in step S107 is executed.

[0108] As illustrated in Figure 10, when the harvest feasibility determination process is started, the object recognition unit 22 recognizes the type of one or more objects Obj contained in the captured image GG indicated by the imaging information DG acquired in step S105, and generates a detected object information DB based on the recognition result (S201).

[0109] Next, the object recognition unit 22 extracts a fruit SF from one or more objects Obj indicated by the detected object information DB, thereby generating information consisting of an object ID and a fruit ID from the detected fruit information DF (S203). Note that in the detected fruit information DF at the stage of step S203, "Null" is set for the fruit maturity information DFS, fruit stalk information DFK, surrounding obstacle information DFB, and harvest feasibility information DFF.

[0110] Next, the object recognition unit 22 determines whether or not one or more fruits SF are present in the captured image GG indicated by the imaging information DG acquired in step S105 (S205). Specifically, in step S205, the object recognition unit 22 determines whether or not the detected fruit information DF generated in step S203 has one or more records. However, the present invention is not limited to this aspect. In step S205, the object recognition unit 22 may determine whether or not a fruit SF is included in the detected object information DB generated in step S201. In this case, the process of step S205 may be performed before the process of step S203.

[0111] If the result of the determination in step S205 is negative, that is, if the captured image GG does not include the fruit SF, the object recognition unit 22 ends the harvest propriety determination process.

[0112] If the result of the determination in step S205 is affirmative, the fruit condition determination unit 231 executes fruit condition determination processing (S207). In the following, it is assumed that when the result of the determination in step S205 is affirmative, the captured image GG includes M fruits SF[1] to SF[M]. Next, the stem condition determination unit 232 executes stem condition determination processing (S209). In the present embodiment, the processing in step S209 may be performed before the processing in step S207.

[0113] Next, the obstacle analysis unit 233 executes obstacle analysis processing (S211) and ends the harvestability determination processing.

[0114] <A.3.3. Fruit Condition Determination Processing> FIG. 11 is a diagram showing an example of an outline of the operation of the plant harvest management server 1 when the fruit condition determination processing in step S207 is executed.

[0115] As illustrated in FIG. 11, when the fruit condition determination processing is started, the fruit condition determination unit 231 sets "1" to the variable m (S301).

[0116] Next, the fruit condition determination unit 231 estimates the maturity of the fruit SF[m] included in the captured image GG using a learning model that has learned the relationship between part or all of the shape, color, and size of the fruit SF in the captured image GG and the maturity of the fruit SF, as described above (S303).

[0117] Next, the fruit condition determination unit 231 determines whether the fruit SF[m] is a mature fruit SF, that is, whether the fruit SF[m] is a mature fruit SFA, based on the estimation result in step S303 (S305).

[0118] If the result of the determination in step S305 is affirmative, the fruit condition determination unit 231 sets the value "1" indicating that the fruit SF[m] is a mature fruit SFA to the fruit maturity information DFS[m] (S307) and advances the processing to step S313.

[0119] If the result of the determination in step S305 is negative, the fruit condition determination unit 231 sets the value "0" indicating that the fruit SF[m] is an immature fruit SFB in the fruit maturity information DFS[m] (S309). Next, the fruit condition determination unit 231 sets the value "0" indicating that the fruit SF[m] is a non-harvestable object OT1 in the harvestability information DFF[m] (S311), and advances the process to step S313.

[0120] Thereafter, the fruit condition determination unit 231 determines whether there is any other fruit SF that has not been the processing target in the fruit condition determination process among the M fruits SF[1] to SF[M] included in the captured image GG (S313).

[0121] If the result of the determination in step S313 is affirmative, the fruit condition determination unit 231 adds "1" to the variable m (S315), and advances the process to step S303. If the result of the determination in step S313 is negative, the fruit condition determination unit 231 ends the fruit condition determination process.

[0122] <A.3.4. Stem Condition Determination Process> FIG. 12 is a diagram showing an example of an outline of the operation of the plant harvest management server 1 when the stem condition determination process of step S209 is executed.

[0123] As illustrated in FIG. 12, when the stem condition determination process is started, the stem condition determination unit 232 sets "1" in the variable m (S401).

[0124] Next, the stem condition determination unit 232 uses the learning model that has learned the relationship between the position and shape of the fruit SF and the position and shape of the stem SK included in the captured image GG, and whether the fruit SF and the stem SK correspond, as described above, to identify the stem SK corresponding to the fruit SF[m] included in the captured image GG (S403). Hereinafter, the stem SK corresponding to the fruit SF[m] may be referred to as the stem SK[m].

[0125] Next, the stem condition determination unit 232 determines whether the stem SK[m] could be specified in step S403, that is, whether the stem SK[m] corresponding to the fruit SF[m] exists (S405).

[0126] If the result of the determination in step S405 is affirmative, the stem condition determination unit 232 causes the object ID of the stem SK[m] corresponding to the fruit SF[m] to be stored in the fruit stem information DFK[m] (S407), and advances the process to step S413.

[0127] If the result of the determination in step S405 is negative, the stem condition determination unit 232 sets "Null", which indicates that the stem SK corresponding to the fruit SF[m] does not exist, in the fruit stem information DFK[m] (S409). Next, the stem condition determination unit 232 sets the value "0", which indicates that the fruit SF[m] is a non-harvestable object OT1, in the harvestability information DFF[m] (S411), and advances the process to step S413.

[0128] Thereafter, the stem condition determination unit 232 determines whether there is any other fruit SF that has not been the processing target in the stem condition determination process among the M fruits SF[1] to SF[M] included in the captured image GG (S413).

[0129] If the result of the determination in step S413 is affirmative, the stem condition determination unit 232 adds "1" to the variable m (S415), and advances the process to step S403. If the result of the determination in step S413 is negative, the stem condition determination unit 232 ends the stem condition determination process.

[0130] <A.3.5. Obstacle Analysis Process> FIGS. 13 to 15 are diagrams showing an example of an outline of the operation of the plant harvest management server 1 when the obstacle analysis process in step S211 is executed.

[0131] As illustrated in FIG. 13, when the obstacle analysis process is started, the obstacle analysis unit 233 sets "1" in the variable m (S501).

[0132] Next, the obstacle analysis unit 233 identifies an obstacle OB corresponding to a fruit SF[m] included in the captured image GG using a learning model that has learned the relationship between the position, shape, and size of the fruit SF included in the captured image GG, the position, shape, and size of the object Obj included in the captured image GG, and whether the object Obj is an obstacle OB corresponding to the fruit SF (S503). Note that, hereinafter, an obstacle OB corresponding to a fruit SF[m] may be referred to as an obstacle OB[m]. In this embodiment, a case is assumed in which one or more obstacles OB[m] may exist corresponding to a fruit SF[m].

[0133] Next, the obstacle analysis unit 233 determines whether or not an obstacle OB[m] has been identified in step S503, that is, whether or not an obstacle OB[m] corresponding to the fruit SF[m] exists (S505).

[0134] If the result of the determination in step S505 is positive, the obstacle analysis unit 233 sets the object ID of the obstacle OB[m] corresponding to the fruit SF[m] in the surrounding obstacle information DFB[m] (S507), and proceeds to the process of step S511.

[0135] If the result of the judgment in step S505 is negative, the obstacle analysis unit 233 sets the surrounding obstacle information DFB[m] to “Null” indicating that there is no obstacle OB[m] corresponding to the fruit SF[m] (S509), and proceeds to step S543.

[0136] Next, the obstacle analysis unit 233 judges whether or not the obstacle OB[m] is a leaf SL of the strawberry S (S511).

[0137] If the result of the determination in step S511 is positive, the obstacle analysis unit 233 advances the process to step S541. If the result of the determination in step S511 is negative, the obstacle analysis unit 233 advances the process to step S521.

[0138] As illustrated in FIG. 14, the obstacle analysis unit 233 determines whether or not the obstacle OB[m] is a stem SK of a strawberry S (S521).

[0139] When the result of the determination in step S521 is positive, the obstacle analysis unit 233 determines whether or not another fruit SF is present at the tip of the stem SK, which is the obstacle OB[m] (S523).

[0140] If the result of the determination in step S523 is negative, the obstacle analysis unit 233 advances the process to step S541.

[0141] If the result of the determination in step S523 is positive, the obstacle analysis unit 233 estimates whether or not another fruit SF present at the tip of the stem SK, which is the obstacle OB[m], can be harvested in the future (S525). Specifically, in step S525, the obstacle analysis unit 233 estimates whether or not another fruit SF at the tip of the stem SK, which is the obstacle OB, can be harvested in the future, even if it is an immature fruit SFB, by using a learning model (not shown) that has learned the relationship between some or all of the shape, color, and size of the fruit SF included in the captured image GG and whether or not the fruit SF will grow into a mature fruit SFA that can be harvested in the future. Note that in this embodiment, it is assumed that the learning model is stored in the storage device 3.

[0142] Next, the obstacle analysis unit 233 determines whether or not another fruit SF present at the tip of the stem SK, which is the obstacle OB, can be harvested in the future, based on the estimation result in step S525 (S527).

[0143] If the result of the determination in step S527 is positive, the obstacle analysis unit 233 advances the process to step S547. If the result of the determination in step S527 is negative, the obstacle analysis unit 233 advances the process to step S541.

[0144] As illustrated in FIG. 14, if the result of the determination in step S521 is negative, the obstacle analysis unit 233 determines whether or not the obstacle OB[m] is a fruit SF different from the fruit SF[m] (S529).

[0145] If the result of the determination in step S529 is negative, that is, if the obstacle OB[m] is a facility OSB or an unidentifiable object, the obstacle analysis unit 233 advances the process to step S547. If the result of the determination in step S529 is positive, the obstacle analysis unit 233 determines whether or not a stem SK corresponding to another fruit SF, which is an obstacle OB[m], has been detected (S531). Note that, although the process of step S531 is performed in this embodiment, the process of step S531 may be omitted.

[0146] If the result of the determination in step S531 is negative, the obstacle analysis unit 233 advances the process to step S547.

[0147] If the result of the determination in step S531 is positive, the obstacle analysis unit 233 estimates whether or not the other fruit SF, which is the obstacle OB[m], can be harvested in the future (S533). Specifically, in step S533, the obstacle analysis unit 233 uses a learning model (not shown) that has learned the relationship between some or all of the shape, color, and size of the fruit SF contained in the captured image GG and whether the fruit SF will grow into a mature fruit SFA that can be harvested in the future, to estimate whether other fruits SF that are obstacles OB can be harvested in the future even if they are immature fruits SFB.

[0148] Next, the obstacle analysis unit 233 determines whether or not the other fruit SF that is the obstacle OB can be harvested in the future based on the estimation result in step S533 (S535).

[0149] If the result of the determination in step S535 is positive, the obstacle analysis unit 233 advances the process to step S547. If the result of the determination in step S535 is negative, the obstacle analysis unit 233 advances the process to step S541.

[0150] As illustrated in FIG. 15, if the result of the judgment in step S511 is positive, if the result of the judgment in step S523 is negative, if the result of the judgment in step S527 is negative, or if the result of the judgment in step S535 is negative, the obstacle analysis unit 233 determines whether or not there is another obstacle OB[m] corresponding to the fruit SF[m] (S541).

[0151] If the result of the determination in step S541 is positive, the obstacle analysis unit 233 advances the process to step S511, thereby performing the processes of steps S511 to S535 for the other obstacles OB[m].

[0152] If the result of the judgment in step S541 is negative, the obstacle analysis unit 233 judges whether the value "0" indicating that the fruit SF[m] is an unharvestable object OT1 is set in the harvest feasibility information DFF[m], for example, whether "Null" is set in the harvest feasibility information DFF[m] (S543).

[0153] If the result of the determination in step S543 is negative, the obstacle analysis unit 233 advances the process to step S549. If the result of the judgment in step S543 is positive, the obstacle analysis unit 233 sets the harvest possibility information DFF[m] to the value "1" indicating that the fruit SF[m] is a harvestable object OT2 (S545), and proceeds to the processing of step S549.

[0154] In addition, if the result of the judgment in step S527 is positive, if the result of the judgment in step S529 is negative, if the result of the judgment in step S531 is negative, or if the result of the judgment in step S535 is positive, the obstacle analysis unit 233 sets the harvest feasibility information DFF[m] to the value "0" indicating that the fruit SF[m] is an unharvestable object OT1 (S547).

[0155] Thereafter, the obstacle analysis unit 233 determines whether there are other fruits SF that are not the processing targets in the obstacle analysis process among the M fruits SF[1] to SF[M] included in the captured image GG (S549). Then, when the result of the determination in step S549 is affirmative, the obstacle analysis unit 233 adds "1" to the variable m (S551), and advances the process to step S503. On the other hand, when the result of the determination in step S549 is negative, the obstacle analysis unit 233 ends the stem situation determination process.

[0156] <A.4. Case of Harvestability Judgment Process> Hereinafter, with reference to FIGS. 16 and 19, it will be described what values are set in the detected fruit information DF according to the specific situation of the fruit SF[m].

[0157] FIG. 16 is a diagram showing the captured image GG when the fruit SF[m] is in the first situation. Here, the first situation is a situation where the fruit SF[m] is a mature fruit SFA, the stem SK[m] corresponding to the fruit SF[m] is detected, and there is no obstacle OB corresponding to the fruit SF[m].

[0158] In the first situation, since the fruit SF[m] is a ripe fruit SFA, the fruit situation determination unit 231 sets the value "1" indicating that the fruit SF[m] is a ripe fruit SFA to the fruit ripeness information DFS[m] in step S307. Also, since the stem SK[m] corresponding to the fruit SF[m] is detected in the first situation, the stem situation determination unit 232 sets the object ID of the stem SK[m] corresponding to the fruit SF[m] to the fruit stem information DFK[m] in step S407. Also, since there is no obstacle OB corresponding to the fruit SF[m] in the first situation, the obstacle analysis unit 233 sets "Null" indicating that there is no obstacle OB[m] corresponding to the fruit SF[m] in step S509 to the surrounding obstacle information DFB[m]. Then, in the first situation, in step S543, the harvest feasibility information DFF[m] is set to "Null" (i.e., the harvest feasibility information DFF[m] is not set to "0"), so in step S545, the obstacle analysis unit 233 sets the harvest feasibility information DFF[m] to the value "1" indicating that the fruit SF[m] is a harvestable object OT2. Therefore, in the first situation, in step S111, the manipulator control section 212 controls the manipulator 53 to harvest the fruit SF[m].

[0159] 17 is a diagram showing a captured image GG when the fruit SF[m] is in a second situation, in which the fruit SF[m] is a ripe fruit SFA, the stem SK[m] corresponding to the fruit SF[m] is not detected, and the obstacle OB[m] corresponding to the fruit SF[m] is a leaf SL.

[0160] In the second situation, since the fruit SF[m] is a ripe fruit SFA, the fruit status determination unit 231 sets the value "1" indicating that the fruit SF[m] is a ripe fruit SFA to the fruit ripeness information DFS[m] in step S307. Also, since the stem SK[m] corresponding to the fruit SF[m] is not detected in the second situation, the stem status determination unit 232 sets the fruit stalk information DFK[m] to "Null" indicating that there is no stem SK corresponding to the fruit SF[m] in step S409, and sets the value "0" indicating that the fruit SF[m] is an unharvestable object OT1 to the harvest feasibility information DFF[m] in step S411. Also, since the obstacle OB[m] corresponding to the fruit SF[m] is a leaf SL in the second situation, the obstacle analysis unit 233 sets the object ID of the leaf SL, which is the obstacle OB[m] corresponding to the fruit SF[m], to the surrounding obstacle information DFB[m] in step S507. Then, in the second situation, in step S543, since the value "0" indicating that the fruit SF[m] is an unharvestable object OT1 has already been set for the harvest feasibility information DFF[m], the obstacle analysis unit 233 terminates the obstacle analysis process with the value indicated by the harvest feasibility information DFF[m] remaining at "0." Therefore, in the second situation, in step S111, the manipulator control section 212 controls the manipulator 53 so as not to harvest the fruit SF[m].

[0161] 18 is a diagram showing a captured image GG when a fruit SF[m] is in a third situation, where the fruit SF[m] is a ripe fruit SFA, a stem SK[m] corresponding to the fruit SF[m] is detected, and an obstacle OB[m] corresponding to the fruit SF[m] is a leaf SL.

[0162] In the third situation, since the fruit SF[m] is a ripe fruit SFA, the fruit status determination unit 231 sets the value "1" indicating that the fruit SF[m] is a ripe fruit SFA to the fruit ripeness information DFS[m] in step S307. Also, since the stem SK[m] corresponding to the fruit SF[m] is detected in the third situation, the stem status determination unit 232 sets the object ID of the stem SK[m] corresponding to the fruit SF[m] to the fruit stalk information DFK[m] in step S407. Also, since the obstacle OB[m] corresponding to the fruit SF[m] is a leaf SL in the third situation, the obstacle analysis unit 233 sets the object ID of the leaf SL, which is the obstacle OB[m] corresponding to the fruit SF[m], to the surrounding obstacle information DFB[m] in step S507. Then, in the third situation, in step S543, the harvest feasibility information DFF[m] is set to "Null" and the harvest feasibility information DFF[m] is not set to "0", so in step S545, the obstacle analysis unit 233 sets the harvest feasibility information DFF[m] to the value "1" indicating that the fruit SF[m] is a harvestable object OT2. Therefore, in the third situation, in step S111, the manipulator control section 212 controls the manipulator 53 to harvest the fruit SF[m].

[0163] 19 is a diagram showing a captured image GG in a case where the fruit SF[m] is in a fourth situation. Here, the fourth situation is a situation where the fruit SF[m] is a ripe fruit SFA, a stem SK[m] corresponding to the fruit SF[m] is detected, and an obstacle OB[m] corresponding to the fruit SF[m] is another fruit SF. In the fourth situation, it is assumed that the other fruit SF, which is the obstacle OB[m] corresponding to the fruit SF[m], is an unripe fruit SFB, but a stem SK corresponding to the other fruit SF is detected, and the other fruit SF will become a ripe fruit SFA in the future and can be harvested in the future.

[0164] In the fourth situation, since the fruit SF[m] is a ripe fruit SFA, the fruit condition determination unit 231 sets the value "1", which indicates that the fruit SF[m] is a ripe fruit SFA, for the fruit ripeness information DFS[m] in step S307. Also, in the fourth situation, since the stem SK[m] corresponding to the fruit SF[m] is detected, the stem condition determination unit 232 sets the object ID of the stem SK[m] corresponding to the fruit SF[m] for the fruit stem information DFK[m] in step S407. Further, in the fourth situation, the obstacle OB[m] corresponding to the fruit SF[m] is another fruit SF (S529: Y), the stem SK corresponding to the other fruit SF is detected (S531: Y), and the other fruit SF has the possibility of future harvest (S535: Y). Therefore, the obstacle analysis unit 233 sets the value "0", which indicates that the fruit SF[m] is a non-harvestable object OT1, for the harvestability information DFF[m] in step S547. Therefore, in the fourth situation, in step S111, the manipulator control unit 212 controls the manipulator 53 so as not to harvest the fruit SF[m].

[0165] <A.5. Conclusion of the Embodiment> As described above, in the present embodiment, even when there is an obstacle OB[m] corresponding to the fruit SF[m], the harvest determination unit 23 determines to harvest the fruit SF[m] based on the characteristics of the obstacle OB[m]. Therefore, according to the present embodiment, when there is an obstacle OB[m] corresponding to the fruit SF[m], the harvest rate of the fruit SF by the automatic harvesting device 5 in the agricultural house 8 can be improved as compared with a mode in which the fruit SF[m] is not uniformly harvested.

[0166] Also, in the present embodiment, when the obstacle OB[m] corresponding to the fruit SF[m] is a fruit SF having a future harvest possibility, the harvest determination unit 23 determines not to harvest the fruit SF[m]. Therefore, according to the present embodiment, even when the obstacle OB[m] corresponding to the fruit SF[m] is a fruit SF having a future harvest possibility, the possibility of damaging the fruit SF that can be harvested in the future can be reduced as compared with the mode of forcibly harvesting the fruit SF[m].

[0167] <B. Modified Example> Each of the above embodiments can be variously modified. Specific modification modes are exemplified below. Two or more modes arbitrarily selected from the following examples can be appropriately combined within a range where they do not conflict with each other. In the modified examples exemplified below, for elements whose actions and functions are equivalent to those of the embodiment, the reference numerals referred to in the above description are used, and the detailed description of each is appropriately omitted.

[0168] <B.1. Modified Example 1> In the above-described embodiment, the case where the harvest determination unit 23 executes the obstacle analysis process based on the flowcharts shown in FIGS. 13 to 15 has been exemplified and described, but the present invention is not limited to such a mode. For example, the harvest determination unit 23 can execute the obstacle analysis process in a plurality of operation modes, and may execute the obstacle analysis process based on a flowchart corresponding to the operation mode.

[0169] In this modified example, it is assumed that the harvest determination unit 23 can operate in two operation modes: a future-priority operation mode (an example of the "first operation mode") and a current-priority operation mode (an example of the "second operation mode").

[0170] Here, the future-priority operation mode is the same operation mode as the above-described embodiment. That is, the future-priority operation mode is an operation mode in which when the obstacle OB[m] corresponding to the fruit SF[m] is another fruit SF, if the other fruit SF is a fruit SF having a future harvest possibility, it is determined not to harvest the fruit SF[m].

[0171] Also, the current priority operation mode is an operation mode in which when the obstacle OB[m] corresponding to the fruit SF[m] is another fruit SF, a determination is made to execute the harvesting of the fruit SF[m] regardless of whether or not the other fruit SF has a future harvesting possibility.

[0172] In this modification, the user of the administrator terminal device 9 supplies a mode designation signal for designating the operation mode of the harvesting determination unit 23 from the administrator terminal device 9 to the plant harvesting management server 1. The harvesting determination unit 23 provided in the plant harvesting management server 1 executes the obstacle analysis process according to the operation mode indicated by the mode designation signal supplied from the administrator terminal device 9.

[0173] Thus, according to this modification, since the obstacle analysis process is executed according to the operation mode instructed by the user of the administrator terminal device 9, the user of the administrator terminal device 9 can control the harvesting rate of the fruit SF by the automatic harvesting device 5 in the agricultural house 8.

[0174] <B.2. Modification 2> In the above-described embodiment and Modification 1, the aspect in which the automatic harvesting device 5 moves from the start position PS to the end position PE in the harvesting process has been exemplified and described, but the present invention is not limited to such an aspect. The plant harvesting management server 1 may control the automatic harvesting device 5 so that the automatic harvesting device 5 moves along the path RT a plurality of times (or reciprocates along the path RT a plurality of times) in the harvesting process.

[0175] Specifically, for example, in the harvesting process, the plant harvesting management server 1 may control the automatic harvesting device 5 so that the automatic harvesting device 5 moves along the path RT a predetermined number of times (for example, 2 times).

[0176] In addition, for example, when the obstacle OB [m] corresponding to the fruit SF [m] is another fruit SF, the plant harvesting management server 1 may control the automatic harvesting device 5 so that the automatic harvesting device 5 moves along the route RT again. More specifically, for example, when the obstacle OB [m] corresponding to the fruit SF [m] is another fruit SF, and the other fruit SF is a ripe fruit SFA, and a value "1" indicating that it is a harvestable object OT2 is set in the harvestability information DFF corresponding to the other fruit SF, and a value "0" indicating that it is a non-harvestable object OT1 is set in the harvestability information DFF [m] corresponding to the fruit SF [m], the plant harvesting management server 1 may control the automatic harvesting device 5 so that the automatic harvesting device 5 moves along the route RT again.

[0177] Note that in this modification example, the administrator terminal device 9 and the plant harvesting management server 1 may be configured so that the user of the administrator terminal device 9 can specify the number of times the automatic harvesting device 5 moves along the route RT from the administrator terminal device 9.

[0178] According to this modification example, since the automatic harvesting device 5 moves along the route RT a plurality of times and the harvesting process is executed, compared with the mode in which the automatic harvesting device 5 moves along the route RT only once and the harvesting process is executed, it is possible to improve the harvesting rate of the fruit SF by the automatic harvesting device 5 in the agricultural house 8.

[0179] <B.3. Modification Example 3> In the above-described embodiments and Modification Examples 1 and 2, the plant harvesting management server 1 executed the harvestability determination process once during the period when the traveling device 52 of the automatic harvesting device 5 was stopped in the harvesting process. However, the present invention is not limited to such a mode. The plant harvesting management server 1 may execute the harvestability determination process a plurality of times during the period when the traveling device 52 of the automatic harvesting device 5 is stopped in the harvesting process.

[0180] FIG. 20 is a diagram showing an example of an outline of the operation of the plant harvesting management server 1 when the harvesting process according to this modification example is executed.

[0181] As illustrated in FIG. 20, when the harvest process is started, the plant harvest managing server 1 executes the processes of steps S101 to S111 described above with reference to FIG.

[0182] In this modification, when the result of the determination in step S109 is negative, the imaging device control section 213 controls the imaging device 54 to capture an image of the cultivation area, thereby acquiring again imaging information DG indicating the captured image GG from the imaging device 54 (S121). Note that in step S121, the control device 2 deletes the detected object information DB and the detected fruit information DF generated in step S107.

[0183] Next, the control device 2 executes a second harvest propriety determination process (S123).

[0184] Next, based on the results of the harvest feasibility determination process in step S123, the control device 2 determines whether or not there is a harvestable fruit SF that has not yet been harvested among one or more objects Obj contained in the captured image GG acquired by the imaging device control unit 213 in step S121 (S125).

[0185] If the result of the determination in step S125 is negative, that is, the control device 2 advances the process to step S101, thereby moving the automatic harvesting device 5 along the route RT. If the result of the judgment in step S125 is positive, the manipulator control unit 212 controls the manipulator 53 to harvest one fruit SF[m] from the unharvested harvestable fruits SF included in the captured image GG (S127), and proceeds to processing in step S125.

[0186] According to this modified example, the harvest feasibility determination process is executed multiple times during the period when the traveling device 52 of the automatic harvesting device 5 is stopped, which makes it possible to improve the harvest rate of fruit SF by the automatic harvesting device 5 in the agricultural greenhouse 8 compared to an embodiment in which the harvest feasibility determination process is executed only once during the period when the traveling device 52 of the automatic harvesting device 5 is stopped.

[0187] <B.4. Variant Example 4> In the above-described embodiments and Variant Examples 1 to 3, the harvesting device control unit 21, the object recognition unit 22, and the harvesting determination unit 23 are illustrated and described as being provided in the plant harvesting management server 1. However, the present invention is not limited to such an aspect. A part of the functions provided in the plant harvesting management server 1 may be provided in the automatic harvesting device 5.

[0188] In this variant example, the plant production and harvesting system Sys is different from the plant production and harvesting system Sys according to the embodiment in that the plant production and harvesting system Sys includes a plant harvesting management server 1B instead of the plant harvesting management server 1 and includes an automatic harvesting device 5B instead of the automatic harvesting device 5.

[0189] FIG. 21 is a functional block diagram showing an example of the outline of the configuration of the automatic harvesting device 5B according to this variant example.

[0190] As illustrated in FIG. 21, the automatic harvesting device 5B is different from the automatic harvesting device 5 according to the embodiment in that the automatic harvesting device 5B includes a control device 51B instead of the control device 51. The control device 51B includes the traveling device control unit 211, the manipulator control unit 212, and the imaging device control unit 213 described above.

[0191] FIG. 22 is a functional block diagram showing an example of the outline of the configuration of the plant harvesting management server 1B according to this variant example.

[0192] As illustrated in FIG. 22, the plant harvesting management server 1B is different from the plant harvesting management server 1 according to the embodiment in that the plant harvesting management server 1B includes a control device 2B instead of the control device 2. The control device 2B is different from the plant harvesting management server 1 according to the embodiment in that the control device 2B includes an imaging information acquisition unit 21B instead of the harvesting device control unit 21. The imaging information acquisition unit 21B (another example of the "acquisition unit") acquires imaging information DG from the imaging device 54 provided in the automatic harvesting device 5B via the imaging device control unit 213 provided in the automatic harvesting device 5B.

[0193] <B.5. Variant Example 5> In the above-described embodiments and Modifications 1 to 4, it was assumed that the plant production and harvesting system Sys includes one agricultural greenhouse 8, but the present invention is not limited to such an aspect.

[0194] FIG. 23 is a functional block diagram showing an example of the outline of the configuration of the plant production and harvesting system Sys-C according to this modification.

[0195] As illustrated in FIG. 23, the plant production and harvesting system Sys-C differs from the plant production and harvesting system Sys including a single agricultural greenhouse 8 in that it includes a plurality of agricultural greenhouses 8. In the plant production and harvesting system Sys-C, the plant harvesting management server 1 performs harvesting processing for a plurality of agricultural greenhouses 8.

[0196] <C. Supplementary Note> From the descriptions of the above-described embodiments and modifications, the following aspects can be grasped.

[0197] <Supplementary Note 1> The plant production and harvesting system Sys includes an imaging device control unit 213 that acquires imaging information DG indicating an imaging image GG obtained by imaging a cultivation area in which fruits SF of strawberries S are cultivated from an imaging device 54 that images the cultivation area, an object recognition unit 22 that recognizes an object Obj included in the imaging image GG indicated by the imaging information DG, and a harvesting determination unit 23 that determines whether to harvest the fruits SF by the automatic harvesting device 5 when there is an obstacle OB that hinders the harvesting of the fruits SF by the automatic harvesting device 5 among the objects Obj recognized by the object recognition unit 22, and a manipulator control unit 212 that controls the automatic harvesting device 5 to harvest the fruits SF when the harvesting determination unit 23 determines to harvest the fruits SF.

[0198] According to the plant production and harvesting system Sys described in Supplementary Note 1, even if an obstacle OB corresponding to strawberries S is present, fruits SF may be harvested based on the determination result by the harvest determination unit 23. Therefore, according to the plant production and harvesting system Sys according to Supplementary Note 1, when an obstacle OB corresponding to strawberries S is present, the harvest rate of fruits SF by the automatic harvesting device 5 can be improved compared to a mode in which the fruits SF are not harvested in any case.

[0199] <Appendix 2> In the plant production and harvesting system Sys described in Supplementary Note 1, the harvesting decision unit 23 decides whether or not to harvest the fruit SF by the automatic harvesting device 5 based on the characteristics of the obstacle OB.

[0200] According to the plant production and harvesting system Sys described in Supplementary Note 2, even if an obstacle OB corresponding to strawberries S is present, the fruits SF may be harvested based on the determination result of the harvest determination unit 23 based on the characteristics of the obstacle OB. Therefore, according to the plant production and harvesting system Sys according to Supplementary Note 2, when an obstacle OB corresponding to strawberries S is present, the harvest rate of the fruits SF by the automatic harvesting device 5 can be improved compared to a mode in which the fruits SF are not harvested in any case.

[0201] <Appendix 3> In the plant production and harvesting system Sys described in Appendix 1 or Appendix 2, the harvest decision unit 23 is characterized in that it determines whether or not an obstacle OB exists among the object Obj included in the captured image GG based on the position of the object Obj included in the captured image GG and the position of the fruit SF included in the captured image GG.

[0202] According to the plant production and harvesting system Sys described in Appendix 3, the presence or absence of an obstacle OB is determined based on the position of the object Obj and the position of the fruit SF in the captured image GG, which makes it possible to reduce the processing load associated with determining whether or not an obstacle OB exists, compared to a mode in which the presence or absence of an obstacle OB is determined based on the position of the object Obj and the position of the fruit SF in real space.

[0203] <Appendix 4> In the plant production and harvesting system Sys described in Appendices 1 to 3, the harvest decision unit 23 is characterized in that, when the obstacle OB is an immature fruit SFB, it decides whether or not to harvest the fruit SF using the automatic harvesting device 5 based on the estimated result of the future harvest possibility of the immature fruit SFB.

[0204] According to the plant production and harvesting system Sys described in Appendix 4, when the obstacle OB corresponding to the fruit SF is an immature fruit SFB, it is determined whether or not to harvest the fruit SF based on the future harvest possibility of the immature fruit SFB, so that when the obstacle OB is an immature fruit SFB, the possibility of damaging immature fruit SFB that can be harvested in the future can be reduced compared to a mode in which the harvesting of the fruit SF is forced when the obstacle OB is an immature fruit SFB.

[0205] <Appendix 5> In the plant production and harvesting system Sys described in Supplementary Note 1 to Supplementary Note 3, the harvest decision unit 23 operates in a plurality of operation modes including a future priority operation mode in which it is decided not to harvest the fruit SF by the automatic harvesting device 5 when the obstacle OB is an immature fruit SFB, and a current priority operation mode in which it is decided to harvest the fruit SF by the automatic harvesting device 5 when the obstacle OB is an immature fruit SFB.

[0206] According to the plant production and harvesting system Sys described in Appendix 5, the harvest decision unit 23 is capable of operating in a future priority operating mode and a present priority operating mode, making it easier to adjust the current harvest yield of fruit SF and the future harvest yield of fruit SF compared to a mode in which the harvest decision unit 23 operates in a single operating mode. [Explanation of symbols]

[0207] 1...plant harvest management server, 2...control device, 3...storage device, 4...communication device, 5...automatic harvesting device, 8...agricultural house, 9...administrator terminal device, 21...harvesting device control unit, 22...object recognition unit, 23...harvesting decision unit, 51...control device, 52...traveling device, 53...manipulator, 54...imaging device, 55...detection device, 56...communication device, 91...control device, 92...display device, 93...operation device, 94...storage device, 95...communication device, 211...traveling device control unit, 212...manipulator control unit, 213...imaging device control unit, 231...fruit status determination unit, 232...stalk status determination unit, 233...obstacle analysis unit, Sys...plant production harvesting system

Claims

1. An acquisition unit that acquires imaging information indicating an image of a cultivation area in which harvest objects are cultivated from an imaging device that images the cultivation area; a recognition unit that recognizes an object included in a captured image indicated by the imaging information; When an obstacle that hinders the harvesting of the harvest object by the harvesting device is present among the objects recognized by the recognition unit, A decision unit that decides whether or not to harvest the harvest object by the harvesting device; an apparatus control unit that controls the harvesting apparatus to harvest the harvest object when the decision unit decides to harvest the harvest object; Equipped with A harvesting system comprising:

2. The determination unit is Based on the characteristics of the obstacle, determining whether to harvest the harvest object by the harvesting device; A harvesting system according to claim 1 , characterized in that

3. The determination unit is A position of an object included in the captured image; Based on the position of the harvest object in the captured image, determining whether the obstacle is present among objects included in the captured image; A harvesting system according to claim 1 , characterized in that

4. The determination unit is When the obstacle is an immature fruit or vegetable, Based on the results of the estimation of future harvest potential of the immature fruit or vegetable, determining whether to harvest the harvest object by the harvesting device; A harvesting system according to claim 1 , characterized in that

5. The determination unit is When the obstacle is an immature fruit or vegetable, a first operating mode for determining not to harvest the harvest object by the harvesting device; When the obstacle is an immature fruit or vegetable, a second operating mode for determining whether the harvesting device is to harvest the harvesting object; Operates in multiple modes of operation, including A harvesting system according to claim 1 , characterized in that

6. Obtaining imaging information indicating an image of a cultivation area in which harvest objects are cultivated from an imaging device that images the cultivation area; Recognizing an object included in a captured image indicated by the imaging information; When an obstacle that hinders the harvesting device from harvesting the harvest object is present among the recognized objects, determining whether to harvest the harvest object by the harvesting device; When it is determined that the harvest object is to be harvested, controlling the harvesting device to harvest the harvest object. A harvesting method characterized by:

7. The processor, An acquisition unit that acquires imaging information indicating an image of a cultivation area in which harvest objects are cultivated from an imaging device that images the cultivation area; a recognition unit that recognizes an object included in a captured image indicated by the imaging information; When an obstacle that hinders the harvesting of the harvest object by the harvesting device is present among the objects recognized by the recognition unit, A decision unit that decides whether or not to harvest the harvest object by the harvesting device; an apparatus control unit that controls the harvesting apparatus to harvest the harvest object when the decision unit decides to harvest the harvest object; and make it function. A program characterized by:

Images