PLANT SELECTION WORK SUPPORT DEVICE, PLANT SELECTION WORK SUPPORT COMPOSITE IMAGE DISPLAY SYSTEM DEVICE, TRAINED MODEL GENERATION DEVICE, PLANT SELECTION WORK SUPPORT METHOD, PLANT SELECTION WORK SUPPORT COMPOSITE IMAGE DISPLAY METHOD, TRAINED MODEL PRODUCTION METHOD, PROGRAM, AND RECORDING MEDIUM
The plant sorting work support system addresses inefficiencies in plant sorting by using image recognition and trained models to generate composite images, enhancing the efficiency and accuracy of tasks like fruit thinning and pruning.
Patent Information
- Application Number
- JP2020198626
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-11-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-11-30
AI Technical Summary
Existing technologies lack efficient support for plant sorting operations such as thinning and fruit thinning, which are crucial for promoting growth by concentrating nutrients and water on specific fruits, and there is a need for improved methods to reduce the physical burden on workers.
A plant sorting work support system comprising an image acquisition unit, work target recognition, removal target determination, composite image generation, and sorting target determination, utilizing trained models to enhance the efficiency of plant sorting operations.
The system enables more efficient sorting operations by providing visual guidance through composite images, reducing worker burden, and improving accuracy and speed in tasks like fruit thinning and pruning.
Smart Images

Figure 0007588310000001 
Figure 0007588310000002 
Figure 0007588310000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a plant selection work support device, a plant selection work support composite image display system device, a trained model generation device, a plant selection work support method, a plant selection work support composite image display method, a trained model production method, a program, and a recording medium. [Background technology]
[0002] In the cultivation of fruits and vegetables, in order to promote growth by concentrating nutrients and moisture in specific fruits, thinning (thinning out) is carried out to limit the number of fruits (young fruits) that have borne fruit in the early stages of development. In order to reduce the physical burden on workers who carry out this thinning work, technology related to a fruit thinning tool that includes a cutting mechanism has been reported (Patent Document 1, etc.). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2016-021936 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, there is a demand for technology that can assist in sorting tasks such as thinning and fruit removal, and enable more efficient sorting tasks.
[0005] Therefore, the present invention aims to provide a plant sorting work support device, a plant sorting work support synthetic image display system device, a trained model generation device, a plant sorting work support method, a plant sorting work support synthetic image display method, a trained model production method, a program, and a recording medium that support sorting work such as thinning and fruit picking, and enable more efficient sorting work. [Means for solving the problem]
[0006] In order to achieve the above object, the plant selection work support device of the present invention comprises: The apparatus includes an image acquisition unit, a work object recognition unit, a removal object determination unit, a composite image generation unit, a transition determination unit, and a selection object determination unit, The image acquisition unit acquires an image of all or a part of a plant that is a work target, The plant has a work target group formed of a plurality of components, The work object recognition unit recognizes each component of the work object group as a work object in the acquired image, The removal target determination unit determines the components to be removed from the recognized work object, the composite image generating unit generates a composite image by superimposing visual information indicating the component determined to be the removal target on the image; the image acquisition unit acquires an image of the whole or part of the plant from which the component to be removed has been removed as a post-removal image; The transition determination unit determines whether or not to perform a selection object determination in the selection object determination unit based on a predetermined criterion in the post-removal image, When the transition determination unit determines that the selection target determination is to be performed, the selection target determination unit determines, in the post-removal image, the components to be left from the components of the work target group remaining after the removal, as selection targets; The composite image generating unit is a device that generates a composite image by superimposing visual information indicating the component determined to be the selection target on the image.
[0007] The plant selection work support composite image display system of the present invention comprises: The plant selection work support device of the present invention includes a display device, The display device is a device that can be carried by an operator, In the plant selection work support device, Each composite image generated by the composite image generation unit is output to the display device; The display device includes an acquisition unit and a display processing unit, The acquisition unit acquires the composite image output from the plant selection work supporting device, The display processing unit is a device that executes a process of displaying the composite image.
[0008] The trained model generation device of the present invention comprises: The present invention includes a training data acquisition unit and a trained model generation unit, the teacher data acquisition unit acquires teacher data in which an image of all or a part of a plant to be worked on, feature information indicating a feature amount of the plant, and judgment information indicating at least one of components to be removed and components to be selected are recorded in association with each other; The plant has a work target group formed of a plurality of components, The trained model generation unit is a device that, when an image containing all or part of the plant is input, generates a trained model by machine learning using the training data, recognizing each component of the work target group as a work target, and outputting at least one of a determination result of determining the components to be removed from the recognized work targets, and a determination result of determining the components to be retained from each component of the work target group as selection targets.
[0009] The plant selection work support method of the present invention comprises the steps of: The method includes an image acquisition step, a work object recognition step, a removal object determination step, a composite image generation step, a transition determination step, and a selection object determination step, The image acquisition step acquires an image of all or a part of a plant to be worked on, The plant has a work target group formed of a plurality of components, The work object recognition step recognizes each component of the work object group as a work object in the acquired image, The removal target determination step determines the components to be removed in the recognized work object, the composite image generating step generates a composite image by superimposing visual information indicating the component determined to be the removal target on the image; The image acquisition step includes acquiring an image of all or a part of the plant from which the component to be removed has been removed as a post-removal image, The transition determination step determines whether or not to perform a selection target determination in the selection target determination step based on a predetermined criterion in the post-removal image, When the determination in the transition determination step is that the selection target determination is to be performed, the selection target determination step determines, in the post-removal image, the components to be left from the components of the work target group remaining after the removal, as selection targets; The composite image generating step is a method of generating a composite image by superimposing visual information indicating the component determined to be the selection target on the image.
[0010] The plant selection work support composite image display method of the present invention comprises the steps of: The present invention includes a plant selection operation support method and a display method, the display method is a process for a device that is portable by a worker, In the plant selection work support method, Each composite image generated in the composite image generating step is output to a device portable by the worker, The display method includes an acquisition step and a display processing step, The acquiring step acquires the composite image output by the plant selection work support method, The display processing step is a method for executing a process for displaying the composite image.
[0011] The trained model production method of the present invention includes: The method includes a teacher data acquisition step and a trained model generation step. The teacher data acquisition step acquires teacher data in which an image of all or a part of a plant to be worked on, characteristic information indicating a characteristic amount of the plant, and judgment information indicating at least one of components to be removed and components to be selected are recorded in association with each other; The plant has a work target group formed of a plurality of components, The trained model generation process is a production method in which, by machine learning using the training data, when an image containing all or part of the plant is input, a trained model is generated which recognizes each of the components of the work target group as work targets and outputs at least one of a determination result of determining the components to be removed from the recognized work targets and a determination result of determining the components to be retained from each of the components of the work target group as selection targets. Effect of the Invention
[0012] The present invention supports sorting operations such as thinning and fruit picking, enabling more efficient sorting operations. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is a block diagram showing a configuration of an example of a plant selection work supporting device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of the plant selection work supporting device according to the first embodiment. [Diagram 3] FIG. 3 is a flowchart showing an example of processing in the plant selection work supporting device of the first embodiment. [Figure 4] FIG. 4(A) is a block diagram showing an example of the configuration of a plant selection work support composite image display system device of embodiment 2, and FIG. 4(B) is a block diagram showing an example of the hardware configuration of a display device in the plant selection work support composite image display system device. [Diagram 5] FIG. 5 is a flowchart showing an example of processing in the plant selection work supporting composite image display system device of the second embodiment. [Figure 6] FIG. 6 is a block diagram illustrating an example of a configuration of a trained model generation device according to the third embodiment. [Figure 7] FIG. 7 is a flowchart showing an example of processing in the trained model generation device of embodiment 3. [Figure 8]FIG. 8 is a diagram showing an example of each image displayed on the display of the display device in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] In the plant selection work support device of the present invention, for example, An aspect may be such that the predefined criterion in the transition determination section is at least one of the number of the components in the post-removal image and the presence or absence of an assisting tool in the post-removal image.
[0015] In the plant selection work support device of the present invention, for example, At least one of the work target recognition unit, the removal target determination unit, and the selection target determination unit executes each process using a trained model; The work object recognition unit inputs the acquired image into a trained model trained to recognize each component of the work object group as a work object from within the input image, and recognizes each component of the work object group as a work object; The removal target determination unit inputs the image in which a work object is recognized into a trained model trained to determine the components to be removed from the work object recognized in the input image, and determines the components to be removed from the recognized work object; The selection target determination unit may input the post-removal image into a trained model trained to determine the components to be left as selection targets from the components of the work target group in the input image, and determine the components to be left as selection targets from the components of the work target group remaining after the removal. In the above aspect, for example, a trained model used by at least one of the removal target determination unit and the selection target determination unit is a trained model trained to make each of the determinations based on an evaluation value; At least one of the removal target determination section and the selection target determination section may make the respective determinations based on evaluation values.
[0016] In the plant selection work support device of the present invention, for example, The composite image generating unit may define display parameters for each result of the judgment, and generate a composite image in which each piece of visual information is superimposed on the image in accordance with the definition of the display parameters.
[0017] In the plant selection work support device of the present invention, for example, At least one of the image acquired by the image acquisition section and the post-removal image may be an image in which an auxiliary tool is used to align the multiple components along the auxiliary tool. In the above aspect, for example, In one embodiment, the color of the surface of the assisting tool that contacts the plurality of components is monochromatic.
[0018] In the plant selection work support device of the present invention, for example, The composite image generation unit may further generate a composite image by superimposing visual information for recognition indicating the component recognized by the work object recognition unit on the image.
[0019] In the plant selection work support composite image display system device and the plant selection work support composite image display method of the present invention, for example, The display device may be a device that can be worn on the head of a worker.
[0020] In the trained model generation device of the present invention, for example, the teacher data acquisition unit acquires teacher data in which an image of all or a part of a plant to be worked on, feature information indicating a feature amount of the plant, judgment information indicating at least one of components to be removed and components to be selected, and evaluation information indicating an evaluation value of each of the components are associated with each other and recorded; The trained model generation unit may be configured to, when an image containing all or part of the plant is input, generate a trained model by machine learning using the training data, recognizing each component of the work target group as a work target, assigning an evaluation value to each component of the recognized work target, and outputting at least one of a determination result of determining the components to be removed from the work target based on the evaluation value, and a determination result of determining the components to be retained from each component of the work target group as selection targets.
[0021] In the plant selection work support method of the present invention, for example, An aspect may be such that the predefined criterion in the transition determination step is at least one of the number of the components in the post-removal image and the presence or absence of an assisting tool in the post-removal image.
[0022] In the plant selection work support method of the present invention, for example, At least one of the work target recognition step, the removal target determination step, and the selection target determination step executes each process using a trained model; The work object recognition step inputs the acquired image into a trained model trained to recognize each component of a work object group as a work object from within an input image, and recognizes each component of the work object group as a work object; The removal target determination step includes inputting the image in which a work object is recognized into a trained model trained to determine the component to be removed from the work object recognized in the input image, and determining the component to be removed from the recognized work object; The selection target determination process may be such that the post-removal image is input to a trained model that has been trained to determine, as selection targets, the components to be retained from the components of the work target group in the input image, and the components to be retained from the components of the work target group remaining after the removal are determined to be selection targets. In the above aspect, for example, A trained model used in at least one of the removal target determination step and the selection target determination step is a trained model trained to make each of the determinations based on an evaluation value, At least one of the removal target determining step and the selection target determining step may be configured to perform the respective determinations based on evaluation values.
[0023] In the plant selection work support method of the present invention, for example, The composite image generating step may be a step of defining display parameters for each result of the judgment, and generating a composite image in which each piece of visual information is superimposed on the image in accordance with the definition of the display parameters.
[0024] In the plant selection work support method of the present invention, for example, At least one of the image acquired in the image acquisition step and the post-removal image may be an image in which an auxiliary tool is used and the plurality of components are aligned along the auxiliary tool. In the above aspect, for example, In one embodiment, the color of the surface of the assisting tool that contacts the plurality of components is monochromatic.
[0025] In the plant selection work support method of the present invention, for example, The composite image generating step may further comprise the step of generating a composite image in which visual information for recognition indicating the component recognized in the work object recognizing step is superimposed on the image.
[0026] In the trained model production method of the present invention, for example, The teacher data acquisition step acquires teacher data in which an image of all or a part of a plant to be worked on, characteristic information indicating a characteristic amount of the plant, judgment information indicating at least one of components to be removed and components to be selected, and evaluation information indicating an evaluation value of each of the components are associated with each other and recorded; The trained model generation process may be such that, when an image containing all or part of the plant is input, a trained model is generated by machine learning using the training data, recognizing each component of the work target group as a work target, assigning an evaluation value to each component of the recognized work target, and outputting at least one of a determination result of determining the components to be removed from the work target based on the evaluation value, and a determination result of determining the components to be retained from each component of the work target group as selection targets.
[0027] The program of the present invention is a program for causing a computer to execute each step of the method of the present invention as a procedure.
[0028] The recording medium of the present invention is a computer-readable recording medium having the program of the present invention recorded thereon.
[0029] In the present invention, the "plant" is not particularly limited as long as it has a work target group formed from a plurality of components. Specific examples include woody plants such as fruit trees; herbaceous plants such as flowers and vegetables; and the like. More specifically, in the case of a fruit tree, the components are, for example, fruits (young fruits, etc.), and the work target group is a fruit bud (a group of fruits growing from one flower bud). In addition, in the case of a woody plant or herbaceous plant other than a fruit tree, the components are, for example, treetops, side buds, lateral branches, main branches, and the like, and the work target group is, for example, lateral branches, main branches, main trunks, and the like.
[0030] Next, an embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiment. In each of the drawings, the same parts are given the same reference numerals. In addition, the explanations of each embodiment can be mutually incorporated unless otherwise specified, and the configurations of each embodiment can be combined unless otherwise specified.
[0031] [Embodiment 1] Fig. 1 is a block diagram showing an example of the configuration of a plant selection work support device 10 according to the present embodiment. As shown in Fig. 1, the device 10 includes an image acquisition unit 11, a work target recognition unit 12, a removal target determination unit 13, a composite image generation unit 14, a transition determination unit 15, and a selection target determination unit 16. Furthermore, the device 10 may include a storage unit 17 and an imaging unit 18 as optional components.
[0032] The device 10 may be, for example, a single device including each of the above-mentioned units, or a device to which each of the above-mentioned units can be connected via a communication line network. The device 10 may be connected to an external device described later via the communication line network. The communication line network is not particularly limited, and a known network may be used, for example, wired or wireless. Examples of the communication line network include an Internet line, WWW (World Wide Web), a telephone line, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), and the like. Examples of wireless communication include WiFi (Wireless Fidelity), Bluetooth (registered trademark), local 5G, LPWA, and the like. Examples of the wireless communication may include a form in which each device directly communicates (Ad Hoc communication), infrastructure (infrastructure communication), indirect communication via an access point, and the like. The device 10 may be incorporated into a server as a system, for example. The device 10 may be, for example, a personal computer (PC, for example, desktop type or notebook type) on which the program of the present invention is installed, a smartphone, a tablet terminal, etc. Furthermore, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the units is on a server and the other units are on a terminal.
[0033] 2 illustrates an example of a block diagram of the hardware configuration of the present device 10. The present device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, and the like. Furthermore, the hardware configuration of the present device 10 may further include, for example, an imaging device 108 as an optional configuration. Each unit of the present device 10 is connected to each other via the bus 103 by each interface (I / F).
[0034] The central processing unit 101 is responsible for the overall control of the present device 10. In the present device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various information. Specifically, for example, the central processing unit 101 functions as an image acquisition unit 11, a work target recognition unit 12, a removal target determination unit 13, a composite image generation unit 14, a transition determination unit 15, and a selection target determination unit 16.
[0035] The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (external database, etc.), a printer, an external input device, an external display device, an external imaging device, etc. The present device 10 can be connected to an external network (the above-mentioned communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.
[0036] An example of the memory 102 is a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operation programs, such as the program of the present invention, stored in the storage device 104 described below, and the central processing unit 101 receives data from the memory 102 and executes the program. The main memory is, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read only memory).
[0037] The storage device 104 is also called an auxiliary storage device, for example, in contrast to the main memory (primary storage device). As described above, the storage device 104 stores an operation program including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive for reading and writing from and to the recording medium. The recording medium is not particularly limited, and may be, for example, an internal or external type, such as a hard disk (HD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, or memory card. The storage device 104 may be, for example, a hard disk drive (HDD) in which a recording medium and a drive are integrated, or a solid state drive (SSD).
[0038] In the present device 10, the memory 102 and storage device 104 can also store user log information as well as information obtained from an external database (not shown).
[0039] The storage unit 17 can use, for example, a memory 102 and a storage device 104 .
[0040] The present device 10 may further include, for example, an input device 105, a display device 106, and an imaging device 108. The input device 105 is, for example, a touch panel, a keyboard, a mouse, etc. The display device 106 is, for example, an LED display, a liquid crystal display, etc. The imaging device 108 is, for example, a camera, etc., and functions as an imaging unit 18.
[0041] Next, an example of the plant selection work support method of this embodiment will be described with reference to the flowchart of Fig. 3. The plant selection work support method of this embodiment is carried out as follows, for example, using the plant selection work support device 10 of Fig. 1. Note that the plant selection work support method of this embodiment is not limited to the use of the plant selection work support device 10 of Fig. 1.
[0042] First, the image acquisition unit 11 acquires an image of all or a part of a plant to be worked on (S11). The plant has a work target group formed of a plurality of components. If the present device 10 includes, for example, an imaging unit 18, the imaging unit 18 may capture an image of all or a part of the plant to be worked on before the step (S11). The captured image may then be acquired by the image acquisition unit 11. The image acquisition unit 11 may also acquire the image captured by an external imaging device, for example, via the communication line network. The image may be, for example, a video or a still image.
[0043] Next, the work object recognition unit 12 recognizes each component of the work object group as a work object in the acquired image (S12). The work object recognition unit 12 may recognize the work object by using, for example, a trained model as described below.
[0044] Next, the removal target determination unit 13 determines the components to be removed in the recognized work object (S13). The removal target determination unit 13 may recognize the work object by using a trained model, for example, as described later. The removal target determination unit 13 may determine, for example, a predefined number of removal targets. The predefined number is not particularly limited, and may be one or more than one, and may be arbitrarily defined by the user. The removal target determination unit 13 may also determine the components to be removed by, for example, determining a removal priority. The removal priority is a degree of likelihood that the components are removal targets. The "components to be removed" can be said to be components that the removal target determination unit 13 has determined to be removed by work such as fruit thinning or pruning.
[0045] Next, the composite image generating unit 14 generates a composite image by superimposing visual information indicating the component determined to be the removal target on the image (S14). When the removal target determining unit 13 determines the removal priority, the composite image generating unit 14 may generate a composite image by superimposing information indicating the removal priority on the image as the visual information. The visual information is not particularly limited, and may be information that can be visually recognized. Specifically, there are frames surrounding the component determined to be the removal target, characters and symbols indicating the component, and the like. Hereinafter, the visual information is also referred to as, for example, visual information for the removal target. The present device 10 may output the composite image to, for example, an external display device or a display device 106. This allows a worker performing fruit thinning or pruning work to recognize the component to be thinned or pruned.
[0046] Next, the image acquiring unit 11 acquires an image of all or a part of the plant from which the component to be removed has been removed as a post-removal image (S15). If the present device 10 includes, for example, an imaging unit 18, the imaging unit 18 may capture an image of all or a part of the plant from which the component to be removed has been removed (post-removal image) after the step (S14) and before the step (S15). The captured post-removal image may then be acquired by the image acquiring unit 11. Furthermore, the image acquiring unit 11 may acquire the post-removal image captured by an external imaging device, for example, via the communication line network.
[0047] Next, the transition determination unit 15 determines whether or not to perform a selection target determination in the selection target determination unit 16 based on a predetermined criterion in the post-removal image (S16). The predetermined criterion is not particularly limited, but may be, for example, the number of the components in the post-removal image, the presence or absence of an auxiliary tool in the post-removal image, and a combination thereof. The number of the components in the post-removal image is not particularly limited, and the user can set any number in advance. Specifically, for example, there are five, three, and the like. The auxiliary tool is, for example, a glove, a plate, and the like. The transition determination unit 15 may determine to perform a selection target determination in the selection target determination unit 16, for example, when the number of the components in the post-removal image is a predetermined number or less, when the auxiliary tool is shown in the post-removal image, and when the number of the components in the post-removal image is a predetermined number or less and the auxiliary tool is shown in the post-removal image.
[0048] When the transition determination unit 15 determines that the selection target determination is to be performed (YES), the selection target determination unit 16 determines the components to be left from the components of the work target group remaining after the removal in the post-removal image as selection targets (S17). The selection target determination unit 16 may determine the selection targets using a trained model, for example, as described below. The number of components to be left may be less than the number of components in the post-removal image, and may be, for example, one or two or more.
[0049] Next, the composite image generating unit 14 generates a composite image by superimposing visual information indicating the components determined to be the selection targets on the image (S18), and ends the process (END). Hereinafter, the visual information is also referred to as, for example, visual information for selection targets. The visual information for selection targets may, for example, be a frame surrounding the components, letters and symbols indicating the components, etc., similar to the visual information for removal targets. The present device 10 may, for example, output the composite image to an external display device or a display device 106. This allows a worker performing work such as fruit thinning or pruning to recognize the components that should be left in the work target group.
[0050] On the other hand, if the transition judgment unit 15 judges that the selection target judgment should not be performed (NO), for example, the judgment by the removal target judgment unit 13 (S13), the generation of a composite image by the composite image generation unit 14 (S14), and the acquisition of the post-removal image by the image acquisition unit 11 (S15) may be repeated.
[0051] The composite image generating unit 14 may, for example, define display parameters for each result of the judgment in the step (S14) and the step (S18), and generate a composite image in which the visual information for the removal target and the identification information for the selection target are superimposed on the image according to the definition of the display parameters. The display parameters are, for example, colors, etc. Specifically, the composite image generating unit 14 may, for example, generate a composite image in which frames (visual information for the removal target) of different colors are superimposed on the components in the image that are the removal targets and the components that are not the removal targets. Furthermore, the composite image generating unit 14 may, for example, generate a composite image in which frames (visual information for the removal target) of different colors are superimposed on the components in the image for each of the removal priorities. Furthermore, the composite image generating unit 14 may, for example, generate a composite image in which frames (visual information for the selection target) of different colors are superimposed on the components in the image that are the selection targets and the components that are not the selection targets.
[0052] For example, the composite image generating unit 14 may further generate a composite image (S19) in which visual information for recognition indicating the component recognized by the work object recognizing unit 12 is superimposed on the image after the step (S12) and before the step (S13). The visual information for recognition may be, for example, a frame surrounding the component, characters and symbols indicating the component, etc., similar to the visual information for the removal target. The present device 10 may output the composite image to an external display device or a display device 106, for example. This allows the worker to know whether or not the component has been recognized. That is, in this embodiment, a composite image in which visual information for the removal target is superimposed only on the component in the image that is the removal target, or a composite image in which visual information for recognition is superimposed on all the recognized components including the component to be removed, may be generated. In the latter case, for example, the visual information for recognition is not included in the composite image generated in the step (S14) after the step (S19).
[0053] At least one of the image acquired by the image acquisition unit 11 in the step (S11) and the post-removal image in the step (S15) may be an image in which the plurality of components are aligned along an auxiliary tool, for example, using the auxiliary tool. The auxiliary tool is, for example, a glove, a plate, etc., as described above. The color of the auxiliary tool is not particularly limited, but is, for example, a color opposite to the color of the components. In addition, the color of the surface of the auxiliary tool that contacts the plurality of components may be, for example, a single color. This prevents the plurality of components from overlapping each other, and makes it possible to clearly distinguish the components from the other parts (for example, components of other fruits, etc.), thereby improving the accuracy of the recognition by the work object recognition unit 12 (S12), the judgment by the removal object judgment unit 13 (S13), the judgment by the transition judgment unit 15 (S15), and the judgment by the selection object judgment unit 16 (S17).
[0054] (Variation 1) A form in which at least one of the work target recognition unit 12, the removal target determination unit 13, and the selection target determination unit 16 executes each process using a trained model will be described.
[0055] The work object recognition unit 12 may, for example, in the step (S12), input the acquired image into a trained model trained to recognize each component of the work object group from within the input image as a work object, and recognize each component of the work object group as a work object.
[0056] The removal target determination unit 13 may, for example, in step (S13), input the image in which the work object is recognized into a trained model trained to determine the components to be removed from the work object recognized in the input image, and determine the components to be removed from the recognized work object.
[0057] The selection target determination unit 16 may, for example, input the post-removal image into a trained model that has been trained to determine, in the step (S17), the components to be left from the components of the work target group in the input image as selection targets, and determine, from the components of the work target group remaining after the removal, the components to be left as selection targets.
[0058] Each of the trained models may be a trained model using a multi-layer network, for example. The trained model using the multi-layer network includes, for example, an input layer that inputs at least one of an image including each component of a work target group, an image including the component to be removed, and an image including the component to be selected, an output layer that outputs at least one of an image in which each component is recognized as a work target, an image in which the component is determined to be the target to be removed, and an image in which the component is determined to be the target to be selected, and at least one intermediate layer provided between the input layer and the output layer. Each of the trained models may be, for example, a program module that is a part of artificial intelligence software. Examples of the multi-layer network include a neural network. Examples of the neural network include a convolution neural network (CNN) that targets images. Note that each of the trained models is not limited to the CNN, and may be a trained model constructed with other learning algorithms such as a neural network other than CNN, a Support Vector Machine (SVM), a Bayesian network, and a regression tree.
[0059] When the present device 10 includes a storage unit 17, for example, the storage unit 17 may store in advance entity files constituting each of the trained models. These entity files may be configured as a part of a program. For example, each of the trained models may be a trained model produced by the production method of the present invention. For example, each of the trained models may be acquired from the outside via the communication line network.
[0060] The trained model may be, for example, a trained model for each part separately, or may be a trained model common to all parts.
[0061] In particular, the trained model used by at least one of the removal target determination unit 13 and the selection target determination unit 16 may be a trained model trained to make the above-mentioned judgments based on the evaluation value. At least one of the removal target determination unit 13 and the selection target determination unit 16 may make the above-mentioned judgments based on the evaluation value.
[0062] The evaluation value will be specifically described below. The trained model is trained by using, as teacher data, images in which evaluation values such as 1 point to 5 points are associated with the components. The evaluation value is arbitrarily set by the user according to the viewpoint. The viewpoints include, for example, the size of the components such as fruit, the shape of the components, the order of the components in the work target group, the thickness of the axis, the color of the components, the direction in which the components are formed, and which flower bud the fruit is from. The evaluation value may be a total score obtained by integrating these multiple viewpoints, an average score, or an evaluation value of any one of the viewpoints. The evaluation value of which viewpoint is to be used among the multiple viewpoints may be arbitrarily set by the user, or the trained model itself may autonomously determine which viewpoint to focus on through learning.
[0063] Specifically, the removal target determination unit 13 may determine, for example, the components having an evaluation value equal to or less than a preset value as the removal target, or may determine the components other than the component having the highest evaluation value as the removal target. Also, the selection target determination unit 16 may determine, for example, the components having an evaluation value equal to or more than a preset value as the selection target, or may determine the component having the highest evaluation value as the selection target.
[0064] Also, at least one of the removal target determination unit 13 and the selection target determination unit 16 may calculate a value of likelihood (probability) as an evaluation value for each component by the trained model using the evaluation value in the steps (S13) and (S17). Then, at least one of the removal target determination unit 13 and the selection target determination unit 16 may make each determination based on the degree of the probability when there are components with the same evaluation value. That is, the removal target determination unit 13 may determine, for example, components other than the component with the highest evaluation value and the highest probability as the removal target. Also, the selection target determination unit 16 may determine, for example, the component with the highest evaluation value and the highest probability as the selection target.
[0065] According to the first embodiment and the first modification, a composite image is generated on which the visual information for the selection target is superimposed, thereby supporting the selection work. Specifically, the worker can easily determine the target for fruit thinning and removal by looking at the composite image, and the work speed is improved. Furthermore, according to the first embodiment and the first modification, the worker can easily determine the target for removal and selection regardless of the worker's skill or experience, which makes it easy to increase the number of new farmers and expand the scale of farms by utilizing their employment.
[0066] [Embodiment 2] 4(A) is a block diagram showing an example of the configuration of the plant selection work supporting composite image display system device 100 of this embodiment. As shown in FIG. 4, this device 100 includes the plant selection work supporting device 10 described in the first embodiment and the first modified example, and a display device 20.
[0067] The display device 20 is a device that can be carried by the worker, and may be, for example, a device that can be worn on the worker's head. Specifically, for example, there are a smartphone, a tablet terminal, a wearable terminal (wristwatch type, bracelet type, glasses type, etc.), etc.
[0068] As shown in Fig. 4(A), the display device 20 includes an acquisition unit 21 and a display processing unit 22. Furthermore, the display device 20 may include, for example, an imaging unit 23 as an optional configuration. The display device 20 may be, for example, a single device including each of the above-mentioned units, or a device in which each of the above-mentioned units can be connected via the communication line network. Furthermore, the display device 20 can be connected to the above-mentioned external device via the communication line network.
[0069] 4(B) illustrates a block diagram of a hardware configuration of the display device 20. Unless otherwise specified, the hardware configuration of the display device 20 is similar to the hardware configuration of the plant selection work support device 10 illustrated in FIG.
[0070] The central processing unit 101 functions as, for example, an acquisition unit 21 and a display processing unit 22.
[0071] The imaging device 108 functions as, for example, the imaging section 23 .
[0072] Next, an example of the plant selection work support composite image display method of this embodiment will be described with reference to the flowchart of Fig. 5. The plant selection work support composite image display method of this embodiment is implemented as follows, for example, using the plant selection work support composite image display system device 100 of Fig. 4. Note that the plant selection work support composite image display method of this embodiment is not limited to use of the plant selection work support composite image display system device 100 of Fig. 4.
[0073] First, in the plant selection work supporting device 10, steps (S11) to (S18) shown in FIG. 5 are executed similarly to the steps (S11) to (S18) shown in FIG.
[0074] The plant selection work support device 10 outputs each composite image generated by the composite image generation unit 14 in the step (S14) and the step (S18) to the display device 20 via the communication line network (S20). Furthermore, the plant selection work support device 10 may also output the composite image generated by the composite image generation unit 14 to the display device 20 in the step (S19).
[0075] Next, the display device 20 acquires the composite image output from the plant selection work supporting device 10 by the acquisition unit 21 (S21).
[0076] Then, the display device 20 executes a process of displaying the composite image by the display processing unit 22 (S22), and ends the process (END). The process is performed on the display device 106 of the display device 20, for example.
[0077] If the display device 20 includes, for example, an imaging unit 23, an image of all or a part of the plant to be the work target may be captured by the imaging unit 23 prior to the step (S11). Then, the display device 20 may output the captured image to the plant sorting work support device 10 via the communication line network. The image acquisition unit 11 of the plant sorting work support device 10 acquires the output image.
[0078] Furthermore, in the case where the display device 20 includes, for example, an imaging unit 23, after the step (S14) and before the step (S15), the imaging unit 23 may capture an image (post-removal image) of all or a part of the plant from which the component to be removed has been removed. Then, the display device 20 may output the captured post-removal image to the plant sorting work support device 10 via the communication line network. The image acquisition unit 11 of the plant sorting work support device 10 acquires the output post-removal image.
[0079] According to this embodiment, the composite image can be displayed on a display device carried by the worker, so that the worker can perform the sorting work more efficiently while looking at the composite image.
[0080] [Embodiment 3] 6A is a block diagram showing an example of a configuration of a trained model generation device 30 according to the present embodiment. As shown in FIG. 6A, the device 30 includes a teacher data acquisition unit 31 and a trained model generation unit 32.
[0081] The device 30 may be, for example, a single device including the above-mentioned units, or a device to which the above-mentioned units can be connected via the communication line network. The device 30 may be connected to the above-mentioned external device via the communication line network. The device 30 may be, for example, incorporated in a server as a system. The device 30 may be, for example, a personal computer (PC, for example, desktop type or notebook type) in which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the device 30 may be in the form of cloud computing or edge computing, for example, in which at least one of the above-mentioned units is on a server and the other units are on a terminal.
[0082] 6(B) illustrates a block diagram of the hardware configuration of the trained model generation device 30. The trained model generation device 30 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, etc. Unless otherwise specified, the description of each part of the hardware configuration of the trained model generation device 30 can be applied to each part of the hardware configuration of the plant selection work support device 10 shown in FIG. 3.
[0083] The central processing unit 101 functions as, for example, a teacher data acquisition unit 31 and a trained model generation unit 32.
[0084] The memory 102 and storage device 104 may store the trained model generated by the device 30.
[0085] Next, an example of the trained model production method of this embodiment will be described based on the flowchart of Fig. 7. The trained model production method of this embodiment is implemented as follows, for example, using the trained model generation device 30 of Fig. 6. Note that the trained model production method of this embodiment is not limited to the use of the trained model generation device 30 of Fig. 6.
[0086] First, the teacher data acquisition unit 31 acquires teacher data that associates and records an image of all or part of the plant to be worked on, characteristic information indicating the characteristic amount of the plant, and judgment information indicating at least one of the components of the removal target and the selection target (S31). The plant is the same as described above. The characteristic information is information on the characteristics of a specific plant, such as the shape, hue, shade, size, etc. of the plant. The removal target and the selection target are the same as described above.
[0087] Next, the trained model generation unit 32 uses machine learning with the training data to generate a trained model that, when an image containing all or part of the plant is input, recognizes each of the components of the work target group as work targets, and outputs at least one of a judgment result of determining which components are to be removed from the recognized work targets, and a judgment result of determining which components should be retained from each of the components of the work target group as selection targets (S32), and then terminates (END).
[0088] In other words, the device 30 can be said to construct (generate) a neural network that receives the image as an input and outputs the recognition and judgment results by performing the machine learning. The neural network may be, for example, the same as described above. The neural network may include, for example, an input layer that receives the image, an output layer that outputs the recognition and judgment results, and at least one intermediate layer provided between the input layer and the output layer.
[0089] The input layer, for example, includes a plurality of neurons that receive input of pixel values of each pixel included in the image, and passes the input image values to the intermediate layer. The intermediate layer includes a plurality of neurons that extract features of the plant, and passes the extracted features to the output layer. The output layer includes one or more neurons that recognize each component as a work object, and outputs at least one of an image determined to be the removal object and an image determined to be the selection object, and recognizes the work object and makes each of the judgments based on the features output from the intermediate layer.
[0090] The teacher data acquisition unit 31 may further acquire teacher data recorded in association with evaluation information indicating an evaluation value of each of the components. The evaluation value may be, for example, the same as described above. The trained model generation unit 32 may generate a trained model that recognizes each of the components of the work target group as a work target, assigns an evaluation value to each component of the recognized work target, and outputs at least one of the judgment results based on the evaluation value, when an image including all or a part of the plant is input, by machine learning using the teacher data.
[0091] The trained model generated by this embodiment is used, for example, in the plant selection work support device 30 described in the above-mentioned embodiments 1 and 2 and modified example 1. As a result, this embodiment supports the selection work by the worker, enabling efficient work.
[0092] [Embodiment 4] An example in which the present invention is applied to thinning young fruits will be described with reference to FIG.
[0093] In this example, the worker uses a glasses-type wearable terminal including an imaging unit 23 (imaging device 108) as the display device 20. FIG. 8 shows an example of each image displayed on the display of the display device 20. The worker follows the instructions displayed on the display of the display device 20 and captures an image of one fruit by the imaging unit 23 via the input device 105. At this time, the worker may use a glove to capture images of multiple young fruits of the fruit, arranged along the glove, as shown in FIG. 8(1). The display device 20 outputs the captured image to the plant selection work support device 10. The captured image and an image showing the viewpoint from the imaging unit 23 before capturing the image may be displayed on the display of the display device 20.
[0094] Next, in the plant sorting work support device 10, the image acquisition unit 11 acquires the image, and the work object recognition unit 12 recognizes the young fruits. At this time, for example, as described above, the composite image generation unit 14 may generate a composite image in which the recognized young fruits in the image are superimposed with the visual information for recognition. Then, the plant sorting work support device 10 may transmit the composite image to the display device 20. The display device 20 may acquire the composite image and display it on a display as shown in FIG. 8(2). This allows the worker to know whether the young fruits have been recognized. Note that in FIG. 8(2), the visual information for recognition is an example displayed on the display as a black frame, but is depicted as a solid line frame in the drawing.
[0095] Next, in the plant sorting work support device 10, the removal target determination unit 13 determines the removal target, and the composite image generation unit 14 generates a composite image on which the visual information for the removal target is superimposed. Then, the plant sorting work support device 10 may transmit the composite image to the display device 20. The display device 20 may acquire the composite image and display it on a display as shown in FIG. 8(3). This allows the worker to recognize the young fruits to be thinned. In FIG. 8(3), the visual information for the removal target is an example displayed on the display as a blue frame, but is depicted as a dashed frame in the drawing. In addition, in the composite image, the visual information for recognition may be superimposed on the young fruits on which the recognition information for the removal target is not superimposed, as shown in FIG. 8(3). That is, the display device 20 can know both that the young fruits have been recognized and that the young fruits should be thinned by only acquiring the image shown in FIG. 8(3) without acquiring the image shown in FIG. 8(2).
[0096] Then, the worker checks the composite image and picks the young fruits on which the visual information for removal is superimposed. When the criterion in the transition determination unit 15, which is a later process, is the number of components in the post-removal image, the display on which the composite image is displayed may display a message such as "Please pick the young fruits within the dashed frame (blue frame) until there are 3 fruits or less remaining." When the criterion in the transition determination unit 15 is the presence or absence of an auxiliary tool (gloves) in the post-removal image, the display may display a message such as "Please pick the young fruits within the dashed frame (blue frame) until there are 3 fruits or less remaining. After picking, hold up gloves."
[0097] After the fruit thinning, the worker follows the instructions displayed on the display of the display device 20 to image the pod from which some of the young fruits have been thinned, using the imaging unit 23 via the input device 105. At this time, the worker may use gloves to image a number of young fruits in the pod, arranging them along the glove, as shown in FIG. 8(4). The display device 20 outputs the imaged image (post-removal image) to the plant selection work support device 10. The post-removal image and an image showing the viewpoint from the imaging unit 23 before imaging, etc., may be displayed on the display of the display device 20.
[0098] Next, in the plant sorting work support device 10, the image acquisition unit 11 acquires the post-removal image, and the transition determination unit 15 performs transition determination. When the transition determination unit 15 determines that the sorting target determination is not to be performed, the image generation and capture shown in FIG. 8(3)-(4) are repeated, for example. On the other hand, when the transition determination unit 15 determines that the sorting target determination is to be performed, in the plant sorting work support device 10, the sorting target determination unit 16 determines the sorting target, and the composite image generation unit 14 generates a composite image on which visual information for the sorting target is superimposed. Then, the plant sorting work support device 10 may transmit the composite image to the display device 20. The display device 20 may acquire the composite image and display it on a display, as shown in FIG. 8(5). Note that, in FIG. 8(5), the visual information for the sorting target is an example displayed on the display as a red frame, but is depicted as a solid frame in the drawing. In this way, the worker can recognize which young fruits should be removed and which young fruits should be left. In addition, in the composite image, the visual information for removal may be superimposed on the young fruits on which the recognition information for the selection target is not superimposed.
[0099] In this way, the plant sorting work support device 10 can improve the accuracy of determining which fruits are to be selected by first determining which fruits with many young fruits are to be removed and then determining which fruits have had some of the young fruits removed as to be selected.
[0100] [Embodiment 5] The program of the present embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, the "procedure" may be read as "processing." The program of the present embodiment may be recorded, for example, in a computer-readable recording medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), and an optical disk.
[0101] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.
[0102] <Additional Notes> Some or all of the above embodiments may be described as follows, but are not limited to the following supplementary notes. (Appendix 1) The apparatus includes an image acquisition unit, a work object recognition unit, a removal object determination unit, a composite image generation unit, a transition determination unit, and a selection object determination unit, The image acquisition unit acquires an image of all or a part of a plant that is a work target, The plant has a work target group formed of a plurality of components, The work object recognition unit recognizes each component of the work object group as a work object in the acquired image, The removal target determination unit determines the components to be removed from the recognized work object, the composite image generating unit generates a composite image by superimposing visual information indicating the component determined to be the removal target on the image; the image acquisition unit acquires an image of the whole or part of the plant from which the component to be removed has been removed as a post-removal image; The transition determination unit determines whether or not to perform a selection object determination in the selection object determination unit based on a predetermined criterion in the post-removal image, When the transition determination unit determines that the selection target determination is to be performed, the selection target determination unit determines, in the post-removal image, the components to be left from the components of the work target group remaining after the removal, as selection targets; The composite image generating unit generates a composite image by superimposing visual information indicating the component determined to be the selection target on the image. A device to assist in plant selection work. (Appendix 2) The plant selection work support device according to claim 1, wherein the predetermined criteria in the transition determination unit are at least one of the number of the components in the post-removal image and the presence or absence of an auxiliary tool in the post-removal image. (Appendix 3) At least one of the work target recognition unit, the removal target determination unit, and the selection target determination unit executes each process using a trained model; The work object recognition unit inputs the acquired image into a trained model trained to recognize each component of the work object group as a work object from within the input image, and recognizes each component of the work object group as a work object; The removal target determination unit inputs the image in which a work object is recognized into a trained model trained to determine the components to be removed from the work object recognized in the input image, and determines the components to be removed from the recognized work object; The plant selection work support device described in Appendix 1 or 2, wherein the selection target determination unit inputs the post-removal image into a trained model trained to determine the components to be left from the components of the work target group in the input image as selection targets, and determines the components to be left from the components of the work target group remaining after the removal as selection targets. (Appendix 4) a trained model used by at least one of the removal target determination unit and the selection target determination unit is a trained model trained to make each of the determinations based on an evaluation value; 4. The plant selection work support device according to claim 3, wherein at least one of the removal target determination unit and the selection target determination unit makes the respective determinations based on evaluation values. (Appendix 5) A plant selection work support device as described in any of Appendices 1 to 4, wherein the composite image generation unit defines display parameters for each result of the judgment, and generates a composite image in which each piece of visual information is superimposed on the image in accordance with the definition of the display parameters. (Appendix 6) A plant sorting work support device as described in any of Appendices 1 to 5, wherein at least one of the image and the post-removal image acquired by the image acquisition unit is an image in which the multiple components are aligned along an auxiliary tool using the auxiliary tool. (Appendix 7) A plant selection work support device as described in Appendix 6, wherein the color of the surface of the auxiliary tool that comes into contact with the multiple components is monochrome. (Appendix 8) A plant selection work support device as described in any of Appendices 1 to 7, wherein the composite image generation unit further generates a composite image in which visual information for recognition indicating the components recognized by the work object recognition unit is superimposed on the image. (Appendix 9) A plant selection work support device according to any one of claims 1 to 8, and a display device; The display device is a device that can be carried by an operator, In the plant selection work support device, Each composite image generated by the composite image generation unit is output to the display device; The display device includes an acquisition unit and a display processing unit, The acquisition unit acquires the composite image output from the plant selection work supporting device, The display processing unit executes a process of displaying the composite image, and is a plant selection work supporting composite image display system device. (Appendix 10) 10. The plant selection work support synthetic image display system according to claim 9, wherein the display device is a device that can be worn on a worker's head. (Appendix 11) The present invention includes a training data acquisition unit and a trained model generation unit, the teacher data acquisition unit acquires teacher data in which an image of all or a part of a plant to be worked on, feature information indicating a feature amount of the plant, and judgment information indicating at least one of components to be removed and components to be selected are associated with each other and recorded; The plant has a work target group formed of a plurality of components, The trained model generation unit generates a trained model through machine learning using the training data, when an image containing all or part of the plant is input, and outputs at least one of a determination result of determining which components of the work target group are to be removed from the recognized work targets, and a determination result of determining which components should be left from each component of the work target group are to be selected. (Appendix 12) the teacher data acquisition unit acquires teacher data in which an image of all or a part of a plant to be worked on, feature information indicating a feature amount of the plant, judgment information indicating at least one of components to be removed and components to be selected, and evaluation information indicating an evaluation value of each of the components are associated with each other and recorded; The trained model generation device according to claim 11, wherein the trained model generation unit, when an image containing all or part of the plant is input, generates a trained model by machine learning using the teacher data, recognizing each of the components of the work target group as a work target, assigning an evaluation value to each component of the recognized work target, and outputting at least one of a determination result of determining the components to be removed from the work target based on the evaluation value, and a determination result of determining the components to be left from each of the components of the work target group as selection targets. (Appendix 13) The method includes an image acquisition step, a work object recognition step, a removal object determination step, a composite image generation step, a transition determination step, and a selection object determination step, The image acquisition step acquires an image of all or a part of a plant to be worked on, The plant has a work target group formed of a plurality of components, The work object recognition step recognizes each component of the work object group as a work object in the acquired image, The removal target determination step determines the components to be removed in the recognized work object, the composite image generating step generates a composite image by superimposing visual information indicating the component determined to be the removal target on the image; The image acquisition step includes acquiring an image of all or a part of the plant from which the component to be removed has been removed as a post-removal image, The transition determination step determines whether or not to perform a selection target determination in the selection target determination step based on a predetermined criterion in the post-removal image, When the determination in the transition determination step is that the selection target determination is to be performed, the selection target determination step determines, in the post-removal image, the components to be left from the components of the work target group remaining after the removal, as selection targets; The composite image generating step generates a composite image by superimposing visual information indicating the component determined to be the selection target on the image. A method for assisting in plant selection work. (Appendix 14) A plant selection work support method as described in Appendix 13, wherein the predetermined criterion in the transition determination process is at least one of the number of the components in the post-removal image and the presence or absence of an auxiliary tool in the post-removal image. (Appendix 15) At least one of the work target recognition step, the removal target determination step, and the selection target determination step executes each process using a trained model; The work object recognition step inputs the acquired image into a trained model trained to recognize each component of a work object group as a work object from within an input image, and recognizes each component of the work object group as a work object; The removal target determination step includes inputting the image in which a work object is recognized into a trained model trained to determine the component to be removed from the work object recognized in the input image, and determining the component to be removed from the recognized work object; The plant selection work support method according to claim 13 or 14, wherein the selection target determination step inputs the post-removal image into a trained model trained to determine, as selection targets, the components to be left from the components of the work target group in the input image, and determines, as selection targets, the components to be left from the components of the work target group remaining after the removal. (Appendix 16) A trained model used in at least one of the removal target determination step and the selection target determination step is a trained model trained to make each of the determinations based on an evaluation value, 16. The plant selection work support method according to claim 15, wherein at least one of the removal target determination step and the selection target determination step performs the determination based on an evaluation value. (Appendix 17) A plant selection work support method according to any one of Appendices 13 to 16, wherein the composite image generation step defines display parameters for each of the judgment results, and generates a composite image in which each of the visual information is superimposed on the image in accordance with the definition of the display parameters. (Appendix 18) A plant selection work support method according to any one of Appendices 13 to 17, wherein at least one of the image acquired in the image acquisition step and the post-removal image is an image in which the plurality of components are aligned along an auxiliary tool using the auxiliary tool. (Appendix 19) The plant selection work support method according to claim 18, wherein the surface of the auxiliary tool that comes into contact with the plurality of components is monochrome. (Appendix 20) A plant selection work support method according to any one of Appendices 13 to 19, wherein the composite image generation process further generates a composite image in which visual information for recognition indicating the components recognized by the work object recognition process is superimposed on the image. (Appendix 21) The plant selection work support method according to any one of appendices 13 to 20 and a display method, the display method is a process for a device that is portable by a worker, In the plant selection work support method, Each composite image generated in the composite image generating step is output to a device portable by the worker, The display method includes an acquisition step and a display processing step, The acquiring step acquires the composite image output by the plant selection work support method, The display processing step executes a process for displaying the composite image. (Appendix 22) 22. The plant selection work support composite image display method according to claim 21, wherein the device portable by the worker is a device that can be worn on the worker's head. (Appendix 23) The method includes a teacher data acquisition step and a trained model generation step. The teacher data acquisition step acquires teacher data in which an image of all or a part of a plant to be worked on, characteristic information indicating a characteristic amount of the plant, and judgment information indicating at least one of components to be removed and components to be selected are recorded in association with each other; The plant has a work target group formed of a plurality of components, The trained model generation process is a trained model production method in which, when an image containing all or part of the plant is input, a trained model is generated by machine learning using the training data, which recognizes each component of the work target group as a work target and outputs at least one of a determination result of determining the components to be removed from the recognized work targets and a determination result of determining the components to be left from each component of the work target group as selection targets. (Appendix 24) The teacher data acquisition step acquires teacher data in which an image of all or a part of a plant to be worked on, characteristic information indicating a characteristic amount of the plant, judgment information indicating at least one of components to be removed and components to be selected, and evaluation information indicating an evaluation value of each of the components are associated with each other and recorded; The trained model generation step is a trained model production method as described in Appendix 23, in which, when an image containing all or part of the plant is input, by machine learning using the training data, a trained model is generated in which each of the components of the work target group is recognized as a work target, an evaluation value is assigned to each component of the recognized work target, and a determination result is generated based on the evaluation value, the determination being made as to which of the components to be removed from the work target, and the determination being made as to which of the components to be left from each of the components of the work target group are to be selected. (Appendix 25) A program for causing a computer to execute procedures including an image acquisition procedure, a work target recognition procedure, a removal target determination procedure, a composite image generation procedure, a transition determination procedure, and a selection target determination procedure: The image acquisition step acquires an image of all or a part of a plant to be worked on, The plant has a work target group formed of a plurality of components, The work object recognition step recognizes each component of the work object group as a work object in the acquired image, The removal target determination step includes determining, in the recognized work object, the component to be removed; the synthetic image generating step includes generating a synthetic image by superimposing visual information indicating the component determined to be the removal target on the image; the image acquisition step includes acquiring an image of all or a part of the plant from which the component to be removed has been removed as a post-removal image; the transition determination step determines whether or not to perform a selection target determination in the selection target determination step based on a predetermined criterion in the post-removal image; When the determination in the transition determination procedure is that the selection target determination is to be performed, the selection target determination procedure determines, in the post-removal image, the components to be left from the components of the work target group remaining after the removal as selection targets; The composite image generating step generates a composite image by superimposing visual information indicating the component determined to be the selection target on the image. (Appendix 26) 26. The program of claim 25, wherein the predetermined criterion in the transition determination procedure is at least one of the number of the components in the post-removal image and the presence or absence of an auxiliary tool in the post-removal image. (Appendix 27) At least one of the work object recognition step, the removal object determination step, and the selection object determination step is performed using a trained model; The work object recognition step inputs the acquired image into a trained model trained to recognize each component of a work object group as a work object from within an input image, and recognizes each component of the work object group as a work object; The removal target determination step includes inputting the image in which a work object is recognized into a trained model trained to determine the component to be removed from the work object recognized in the input image, and determining the component to be removed from the recognized work object; The program described in Appendix 25 or 26, wherein the selection target determination step inputs the post-removal image into a trained model trained to determine, as selection targets, the components to be left from the components of the work target group in the input image, and determines, as selection targets, the components to be left from the components of the work target group remaining after the removal. (Appendix 28) a trained model used in at least one of the removal target determination procedure and the selection target determination procedure is a trained model trained to make each of the determinations based on an evaluation value; 28. The program according to claim 27, wherein at least one of the removal target determination step and the selection target determination step performs the respective determinations based on an evaluation value. (Appendix 29) A program described in any of Appendices 25 to 28, wherein the composite image generation procedure defines display parameters for each result of the judgment, and generates a composite image in which each visual information is superimposed on the image in accordance with the definition of the display parameters. (Appendix 30) 30. The program according to any one of appendices 25 to 29, wherein at least one of the image and the post-removal image acquired by the image acquisition procedure is an image in which the multiple components are aligned along an auxiliary tool using the auxiliary tool. (Appendix 31) The program of claim 30, wherein the surface of the assistive device that contacts the plurality of components is a single color. (Appendix 32) A program described in any of Appendices 25 to 31, wherein the composite image generation procedure further generates a composite image in which visual information for recognition indicating the component recognized by the work object recognition procedure is superimposed on the image. (Appendix 33) A plant selection work support program according to any one of appendices 25 to 32, and a display program; the display program is a program in a device that is portable by the worker, In the plant selection work support program, Each composite image generated by the composite image generating procedure is output to a device portable by the worker; The display program is a program for causing a computer to execute steps including an acquisition step and a display processing step: the obtaining step obtains the composite image output by the plant selection work support method, The display processing step executes a process of displaying the composite image. (Appendix 34) 34. The program of claim 33, wherein the device portable by the worker is a device wearable on the worker's head. (Appendix 35) A program for causing a computer to execute a procedure including a procedure for obtaining training data and a procedure for generating a trained model: The teacher data acquisition step includes acquiring teacher data in which an image of all or a part of a plant to be worked on, characteristic information indicating a characteristic amount of the plant, and judgment information indicating at least one of components to be removed and components to be selected are recorded in association with each other; The plant has a work target group formed of a plurality of components, The trained model generation procedure generates a trained model that, when an image containing all or part of the plant is input, recognizes each component of the work target group as a work target, and outputs at least one of a determination result of determining the components to be removed from the recognized work targets, and a determination result of determining the components to be retained from each component of the work target group as selection targets, through machine learning using the training data. (Appendix 36) The teacher data acquisition step includes acquiring teacher data in which an image of all or a part of a plant to be worked on, characteristic information indicating a characteristic amount of the plant, judgment information indicating at least one of components to be removed and components to be selected, and evaluation information indicating an evaluation value of each of the components are associated with each other and recorded; The trained model generation procedure is a program described in Appendix 35, which generates a trained model by machine learning using the training data, when an image containing all or part of the plant is input, recognizing each of the components of the work target group as a work target, assigning an evaluation value to each component of the recognized work target, and outputting at least one of a determination result of determining the components to be removed from the work target based on the evaluation value, and a determination result of determining the components to be retained from each component of the work target group as selection targets. (Appendix 37) A computer-readable recording medium having recorded thereon a program according to any one of appendices 25 to 36. [Industrial Applicability]
[0103] INDUSTRIAL APPLICABILITY The present invention supports sorting work and enables more efficient sorting work, and is therefore useful in work such as fruit thinning, thinning, and pruning. [Explanation of symbols]
[0104] 10 Plant selection support device 11 Image acquisition section 12 Work object recognition unit 13 Removal target determination section 14. Composite image generation unit 15 Transition Judgment Section 16 Selection Object Judgment Section 17 Memory section 18 Imaging section 20 Display device 21 Acquisition Department 22 Display processing section 23 Imaging unit 30 Trained model generator 31 Teacher Data Acquisition Department 32 Trained model generation unit 100 Plant selection work support synthetic image display system device 101 Central Processing Unit 102 Memory 103 Bus 104 Storage device 105 Input Device 106 Display device 107 Communication Devices 108 Imaging device
Claims
1. The apparatus includes an image acquisition unit, a work object recognition unit, a removal object determination unit, a composite image generation unit, a transition determination unit, and a selection object determination unit, The image acquisition unit acquires an image of all or a part of a plant that is a work target, The plant has a work target group formed of a plurality of components, The work object recognition unit recognizes each component of the work object group as a work object in the acquired image, The removal target determination unit determines the components to be removed in the recognized work object, the composite image generating unit generates a composite image by superimposing visual information indicating the component determined to be the removal target on the image; the image acquisition unit acquires an image of the whole or part of the plant from which the component to be removed has been removed as a post-removal image; The transition determination unit determines whether or not to perform a selection object determination in the selection object determination unit based on a predetermined criterion in the post-removal image, When the transition determination unit determines that the selection target determination is to be performed, the selection target determination unit determines, in the post-removal image, the components to be left from the components of the operation target group remaining after the removal, as selection targets; The composite image generating unit generates a composite image by superimposing visual information indicating the component determined to be the selection target on the image. A device to assist in plant selection work.
2. At least one of the work target recognition unit, the removal target determination unit, and the selection target determination unit executes each process using a trained model; The work object recognition unit inputs the acquired image into a trained model trained to recognize each component of the work object group as a work object from within the input image, and recognizes each component of the work object group as a work object; The removal target determination unit inputs the image in which a work object is recognized into a trained model trained to determine the components to be removed from the work object recognized in the input image, and determines the components to be removed from the recognized work object; The plant selection work support device of claim 1, wherein the selection target determination unit inputs the post-removal image into a trained model trained to determine the components to be left as selection targets from the components of the work target group in the input image, and determines the components to be left as selection targets from the components of the work target group remaining after the removal.
3. A plant selection operation support device according to claim 1 or 2, and a display device, The display device is a device that can be carried by an operator, In the plant selection work support device, Each composite image generated by the composite image generation unit is output to the display device; The display device includes an acquisition unit and a display processing unit, The acquisition unit acquires the composite image output from the plant selection work supporting device, The display processing unit executes a process of displaying the composite image, and is a plant selection work supporting composite image display system device.
4. The present invention includes a training data acquisition unit and a trained model generation unit, the teacher data acquisition unit acquires teacher data in which an image of all or a part of a plant to be worked on, feature information indicating a feature amount of the plant, and judgment information indicating at least one of components to be removed and components to be selected are recorded in association with each other; The plant has a work target group formed of a plurality of components, The trained model generation unit generates a trained model through machine learning using the training data, when an image containing all or part of the plant is input, and outputs at least one of a determination result of determining which components of the work target group are to be removed from the recognized work targets, and a determination result of determining which components should be left from each component of the work target group are to be selected.
5. The method includes an image acquisition step, a work object recognition step, a removal object determination step, a composite image generation step, a transition determination step, and a selection object determination step, The image acquisition step acquires an image of all or a part of a plant to be worked on, The plant has a work target group formed of a plurality of components, The work object recognition step recognizes each component of the work object group as a work object in the acquired image, The removal target determination step determines the components to be removed in the recognized work object, the composite image generating step generates a composite image by superimposing visual information indicating the component determined to be the removal target on the image; The image acquisition step includes acquiring an image of all or a part of the plant from which the component to be removed has been removed as a post-removal image, The transition determination step determines whether or not to perform a selection target determination in the selection target determination step based on a predetermined criterion in the post-removal image, When the determination in the transition determination step is that the selection target determination is to be performed, the selection target determination step determines, in the post-removal image, the components to be left from the components of the work target group remaining after the removal, as selection targets; The composite image generating step generates a composite image by superimposing visual information indicating the component determined to be the selection target on the image. A method for assisting in plant selection work.
6. The plant selection operation support method according to claim 5 and a display method are included, the display method is a process for a device that is portable by a worker, In the plant selection work support method, Each composite image generated in the composite image generating step is output to a device portable by the worker, The display method includes an acquisition step and a display processing step, The acquiring step acquires the composite image output by the plant selection work support method, The display processing step executes a process for displaying the composite image.
7. The method includes a teacher data acquisition step and a trained model generation step. The teacher data acquisition step acquires teacher data in which an image of all or a part of a plant to be worked on, characteristic information indicating a characteristic amount of the plant, and judgment information indicating at least one of components to be removed and components to be selected are recorded in association with each other; The plant has a work target group formed of a plurality of components, The trained model generation process is a trained model production method in which, when an image containing all or part of the plant is input, a trained model is generated by machine learning using the training data, which recognizes each component of the work target group as a work target and outputs at least one of a determination result of determining the components to be removed from the recognized work targets and a determination result of determining the components to be left from each component of the work target group as selection targets.
8. A program for causing a computer to execute procedures including an image acquisition procedure, a work target recognition procedure, a removal target determination procedure, a composite image generation procedure, a transition determination procedure, and a selection target determination procedure: The image acquisition step acquires an image of all or a part of a plant to be worked on, The plant has a work target group formed of a plurality of components, The work object recognition step recognizes each component of the work object group as a work object in the acquired image, The removal target determination step includes determining, in the recognized work object, the component to be removed; the synthetic image generating step includes generating a synthetic image by superimposing visual information indicating the component determined to be the removal target on the image; the image acquisition step includes acquiring an image of all or a part of the plant from which the component to be removed has been removed as a post-removal image; the transition determination step determines whether or not to perform a selection target determination in the selection target determination step based on a predetermined criterion in the post-removal image; When the determination in the transition determination procedure is that the selection target determination is to be performed, the selection target determination procedure determines, in the post-removal image, the components to be left from the components of the work target group remaining after the removal as selection targets; The composite image generating step generates a composite image by superimposing visual information indicating the component determined to be the selection target on the image.
9. A program for supporting a plant selection operation according to claim 8 and a program for display, the display program is a program in a device that is portable by the worker, In the plant selection work support program, Each composite image generated by the composite image generating procedure is output to a device portable by the worker; The display program is a program for causing a computer to execute procedures including an acquisition procedure and a display processing procedure: the acquiring step acquires the composite image output by the plant selection task support program; The display processing step executes a process of displaying the composite image.
10. A program for causing a computer to execute a procedure including a procedure for acquiring training data and a procedure for generating a trained model: The teacher data acquisition step includes acquiring teacher data in which an image of all or a part of a plant to be worked on, characteristic information indicating a characteristic amount of the plant, and judgment information indicating at least one of components to be removed and components to be selected are recorded in association with each other; The plant has a work target group formed of a plurality of components, The trained model generation procedure generates a trained model that, when an image containing all or part of the plant is input, recognizes each component of the work target group as a work target, and outputs at least one of a determination result of determining the components to be removed from the recognized work targets, and a determination result of determining the components to be retained from each component of the work target group as selection targets, through machine learning using the training data.
Citation Information
Patent Citations
Fruit picking tool
JP2016021936A
Skill learning support system, skill learning support method and program
JP2016194564A
Robot arm control system
JP2021133445A
Cited By
Agricultural support system, method, and program
JP7668075B1
Agricultural support system, method, and program
WO2026176637A1
