Discrimination device and removal system

Abstract

To provide a discrimination device capable of improving the accuracy of discriminating between harvested products and foreign objects by updating a trained model. [Solution] The discrimination device (1) includes a camera (10) that captures an image of an object, and a discrimination unit (43) that discriminates whether the object is a harvested product or a foreign object by inputting the image of the object captured by the camera (10) into a trained model (52). The trained model (52) is additionally trained using additional training data (54) that includes the image of the object and a discrimination result obtained by an operator as to whether the object is a harvested product or a foreign object.

Description

[Technical Field]

[0001] The present invention relates to a discrimination device and a removal system. [Background technology]

[0002] In recent years, the automation and efficiency of agricultural work has been promoted by introducing AI (Artificial Intelligence) technology into agricultural work. For example, in the removal device disclosed in Patent Document 1, image data of an image of a transported object taken by a camera is input into a machine-learned learning model, which enables the device to distinguish between harvested products and foreign objects and remove the clods of soil that are foreign objects. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-137755 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the removal device in Patent Document 1 does not disclose how to train the learning model with additional training data, and there is a problem in that the accuracy of distinguishing between harvested products and foreign objects cannot be improved by updating the learning model.

[0005] An object of one aspect of the present invention is to provide a discrimination device and a removal system that are capable of updating a trained model to improve discrimination accuracy. [Means for solving the problem]

[0006] In order to solve the above problem, a discrimination device according to one aspect of the present invention includes an imaging unit that captures an image of an object, and a discrimination unit that inputs the image of the object captured by the imaging unit into a trained model to discriminate whether the object is a harvested product or a foreign object other than the harvested product. The trained model is trained to input the image of the object captured by the imaging unit and output a discrimination result indicating whether the object is the harvested product or the foreign object, and is additionally trained using additional training data including the image of the object and a discrimination result obtained by an operator as to whether the object is the harvested product or the foreign object. [Effects of the Invention]

[0007] According to one aspect of the present invention, the trained model can be updated to improve the accuracy of distinguishing between harvested products and foreign objects. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing the overall configuration of a removal system including a discrimination device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a schematic cross-sectional view showing the configuration of a removal device of the removal system according to the embodiment. [Figure 3] FIG. 2 is a cross-sectional view showing the configuration of a work machine of the removal system according to the embodiment. [Figure 4] 1 is a flowchart showing an example of the flow of harvesting processing of harvested crops by a work machine according to an embodiment. [Figure 5] 5 is a flowchart showing an example of the flow of a discrimination process performed by the discrimination device of FIG. 4. [Figure 6] 6 is a flowchart showing an example of the flow of the brightness adjustment process in FIG. 5. [Figure 7] 10A and 10B are diagrams for explaining brightness adjustment of an image of an object using a plurality of illuminance reference plates. [Figure 8] 10A and 10B are diagrams illustrating an example of a discrimination result of a discrimination process performed by a discrimination unit of the discrimination device according to the embodiment. [Figure 9]10 is a flowchart showing an example of the flow of a detection process performed by the discrimination device according to the embodiment. [Figure 10] 10 is a flowchart illustrating an example of the flow of a process for updating a trained model used in the discrimination device according to the embodiment. [Figure 11] 10 is a flowchart showing the flow of a detection process performed by a discrimination device according to a modified example. [Figure 12] 10 is a top view showing a state in which an operator's hand is placed on the carry-in conveyor of the discrimination device according to the embodiment. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] A removal system 100 including a discrimination device 1 according to an embodiment of the present invention will be described below with reference to FIGS.

[0010] <Outline of removal system configuration> Fig. 1 is a block diagram showing the overall configuration of a removal system 100 including a discrimination device 1. Fig. 2 is a schematic cross-sectional view showing the configuration of a removal device 2 of the removal system 100. Fig. 3 is a cross-sectional view showing the configuration of a work machine 3 of the removal system 100.

[0011] As shown in Figures 1 to 3, the removal system 100 includes a discrimination device 1, a removal device 2, and a work machine 3. The discrimination device 1 is attached to the work machine 3 and is a device for discriminating foreign matter F that has become mixed into the harvested product C harvested by the work machine 3.

[0012] The removal device 2 is a device for removing foreign matter F based on the discrimination result of the discrimination device 1. The work machine 3 is a device for harvesting crop C while moving through a field G or the like.

[0013] [Configuration of discrimination device] As shown in FIG. 1, the discrimination device 1 includes a camera 10, an input unit 20, a brightness sensor 30, an environmental sensor 30A, a main control unit 40, a memory unit 50, a display unit 60, an output unit 70, and artificial lighting 80.

[0014] The camera 10 is an example of an imaging unit, and captures an image of an object carried in by the carry-in conveyor 24. The camera 10 is also an example of a sensor that detects the size of the imaged object. The camera 10 is, for example, an RGB camera. The camera 10 may also be an infrared camera, an X-ray camera, or the like. In this case, it is possible to improve the accuracy of object detection by the camera 10, for example, at night.

[0015] The camera 10 is attached to the work machine 3, for example, via a mounting member (not shown), and is disposed above the carry-in conveyor 24 of the removal device 2, as shown in Fig. 2. The camera 10 may also be attached directly to a member of the work machine 3.

[0016] The camera 10 has an adjustment unit 11. The adjustment unit 11 adjusts the imaging conditions of the object in response to changes in the lighting environment so that the image condition of the object falls within a range suitable for discrimination by the discrimination unit 43. The image condition is, for example, the brightness of the image. Note that the image condition may also include image quality. The imaging conditions include, for example, the exposure and gain of the camera 10.

[0017] Furthermore, as will be described later, the camera 10 has a function of selecting an illuminance reference plate 12 to be used as a reference from among a plurality of illuminance reference plates 12 according to the brightness of the lighting environment. The adjustment unit 11 adjusts the brightness of the image of the object to be appropriate, using the brightness of the selected illuminance reference plate 12 in the captured image as a reference.

[0018] The target objects include a harvested product C harvested by a work machine 3 in a farm field G, and foreign matter F. The harvested product C is, for example, a potato. Note that the harvested product C is not limited to a potato, and may also be other agricultural products such as sweet potatoes, taro, onions, pumpkins, watermelons, and cabbages.

[0019] Foreign matter F is something different from the harvested product C, such as a lump of earth or a stone. Note that foreign matter F may also include harvested product C that does not meet predetermined conditions. Examples of harvested product C that does not meet predetermined conditions include agricultural produce that does not meet shipping conditions because it is discolored, deformed, or missing.

[0020] The input unit 20 is a device that allows an operator to perform various operations related to the discrimination apparatus 1. The input unit 20 is configured to have, for example, a keyboard, a mouse, etc. The input unit 20 may also be configured to have a touch panel and a GUI (Graphical User Interface).

[0021] The brightness sensor 30 is, for example, an illuminance sensor, and detects the brightness of the environment surrounding the camera 10, i.e., the lighting environment in which the worker performs the discrimination work. The lighting environment refers to the lighting conditions in the workplace where the worker performs the work. The lighting environment includes direct or indirect natural light and the brightness conditions of artificial lighting 80. The brightness of natural light changes depending on the season, time of day, weather, etc. The artificial lighting 80 emits visible light that is harmless to the worker, and is turned on or off depending on the brightness of the workplace. The environmental sensor 30A is, for example, a temperature and humidity sensor, and detects the temperature and humidity of the field G where the harvest C is harvested.

[0022] The main control unit 40 is configured to include, for example, a CPU (Central Processing Unit), etc. The main control unit 40 includes an acquisition unit 41, a first learning unit 42, a discrimination unit 43, a second learning unit 44, and a control unit 45.

[0023] The main control unit 40 executes various processes such as discrimination processing by executing the control program 51. The main control unit 40 is electrically connected to the camera 10 via a wired or wireless connection. The main control unit 40 acquires image data of an image captured by the camera 10 and outputs a control signal to the camera 10.

[0024] The acquisition unit 41 acquires an image of an object captured by the camera 10. The acquisition unit 41 also acquires learning data 53 input via the input unit 20. The acquisition unit 41 also acquires the temperature and humidity of the field G detected by the environmental sensor 30A. The acquisition unit 41 also acquires meteorological information related to the weather, solar radiation, precipitation, sunshine hours, etc. of the field G via the input unit 20. As the meteorological information, for example, meteorological data provided by the Japan Meteorological Agency is used. The temperature and humidity of the field G and meteorological information are examples of environmental information.

[0025] The harvested product C is, for example, a potato. The harvested product C is not limited to a potato, and may be other agricultural products such as sweet potatoes, taro, onions, pumpkins, watermelons, and cabbages. The foreign matter F is an object different from the harvested product C, such as a lump of earth or a stone. The foreign matter F also includes discolored, deformed, or missing agricultural products.

[0026] The first learning unit 42 generates a trained model 52 by performing machine learning or the like using the training data acquired by the acquisition unit 41. The trained model 52 is trained to receive an input of an image of an object captured by the camera 10 and output whether the object is a harvested product C or a foreign object F.

[0027] The discrimination unit 43 inputs the image of the object captured by the camera 10 into the trained model, thereby discriminating whether the object is a harvested product C or a foreign object F.

[0028] The second learning unit 44 updates the trained model 52 by additionally training the trained model 52 generated by the first learning unit 42 using additional training data 54.

[0029] The control unit 45 controls the display on the display unit 60 based on the discrimination result by the discrimination unit 43. The control unit 45 also controls the driving of the flap 21, discharge conveyor 22, carry-out conveyor 23, and carry-in conveyor 24 of the removal device 2.

[0030] The storage unit 50 is configured to include, for example, a read-only memory (ROM), a random access memory (RAM), etc. The storage unit 50 stores a control program 51, a trained model 52, training data 53, and additional training data 54.

[0031] The control program 51 is a program for causing the main control unit 40 to execute various processes such as discrimination processing. The trained model 52 is a model that has been trained to input environmental information including the temperature and humidity of the field G and meteorological information in addition to an image of the object, and output whether the object is a harvested product C or a foreign object F. As the trained model 52, for example, a Single Shot Detection model can be used.

[0032] The learning data 53 is a plurality of annotated image data including image data indicating that the harvested product C has been captured and image data indicating that the foreign object F has been captured.

[0033] The learning data 53 also includes environmental information, including temperature, humidity, and weather information, for the field G. As a result, if, for example, it rained before the harvest of the crop C and the humidity in the field G is higher than usual, image data showing wet soil, mud, etc. attached to the crop C is learned.

[0034] The additional learning data 54 includes an image of the object and a discrimination result obtained by the operator as to whether the object is a harvested product C or a foreign object F. The additional learning data 54 may also include environmental information about the field G. This makes it easier to manage the additional learning data 54.

[0035] The display unit 60 is, for example, a liquid crystal display, and displays a moving or still image of the object captured by the camera 10. The display unit 60 also displays information such as the discrimination result by the discriminator 43, the positions of the harvested product C and the foreign object F, etc.

[0036] The output unit 70 is controlled by the control unit 45 and outputs sound and light. The output unit 70 has a speaker that emits sound and a light that emits light. The control unit 45 outputs sound and light when the camera 10 detects an object whose size exceeds a reference size.

[0037] The artificial light 80 is, for example, a lamp or a fluorescent light, and is turned on by the control unit 45. The control unit 45 turns on the artificial light 80 in accordance with the brightness of the lighting environment detected by the brightness sensor 30.

[0038] [Configuration of removal device] Next, the configuration of the removal device 2 will be described with reference to Fig. 1 and Fig. 2. As shown in Fig. 1 and Fig. 2, the removal device 2 is electrically connected to the control unit 45 of the main control unit 40. The removal device 2 has a flap 21, a discharge conveyor 22, a carry-out conveyor 23, and a carry-in conveyor 24. The flap 21 is an example of a removal unit.

[0039] The flap 21 is controlled by the control unit 45 and is configured to be movable between a first position P1 and a second position P2. Specifically, when the discrimination unit 43 determines that the target object is harvested product C, the control unit 45 moves the flap 21 to the first position P1. On the other hand, when the discrimination unit 43 determines that the target object is foreign matter F, the control unit 45 moves the flap 21 to the second position P2.

[0040] When the flap 21 is moved to the first position P1, the harvested product C carried in from the work machine 3 by the carry-in conveyor 24 is guided toward the carry-out conveyor 23. The carry-out conveyor 23 carries the harvested product C guided by the flap 21 out to a storage section (not shown). The harvested product C carried out by the carry-out conveyor 23 is stored in the storage section.

[0041] On the other hand, when the flap 21 is moved to the second position P2, the foreign matter F is removed from the carry-in conveyor 24 and is guided toward the discharge conveyor 22. The discharge conveyor 22 discharges the foreign matter F removed by the flap 21 to a discharge section (not shown). The foreign matter F discharged by the discharge conveyor 22 is stored in the discharge section.

[0042] [Work equipment configuration] Next, the configuration of the work machine 3 will be described in detail with reference to Fig. 3. As shown in Fig. 3, the work machine 3 is a device for harvesting crops C in a field G. The work machine 3 has a main body section 31, a traveling section 32, a steering section 33, a harvesting section 34, and a transport section 35. For ease of explanation, the left side of Fig. 3 is defined as the front of the work machine 3, and the right side of Fig. 3 is defined as the rear of the work machine 3.

[0043] The main body 31 travels when driven by the travel section 32. The travel section 32 is configured to have, for example, a pair of travel rollers. The work machine 3 moves within the farm field G by the travel section 32.

[0044] The control unit 33 is provided at one end of the main body 31 and has operating devices such as a handle. An operator operates the control unit 33 to control the operations of the traveling unit 32, the harvesting unit 34, the transport unit 35, etc.

[0045] The harvesting unit 34 digs up the harvested product C from the soil of the field G. The transporting unit 35 is, for example, a belt conveyor, and transports the harvested product C dug up by the harvesting unit 34 to the rear of the work machine 3. The transporting unit 35 is installed in a forward-leaning position so that it is higher towards the rear of the work machine 3. The transporting unit 35 transports the harvested product C diagonally upward and rearward, and transports it to the carry-in conveyor 24.

[0046] The conveying section 35 may be formed with holes smaller than the harvest C. This allows clods of soil and the like smaller than the harvest C to fall onto the field G while the harvest C is being conveyed.

[0047] <Harvesting and processing of harvested crops using machinery> Fig. 4 is a flowchart showing an example of the flow of harvesting processing of the harvest product C by the work machine 3. As shown in Fig. 4, first, in the field G, the worker operates the control unit 33 of the work machine 3 to start harvesting the harvest product C by the harvesting unit 34 of the work machine 3 (step S1).

[0048] When the harvesting of the harvested product C by the work machine 3 is started, step S2, step S3, and step S4 are carried out in parallel. Specifically, the discrimination process by the discrimination device 1 and the discrimination work by the worker to discriminate between the harvested product C and the foreign object F are carried out in cooperation with each other.

[0049] In the above-described cooperative discrimination, it is preferable that, for example, discrimination processing is performed by discrimination device 1 upstream in the conveying direction of carry-in conveyor 24, and discrimination work is performed by an operator downstream in the conveying direction of carry-in conveyor 24. This allows the operator to supplementarily discriminate objects that could not be discriminated by discrimination device 1. In this case, safety can be ensured by having the operator perform discrimination work upstream of the most downstream part in the conveying direction of carry-in conveyor 24.

[0050] It should be noted that after a worker performs discrimination upstream in the conveying direction of the carry-in conveyor 24, discrimination processing may be performed by the discrimination device 1 downstream of the carry-in conveyor 24. In this case, the worker is positioned away from the flap 21, so the safety of the worker is ensured.

[0051] In step S2, a discrimination process is performed by the discrimination device 1, which will be described later. In step S2, the discrimination device 1 automatically discriminates whether the object is a harvested product C or a foreign object F by inputting the image of the object captured by the camera 10 into the trained model 52 using the discrimination unit 43.

[0052] In step S3, a worker performs a discrimination task to determine whether the object is a harvested product C or a foreign object F. The discrimination by the worker in step S3 is performed in the same place where the camera 10 captures the image of the object.

[0053] In step S3, the worker performs the task of distinguishing foreign matter F that could not be distinguished by the discrimination device 1 in step S2 while visually checking, for example, the discrimination result by the discrimination device 1 displayed on the display unit 60 (see the lower diagram in FIG. 8). Foreign matter F that is difficult for the discrimination device 1 to distinguish includes agricultural produce that has turned green or rotted and become discolored.

[0054] After step S3, the discrimination device 1 stores the discrimination result by the worker in the memory unit 50 (step S4). In step S4, the discrimination device 1 uses the camera 10 to capture an image of the harvested product C or foreign object F discriminated by the worker, and stores the captured image in the memory unit 50 as additional learning data 54.

[0055] After the harvest of the crop C is completed, the worker performs annotation work on the image stored in step S3 at a location other than the field G. The image data after the annotation work is used as additional training data 54 when updating the trained model 52, which will be described later.

[0056] When steps S2 to S4 are completed, the worker ends the harvesting of the crop C by the work machine 3. It is assumed that the worker who operates the work machine 3 and the worker who performs the discrimination work are different persons.

[0057] Next, the discrimination process by the discrimination device 1 will be described in detail with reference to Fig. 5 to Fig. 9. In the flowchart of Fig. 5, first, the adjustment unit 11 of the camera 10 performs the brightness adjustment process shown in Fig. 6 (step S11).

[0058] [Brightness adjustment processing] Fig. 6 is a flowchart showing an example of the flow of the brightness adjustment process of Fig. 5. In the flowchart shown in Fig. 6, the camera 10 of the discrimination device 1 selects the illuminance reference plate 12 to be used as a reference in accordance with the brightness of the lighting environment (step S21).

[0059] Here, illuminance reference plates 12 of a plurality of colors (see FIG. 7) are arranged within the imaging range in which the object is imaged by the camera 10 near the carry-in conveyor 24. Specifically, although not shown, three illuminance reference plates 12, namely, a white illuminance reference plate 121, a gray illuminance reference plate 122, and a black illuminance reference plate 123, are arranged side by side within the imaging range of the camera 10.

[0060] In step S21, the camera 10 uses ROI (Region of Interest) setting to detect the brightness of a specific region R on the illuminance reference plate 12. As a result, the camera 10 selects the illuminance reference plate 12 that corresponds to the brightness of the lighting environment.

[0061] In addition, when the brightness of the lighting environment meets the standard with natural light alone, the camera 10 may capture an image of the illuminance reference plate 12 in a lighting environment with natural light, whereas when the brightness of the lighting environment does not meet the standard, the camera 10 may capture an image of the illuminance reference plate 12 in a lighting environment with natural light and artificial lighting 80.

[0062] The camera 10 then automatically selects an appropriate illuminance reference plate 12 from the three illuminance reference plates 12, namely, the white illuminance reference plate 121, the gray illuminance reference plate 122, and the black illuminance reference plate 123, so as to respond to changes in the brightness of the lighting environment over time.

[0063] 7 is a diagram illustrating the adjustment of the brightness of an image of an object using a plurality of illuminance reference plates 12. For example, when the brightness of the lighting environment is normal, the camera 10 selects a gray illuminance reference plate 122 as shown in the center diagram of FIG. 7. When the brightness of the lighting environment is brighter than normal, the camera 10 selects a white illuminance reference plate 121 as shown in the upper diagram of FIG. 7, thereby adjusting the brightness of the image of the object to be darker and preventing the image of the object from being blown out.

[0064] On the other hand, when the brightness of the lighting environment is darker than normal, the camera 10 can adjust the brightness of the image of the object to be brighter and prevent the image of the object from being crushed into black by selecting the black illuminance reference plate 123, as shown in the lower diagram of Fig. 7. In this way, by using different colors of the illuminance reference plate 12 depending on the lighting environment, the adjustment unit 11 can perform good brightness adjustment.

[0065] After step S21, camera 10 captures an image of selected illuminance reference plate 12 (step S22). After step S22, adjustment unit 11 of camera 10 uses the brightness in the captured image of selected illuminance reference plate 12 as a reference, and then adjusts the brightness of the image of the object to be appropriate (step S23). In step S23, adjustment unit 11 adjusts the exposure and gain of camera 10 to ensure that the brightness of the image of the object is appropriate.

[0066] After step S23, the control unit 45 determines whether the brightness of the lighting environment satisfies a standard based on the detection result by the brightness sensor 30 (step S24).

[0067] If the brightness of the lighting environment detected by the brightness sensor 30 does not satisfy the criteria (step S24: NO), the control unit 45 turns on the artificial light 80 (step S25). On the other hand, if the brightness of the lighting environment detected by the brightness sensor 30 satisfies the criteria (step S24: YES), or after step S25, the control unit 45 ends the adjustment process by the discrimination device 1 shown in FIG.

[0068] Returning to FIG. 5, after step S11, the discrimination device 1 uses the camera 10 to capture an image of the object carried in by the carry-in conveyor 24 (step S12), as indicated by the arrow X1 in FIG. 2. As described above, the object contains a mixture of harvested product C and foreign matter F. The camera 10 captures an image of the object under a lighting environment using at least one of natural light and artificial lighting 80.

[0069] After step S12, the discrimination device 1 acquires an image of the object captured by the camera 10 and environmental information using the acquisition unit 41 (step S13). After step S13, the discrimination unit 43 of the discrimination device 1 inputs the image of the object and environmental information acquired by the acquisition unit 41 into the trained model 52 (step S14). The image of the object may be a still image or a video.

[0070] After step S14, the discrimination device 1 executes the detection process shown in Fig. 9 (step S15). Note that the discrimination device 1 may execute step S15 before step S14.

[0071] [Detection process] Fig. 9 is a flowchart showing an example of the flow of detection processing by the discrimination device 1. As shown in the flowchart of Fig. 9, the discrimination device 1 detects the size of an object using the camera 10 (step S31). The camera 10 is an example of a sensor that detects the size of an object.

[0072] After step S31, the acquisition unit 41 acquires information about the size of the object detected by the camera 10. Then, the control unit 45 determines whether the size of the object acquired by the acquisition unit 41 exceeds a threshold value (step S32). The threshold value is set to, for example, a size of the object that makes it difficult to guide the object using the flap 21.

[0073] If the size of the object exceeds the threshold value (step S32: YES), the control unit 45 controls the output unit 70 to output sound and light (step S33). This allows the worker to recognize that there is harvested product C or foreign object F that exceeds the threshold size among the objects, and to remove the harvested product C or foreign object F from the carry-in conveyor 24.

[0074] If the size of the object does not exceed the threshold value (step S32: NO), or after step S33, the size detection process shown in FIG. 9 ends.

[0075] Returning to FIG. 5, after step S15, the discrimination unit 43 discriminates whether the target object is the harvested product C or the foreign object F based on the output result of the trained model 52 (step S16).

[0076] Here, Fig. 8 is a diagram showing an example of a discrimination result of the discrimination process by the discrimination unit 43 of the discrimination device 1. The example shown in Fig. 8 shows that the discrimination unit 43 inputs image data of image D1 shown in the upper diagram of Fig. 8 into the trained model 52, thereby obtaining image D2 shown in the lower diagram of Fig. 8 as an output.

[0077] Specifically, the upper diagram in Fig. 8 shows image D1 of an object including harvested products C1 to C4, such as potatoes, and foreign objects F1 to F4, such as clods of soil and stones. The lower diagram in Fig. 8 shows image D2, which is an output result of trained model 52, in which harvested products C1 to C4 are surrounded by bounding boxes B1 to B4, respectively, and foreign objects F1 to F4 are surrounded by bounding boxes B5 to B8, respectively.

[0078] In image D2 of Figure 8, bounding boxes B5 to B8 corresponding to foreign objects F1 to F4 are shown with thicker lines than the bounding boxes B1 to B4. This allows the worker to recognize that foreign objects F1 to F4 are present on the input conveyor 24 by visually checking image D2 displayed on display unit 60. Note that the word "harvested product" may be displayed inside the bounding boxes B1 to B4, and the word "foreign object" may be displayed inside the bounding boxes B5 to B8.

[0079] If the discrimination unit 43 determines that the target object is the harvested product C (step S16: YES), the control unit 45 moves the flap 21 to the first position P1 (step S17) to guide the harvested product C to the discharge conveyor 23. Then, the removal device 2 discharges the harvested product C to the storage unit by the discharge conveyor 23, as shown by arrow X2 in FIG. 2 (step S18).

[0080] On the other hand, if the discrimination unit 43 determines that the object is a foreign object F (step S16: NO), the control unit 45 moves the flap 21 to the second position P2 (step S19) and guides the foreign object F to the discharge conveyor 22, as shown by arrows X3 and X4 in Figure 2.

[0081] Then, the removal device 2 discharges the foreign matter F to the discharge section by the discharge conveyor 22, as shown by arrow X5 in Figure 2 (step S20). In this way, the discrimination device 1 repeatedly executes the processes of steps S11 to S20 described above from the start of harvesting of the harvest C by the work machine 3 until the end of harvesting of the harvest C.

[0082] <Updating a trained model> Next, the update process of the trained model 52 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the flow of the update process of the trained model 52 used in the discrimination device 1.

[0083] After the worker has finished harvesting the crop C using the work machine 3, the worker creates additional learning data 54 using an information processing device having a GPU (Graphics Processing Unit) or the like in a building or the like other than the field G. Specifically, the information processing device acquires the discrimination result by the worker stored in the memory unit 50 in step S3 of Fig. 4 from the memory unit 50 of the discrimination device 1 and stores it in the memory of the information processing device.

[0084] The discrimination results by the worker include, for example, an image of foreign matter F that could not be discriminated by the discrimination device 1, and an image of harvested product C that was erroneously discriminated as a foreign matter in the discrimination process by the discrimination device 1.

[0085] The worker operates the information processing device to perform annotation work, adding annotations as to whether the object is a harvested product or a foreign object to the above-mentioned discrimination results by the worker. Specifically, the worker acquires from the discrimination device 1 images of objects that could not be discriminated by the discrimination device 1, images in which the discrimination process by the discrimination device 1 has been erroneous, etc. Then, the worker uses the information processing device to add annotations to the images of objects that could not be discriminated, or correct annotation images in which the discrimination process by the discrimination device 1 has been erroneous. In this way, by using the image data obtained by the discrimination device 1, it is possible to reduce the time required for the annotation work.

[0086] Next, in the flowchart shown in FIG. 10, the acquisition unit 41 of the discrimination device 1 acquires the additional learning data 54 created by the information processing device described above via the input unit 20 (step S41).

[0087] After step S41, the second learning unit 44 of the discrimination device 1 uses the training data 53 and the additional training data 54 to generate a new trained model 52 and update the trained model 52 (step S42). In step S42, the second learning unit 44 uses the trained model 52 that has already been trained by the first learning unit 42 and updates the trained model 52 by additionally training the additional training data 54.

[0088] After step S42, the second learning unit 44 of the discrimination device 1 stores the updated trained model 52 in the memory unit 50 (step S43). As a result, by using the updated trained model 52 in the next discrimination process by the discrimination device 1, the accuracy of discriminating between harvested product C and foreign object F can be improved.

[0089] [Effects of the embodiment] In the discrimination device 1 of this embodiment described above, an image of an object captured by the camera 10 is input to the trained model 52, and the discrimination unit 43 discriminates whether the object is harvested product C or foreign matter F based on the output result of the trained model 52. This makes it possible to automatically determine whether foreign matter F is mixed into the harvested product C harvested by the work machine 3. Then, based on the discrimination result by the discrimination unit 43 of the discrimination device 1, the removal device 2 can automatically remove the foreign matter F.

[0090] In particular, the trained model 52 described above is updated in step S32 of Fig. 9 using the training data 53 and additional training data 54 including the discrimination result of the discrimination work performed by the operator in step S2 of Fig. 4. In this way, according to the discrimination device 1 of this embodiment, the trained model 52 can be updated to improve the discrimination accuracy between the harvested product C and the foreign object F.

[0091] Furthermore, according to the discrimination device 1 of the above-described embodiment, the discrimination unit 43 discriminates discolored, deformed, or damaged agricultural products that do not meet the shipping conditions as foreign matter F. This allows the removal device 2 to remove agricultural products that do not meet the shipping conditions.

[0092] Furthermore, according to the discrimination device 1 of the above-described embodiment, when the lighting environment of the worker changes, the adjustment unit 11 of the camera 10 adjusts the imaging conditions of the object by the camera 10, so that the state of the image of the object can be kept within a range suitable for discrimination. This makes it possible to prevent a decrease in the discrimination accuracy of the discrimination unit 43 when the lighting environment changes.

[0093] In particular, in the discrimination device 1 of the above-described embodiment, the camera 10 selects an illuminance reference plate 12 to be used as a reference from among a plurality of illuminance reference plates 12 in accordance with the brightness of the lighting environment (step S23 in FIG. 6). Then, the adjustment unit 11 adjusts the exposure and gain of the camera 10 so that the brightness of the image of the object becomes appropriate, using the brightness in the captured image of the selected illuminance reference plate as a reference (step S25 in FIG. 6). As a result, when the brightness of the lighting environment changes, the camera 10 automatically changes the illuminance reference plate 12 to be used as a reference, so that the brightness of the image of the object can be maintained appropriate.

[0094] In the above embodiment, the discrimination work by the worker in step S2 of Fig. 4 is performed in the same location where the camera 10 captures the image of the object. This allows the worker to remove the foreign matter F in cooperation with the discrimination device 1. This reduces the number of workers required for the discrimination work of the foreign matter F. Furthermore, when the discrimination device 1 and the worker perform discrimination on the same object and the discrimination results differ, data can be used as additional learning data 54.

[0095] Furthermore, in the discrimination device 1 of the above-described embodiment, the camera 10 captures images of the object and the illuminance reference plate 12 in a lighting environment for the worker using natural light and / or artificial lighting 80, so there is no need to use a special light source, and the worker can safely perform the discrimination work. Furthermore, since the artificial lighting 80 is turned on according to the brightness of the lighting environment, the worker can continue to discriminate foreign matter F even at night.

[0096] Furthermore, according to the discrimination device 1 of the above-described embodiment, when the worker performs the discrimination operation shown in Fig. 4, the positions of the harvested product C and the foreign matter F, and the discrimination result as to whether it is the harvested product C or the foreign matter F, are displayed on the display unit 60. This makes it easier for the worker to remove the foreign matter F.

[0097] Furthermore, according to the discrimination device 1 of the above-described embodiment, when the camera 10 detects a foreign object F of a size exceeding the standard (step S32 in FIG. 9: YES), the output unit 70 outputs a warning sound or light (step S33 in FIG. 9). This notifies the worker that a foreign object F that cannot be removed by the removal device 2 has been mixed in, and enables the worker to take appropriate measures.

[0098] Furthermore, according to the discrimination device 1 of the above-described embodiment, the discrimination unit 43 discriminates whether the object is a harvested product C or a foreign object F by inputting environmental information, including the temperature and humidity of the field G and meteorological information, in addition to an image of the object, into the trained model 52. This allows the discrimination unit 43 to discriminate whether the object is a harvested product C or a foreign object F with high accuracy, taking into account the temperature and humidity of the field G and meteorological information, including the weather, solar radiation, precipitation, and sunshine hours of the field G.

[0099] Furthermore, the discrimination device 1 of the above-described embodiment can be detachably attached to the work machine 3. Therefore, by retrofitting the discrimination device 1 to an existing work machine 3, it becomes possible to automatically discriminate between foreign objects F with a simple configuration.

[0100] Furthermore, according to the above-described embodiment of the discrimination device 1, the control unit 45 controls the position of the flap 21 of the removal device 2 based on the discrimination result by the discrimination unit 43, thereby automatically separating the destinations of the harvested product C and the foreign matter F.

[0101] The above-described discrimination device 1 can improve the efficiency of crop harvesting work. Such an effect also contributes to achieving, for example, Goal 2 of the Sustainable Development Goals (SDGs) advocated by the United Nations, "End hunger, achieve food security and improved nutrition, and promote sustainable agriculture."

[0102] [Modification] Next, the flow of detection processing by the discrimination device 1 in the modified example will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is a flowchart showing the flow of detection processing by the discrimination device 1 in the modified example. Fig. 12 is a top view showing a state in which a worker's hand H has entered onto the carry-in conveyor 24 of the discrimination device 1.

[0103] In the discrimination device 1 of the modified example, the processing content of the detection process (step S15A) shown in Fig. 11 differs from that of the above embodiment. In the discrimination device 1 of the modified example, the discrimination work of step S3 in Fig. 4 is performed by an operator downstream in the conveying direction of the discharge conveyor 23, and the discrimination process is performed by the discrimination device 1 downstream in the conveying direction of the carry-in conveyor 24 (see Fig. 2).

[0104] 12, if a hand H of a worker near the unloading conveyor 23 accidentally enters the load conveyor 24, the worker may injure his or her hand H. Therefore, in the detection process S15A by the discrimination device 1 in the modified example, when the worker's hand H enters the load conveyor 24, a process is executed to notify the worker of the danger.

[0105] [Detection process] 11 (step S15A), the discrimination unit 43 of the discrimination device 1 determines whether or not the camera 10 has detected the worker's hand H on the carry-in conveyor 24 (step S51). The camera 10 is an example of a sensor that detects the worker's hand H. Note that the camera 10 may detect the size of an object and the worker's hand H, and the control unit 45 may determine whether or not the size of the object exceeds a threshold, and the discrimination unit 43 may determine whether or not the worker's hand H has been detected.

[0106] In step S51, the discrimination unit 43 inputs the image of the carry-in conveyor 23 captured by the camera 10 into the trained model 52 to discriminate whether or not it is a worker's hand H. The trained model 52 is trained to receive an input of the image of the carry-in conveyor 24 captured by the camera 10 and output whether or not it is a worker's hand H.

[0107] 12, when a worker's hand H is detected on the carry-in conveyor 24 (step S51: YES), the control unit 45 controls the output unit 70 to output an alarm such as a warning sound and light for a predetermined time to alert the worker of danger (step S52). The output unit 70 is preferably installed near the carry-out conveyor 23.

[0108] This allows a worker near the discharge conveyor 23 to recognize that his / her hand H has entered onto the carry-in conveyor 24, and to move his / her hand H away from the carry-in conveyor 24 to avoid injury. Note that in step S52, the control unit 45 may stop driving the flap 21 of the removal device 2 and the carry-in conveyor 24.

[0109] If the worker's hand H is not detected on the carry-in conveyor 24 (step S51: NO), or after step S52, the control unit 45 ends the flow of FIG.

[0110] 11, the control unit 45 may increase the volume of the sound output from the output unit 70 and the intensity of the light compared to step 33 in Fig. 9. Furthermore, in step S52 in Fig. 11, the control unit 45 may notify the worker by vibrating a wearable terminal or the like worn by the worker based on an instruction from the discrimination device 1.

[0111] Other Embodiments The discrimination device 1 of the above-described embodiment may be connected to an external device such as a PC via wireless communication. The external device performs additional learning of the trained model 52. This configuration allows the external device to perform high-load additional learning and update the trained model, thereby enabling the discrimination accuracy of the discrimination device 1 to be quickly improved at the sorting site.

[0112] In the discrimination device 1 of the above-described embodiment, the removal device 2 has the flap 21 as an example of a removal unit that removes the foreign matter F, but this is not limiting. For example, the removal device 2 may have a robot arm as the removal unit. In this case, the control unit 45 may remove the foreign matter F from the carry-in conveyor 24 by controlling the robot arm based on the discrimination result by the discrimination unit 43.

[0113] In the discrimination device 1 of the above-described embodiment, the camera 10 is used as a sensor for detecting the size of the object, but this is not limiting. The size of the object may be detected by using a trained model that is trained to input an image of the object captured by the camera 10 and output the size of the harvested product C or the foreign object F.

[0114] In the discrimination device 1 of the above-described embodiment, three illuminance reference plates, namely, a white illuminance reference plate 121, a gray illuminance reference plate 122, and a black illuminance reference plate 123, are arranged within the imaging range in which the object is imaged by the camera 10, but this is not limiting. For example, four or more illuminance reference plates that gradually become darker from white to black may be arranged. Furthermore, a white region, a gray region, and a black region may be formed on the surface of one illuminance reference plate facing the camera 10, and a specific region R may be provided in each region.

[0115] [Software implementation example] The functions of the discrimination device 1 (hereinafter referred to as the "device") can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly, each part included in the main control unit 40).

[0116] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing each function described in the above embodiment.

[0117] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0118] Furthermore, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of the control blocks can also be realized by, for example, a quantum computer.

[0119] Furthermore, each process described in the above embodiment may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0120] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0121] 〔summary〕 A discrimination device according to a first aspect of the present invention includes an imaging unit that captures an image of an object, and a discrimination unit that inputs the image of the object captured by the imaging unit into a trained model to discriminate whether the object is the harvested product or a foreign object other than the harvested product. The trained model is trained to input the image of the object captured by the imaging unit and output a discrimination result indicating whether the object is the harvested product or the foreign object, and is additionally trained using additional training data including the image of the object and a discrimination result obtained by an operator as to whether the object is the harvested product or the foreign object.

[0122] In the discrimination device according to aspect 2 of the present invention, in the above aspect 1, the imaging unit may adjust the imaging conditions of the object in accordance with changes in the lighting environment so that the state of the image of the object falls within a range suitable for discrimination by the discrimination unit.

[0123] In the discrimination device according to aspect 3 of the present invention, in the above aspect 1 or 2, the foreign matter may include the harvested product that does not satisfy a predetermined condition.

[0124] In the discrimination device according to a fourth aspect of the present invention, in any one of the first to third aspects, the discrimination by the worker may be performed in the same place as the place where the image of the object is captured by the imaging unit.

[0125] In a discrimination device according to a fifth aspect of the present invention, in any one of the first to fourth aspects, the discrimination device may be connected via wireless communication to an external device that performs additional learning of the trained model.

[0126] A discrimination device according to a sixth aspect of the present invention is any one of the first to fifth aspects, further comprising an illuminance reference plate arranged within an imaging range in which the object is imaged by the imaging unit. The imaging unit may have an adjustment unit that adjusts at least one of exposure and gain of the imaging unit by imaging the illuminance reference plate so that the brightness of the image of the object is appropriate based on the brightness of the illuminance reference plate in the captured image.

[0127] A discrimination device according to a seventh aspect of the present invention is the same as in the sixth aspect, except that the illuminance reference plates are arranged in a plurality of colors. The imaging unit has a function of selecting the illuminance reference plate to be used as a reference from the plurality of illuminance reference plates according to the brightness of the lighting environment. The adjustment unit may adjust the brightness of the image of the object to be appropriate using the brightness in the captured image of the selected illuminance reference plate as a reference.

[0128] A discrimination device according to an eighth aspect of the present invention is the discrimination device according to the seventh aspect, wherein the color of the surface of the illuminance reference plate facing the imaging unit may be at least one of gray, white, and black.

[0129] In a discrimination device according to a ninth aspect of the present invention, in any one of the sixth to eighth aspects, the imaging unit images the object and the illuminance reference plate in a lighting environment for the worker using natural light and / or artificial lighting. The artificial lighting may be turned on according to the brightness of the lighting environment.

[0130] A discrimination device according to a tenth aspect of the present invention is any one of the first to ninth aspects, further comprising a display unit that displays the object imaged by the imaging unit. The display unit may display the position of the foreign object and the discrimination result by the discrimination unit.

[0131] The discrimination device of aspect 11 of the present invention, in any of aspects 1 to 10 above, may further include a sensor that detects the size of the object and the worker's hand, and an output unit that outputs at least one of sound and light when the sensor detects an object of a size exceeding a standard or when the sensor detects the worker's hand.

[0132] A discrimination device according to a twelfth aspect of the present invention is the device according to any one of the first to eleventh aspects, further comprising an environmental sensor for detecting the temperature and humidity of the field where the harvested product is harvested. The discrimination unit discriminates whether the object is the harvested product or the foreign object by inputting the temperature and humidity of the field detected by the environmental sensor, in addition to an image of the object, into the trained model. The trained model may be trained to receive the image of the object and the temperature and humidity of the field as input and output the discrimination result.

[0133] A discrimination device according to aspect 13 of the present invention is any of aspects 1 to 12 above, further comprising an acquisition unit that acquires meteorological information about the field. The discrimination unit determines whether the object is the harvested product or the foreign object by inputting environmental information, including the temperature and humidity of the field and the meteorological information, in addition to an image of the object, into the trained model. The trained model may be trained to receive as input an image of the object and environmental information, including the temperature and humidity of the field and the meteorological information, and output the discrimination result.

[0134] A fourteenth aspect of the present invention provides a discrimination device according to any one of the first to thirteenth aspects, wherein the harvested product includes at least one of the following agricultural products: potato, sweet potato, taro, onion, pumpkin, watermelon, and cabbage. The foreign matter may include discolored, deformed, or missing parts of the agricultural product.

[0135] A removal system according to aspect 15 of the present invention includes a discrimination device according to any one of aspects 1 to 14 above, and a removal device having a removal unit that removes the foreign matter based on the discrimination result by the discrimination unit of the discrimination device.

[0136] A sixteenth aspect of the present invention relates to the removal system of the fifteenth aspect, further comprising a work machine that harvests the harvested product. The discrimination device may be attached to the work machine.

[0137] In a removal system according to a seventeenth aspect of the present invention, in the sixteenth aspect, the removal device includes an input conveyor that receives the harvested product from the work machine, an output conveyor that discharges the foreign object removed by the removal unit, and an output conveyor that transports the harvested product to a storage unit that accommodates the harvested product. The discrimination device includes a control unit that controls the removal unit. When the discrimination unit determines that the object is the harvested product, the control unit may control the removal unit to guide the harvested product to the output conveyor, while when the discrimination unit determines that the object is the foreign object, the control unit may control the removal unit to remove the foreign object from the input conveyor and guide the foreign object to the output conveyor.

[0138] In the removal system of aspect 18 of the present invention, in aspect 17 above, the removal section may be a flap that can be moved between a first position that guides the harvested product to the discharge conveyor and a second position that guides the foreign matter to the discharge conveyor under the control of the control section. [Explanation of symbols]

[0139] 1 Discrimination device 2 Removal device 3 Work equipment 10 Camera 11 Adjustment part 12, 121, 122, 123 Illuminance reference plate 21 Flap 22 Discharge conveyor 23 Discharge conveyor 24 Intake conveyor 30 Brightness sensor 30A Environmental Sensor 41 Acquisition Department 42 First Study Section 43 Discrimination part 44 Second Learning Section 45 Control Unit 50 Storage section 52 trained models 53 Training data 54 Additional training data 60 Display section 70 Output section 80 Artificial lighting 100 Removal System C, C1 to C4 harvests F, F1~F4 Foreign matter P1 1st position P2 2nd position

Claims

1. an imaging unit that captures an image of an object; a discrimination unit that discriminates whether the object is a harvested product or a foreign object different from the harvested product by inputting an image of the object captured by the imaging unit into a trained model; Equipped with the trained model is trained to input an image of the object captured by the imaging unit and output a discrimination result indicating whether the object is the harvested product or the foreign object, and is additionally trained by using additional training data including the image of the object and a discrimination result obtained by an operator as to whether the object is the harvested product or the foreign object; the imaging unit adjusts imaging conditions of the object in accordance with a change in a lighting environment so that the state of the image of the object falls within a range suitable for discrimination by the discrimination unit; further comprising a plurality of colored illuminance reference plates arranged within an imaging range in which the object is imaged by the imaging unit; The imaging unit a function of selecting the illuminance reference plate to be used as a reference from among the illuminance reference plates of the plurality of colors according to the brightness of the lighting environment; an adjustment unit that adjusts at least one of exposure and gain of the imaging unit by capturing an image of the illuminance reference plate so that the brightness of the image of the object becomes appropriate based on the brightness of the illuminance reference plate in the captured image; The adjustment unit adjusts the brightness of the image of the object to be appropriate based on the brightness in the captured image of the selected illuminance reference plate.

2. The discrimination device according to claim 1 , wherein the foreign matter includes the harvested product that does not satisfy a predetermined condition.

3. The discrimination device according to claim 1 , wherein the discrimination by the worker is performed on a conveyor that transports the harvested product.

4. The discrimination device according to claim 1 , wherein the discrimination device is connected via wireless communication to an external device that executes additional learning of the trained model.

5. The discrimination device according to claim 1 , wherein the color of the surface of the illuminance reference plate facing the imaging unit is at least one of gray, white, and black.

6. the imaging unit images the object and the illuminance reference plate under a lighting environment for the worker using natural light and / or artificial lighting, The determination device according to claim 5 , wherein the artificial lighting is turned on in accordance with the brightness of the lighting environment.

7. a display unit that displays the object captured by the imaging unit; The discrimination device according to claim 1 , wherein the display unit displays the position of the foreign matter and the discrimination result by the discrimination unit.

8. a sensor for detecting the size of the object and the worker's hand; 2. The discrimination device according to claim 1, further comprising an output unit that outputs at least one of sound and light when the sensor detects the object whose size exceeds a standard or when the sensor detects the worker's hand.

9. Further, an environmental sensor is provided for detecting the temperature and humidity of the field where the harvested product is harvested; the discrimination unit discriminates whether the object is the harvested product or the foreign object by inputting the temperature and humidity of the field detected by the environmental sensor, in addition to the image of the object, into the trained model; The discrimination device according to claim 1 , wherein the trained model is trained to receive an image of the object and the temperature and humidity of the field as inputs and output the discrimination result.

10. An acquisition unit that acquires meteorological information about the field, the discrimination unit discriminates whether the object is the harvested product or the foreign object by inputting environmental information, including the temperature and humidity of the field and the meteorological information, in addition to the image of the object into the trained model; The discrimination device according to claim 9 , wherein the trained model is trained to output the discrimination result using an image of the object and environmental information including the temperature and humidity of the field and meteorological information as input.

11. The harvested product includes at least one agricultural crop selected from the group consisting of potatoes, sweet potatoes, taro, onions, pumpkins, watermelons, and cabbages; The discrimination device according to claim 2 , wherein the foreign matter includes discolored, deformed, or missing crops.

12. The discrimination device according to any one of claims 1 to 11, a removal device having a removal unit that removes the foreign matter based on a determination result by the determination unit of the determination device; A removal system with

13. Further provided is a work machine for harvesting the crop, The removal system according to claim 12 , wherein the discrimination device is attached to the work machine.

14. The removal device comprises: a carry-in conveyor that carries the harvested product from the work machine; a discharge conveyor that discharges the foreign matter removed by the removal unit; a discharge conveyor for discharging the harvested product to a storage section for storing the harvested product, The discrimination device is a control unit that controls the removal unit, The control unit When the discrimination unit discriminates that the object is the harvested product, the removal unit is controlled to guide the harvested product to the discharge conveyor. The removal system according to claim 13, wherein when the discrimination unit determines that the object is the foreign object, the removal unit is controlled to remove the foreign object from the carry-in conveyor and guide the foreign object to the discharge conveyor.

15. The removal system of claim 14, wherein the removal unit is a flap that can be moved, under the control of the control unit, between a first position that guides the harvested product to the discharge conveyor and a second position that guides the foreign object to the discharge conveyor.

Images