Determination device, learning device, determination method, learning method, and control program

Abstract

A determination device (1) according to an embodiment of the present invention comprises: an acquisition unit (12) that acquires an image including one or more grains of a granular agricultural product; and a determination unit (14) to which the image is input and that, using a learned machine learning model in which a determination result regarding the mycotoxin concentration or the degree of contamination in the grains of the granular agricultural product is output for each grain, determines whether individual grains or all of the one or more grains of the granular agricultural product are contaminated with a mycotoxin to a degree equal to or greater than a predetermined reference value or in a predetermined concentration range, or determines the mycotoxin concentration or the contamination degree of the individual grains or of all of the one or more grains of the granular agricultural product.

Description

[Technical Field]

[0001] The present invention relates to a determination device, a learning device, a determination method, a learning method, and a control program. [Background technology]

[0002] During the growing period or post-harvest storage period of grains such as wheat and other agricultural crops, mycotoxin-producing fungi such as Fusarium head blight can infect or multiply, resulting in some or all of the harvested product becoming contaminated with mycotoxins.

[0003] To prevent mycotoxin contamination caused by Fusarium head blight in grains such as wheat, it is important to carry out appropriate management (such as timely spraying of pesticides) during the cultivation stage in the field to prevent infection by the Fusarium head blight fungus as much as possible and prevent the accumulation of mycotoxins. However, due to factors such as the weather conditions of the year, it may not always be possible to prevent this completely.

[0004] Patent Document 1 discloses a color sorter that irradiates raw wheat with near-infrared light to separate out red mold wheat, while Patent Document 2 discloses a method for classifying the level of contamination in grains by performing multivariate data analysis on the diffuse light absorption spectrum of the grains. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2005-28302 [Patent Document 2] Japanese Patent Publication No. 2019-510968 Summary of the Invention [Problem to be solved by the invention]

[0006] In particular, in barley, mycotoxin-infected grains due to Fusarium head blight do not show obvious external symptoms. Although grain thickness sorting is somewhat effective, its effectiveness is not always high. Therefore, there is no effective method for postharvest mycotoxin reduction. In wheat, mycotoxin-infected grains often show symptoms such as whitening and shrinkage. While postharvest grain thickness sorting, gravity sorting, and color sorting are somewhat effective in reducing mycotoxins, mycotoxins can accumulate in grains without obvious external symptoms, and no method for selecting such grains has been available. Furthermore, when examining mycotoxin concentrations or levels of contamination in harvested grains and other crops, commonly used chemical analysis methods and immunological methods such as ELISA require sample crushing for analysis, which is costly and time-consuming. Therefore, a nondestructive and simple method for estimating mycotoxin concentrations or levels of contamination is needed.

[0007] The present inventors attempted to use machine learning to perform image discrimination using datasets of image data and mycotoxin concentration data for individual grains of Fusarium head blight-infected barley, which generally exhibit unclear external symptoms. Contrary to expectations, the results demonstrated that image discrimination between grains contaminated with high levels of mycotoxins, at levels that should be removed from the harvest, and non-contaminated grains was possible. Further analysis showed that discrimination between grains contaminated with high levels of mycotoxins and non-contaminated or low levels of mycotoxins was also possible under other analytical conditions, leading to the discovery of the present invention.

[0008] One aspect of the present invention aims to improve the efficiency and accuracy of determining the mycotoxin concentration or degree of contamination in grains such as wheat and other granular agricultural products, and of sorting and removing mycotoxin-contaminated grains based on that determination. [Means for solving the problem]

[0009] In order to solve the above problems, a determination device according to one embodiment of the present invention comprises an acquisition unit that acquires an image including one or more grains of granular agricultural product, and a determination unit that uses a trained machine learning model that receives the image and outputs a determination result regarding the mycotoxin concentration or degree of contamination in the grains of the granular agricultural product for each grain, to (1) determine whether each or all of the one or more grains of the granular agricultural product are contaminated with mycotoxins above a predetermined standard value or within a predetermined concentration range, or (2) determine the mycotoxin concentration or degree of contamination in each or all of the one or more grains of the granular agricultural product.

[0010] In order to solve the above problems, a learning device according to one embodiment of the present invention comprises an acquisition unit that acquires an image containing one or more grains of granular agricultural product and concentration information indicating the mycotoxin concentration or degree of contamination for each grain of the granular agricultural product, and a learning unit that uses the set of image and concentration information acquired by the acquisition unit as training data to train a machine learning model that receives an input of an image containing one or more grains of the granular agricultural product and outputs a judgment result regarding the mycotoxin concentration or degree of contamination for each grain of the granular agricultural product.

[0011] In order to solve the above problem, a determination method according to one embodiment of the present invention includes an acquisition step of acquiring an image including one or more grains of granular agricultural product, and a determination step of (1) determining whether each or all of one or more grains of the granular agricultural product are contaminated with mycotoxins above a predetermined standard value or within a predetermined concentration range, or (2) determining the mycotoxin concentration or contamination level of each or all of one or more grains of the granular agricultural product, using a trained machine learning model that receives the image and outputs a determination result regarding the mycotoxin concentration or contamination level in the grains of the granular agricultural product for each grain.

[0012] In order to solve the above problem, a learning method according to one embodiment of the present invention includes an acquisition step of acquiring an image including one or more grains of granular agricultural product and concentration information indicating the mycotoxin concentration or degree of contamination in each grain of the granular agricultural product, and a learning step of training a machine learning model using the set of image and concentration information acquired in the acquisition step as training data, which receives an image including one or more grains of the granular agricultural product as input and outputs a judgment result regarding the mycotoxin concentration or degree of contamination in each grain of the granular agricultural product. [Effects of the Invention]

[0013] According to one aspect of the present invention, it is possible to improve the efficiency and accuracy of determining the mycotoxin concentration or degree of contamination of grains such as wheat and other granular agricultural products, and of sorting and removing contaminated grains based on that determination. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a functional block diagram of a determination device (learning device) according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing an example of an image of individual grains of harvested wheat and barley. [Figure 3] 10 is a flowchart illustrating a flow of an example of a determination process according to an embodiment of the present invention. [Figure 4] 10 is a flowchart illustrating a flow of an example of a learning process according to an embodiment of the present invention. [Figure 5] FIG. 1 is a diagram showing information regarding wheat and barley samples used in Example 1 and the like. [Figure 6] FIG. 6 is a diagram showing the number of images for each contamination concentration of the wheat grains shown in FIG. 5. [Figure 7] FIG. 10 is a diagram showing a screen displaying the learning process of a neural network. [Figure 8] FIG. 10 is a diagram illustrating a result of a determination process according to the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating a result of a determination process according to the third embodiment. [Figure 10] FIG. 10 is a diagram illustrating a result of a determination process according to the fourth embodiment. [Figure 11] FIG. 10 is a diagram illustrating a result of a determination process according to the fourth embodiment. [Figure 12] FIG. 10 is a diagram illustrating a result of a determination process according to the fourth embodiment. [Figure 13] FIG. 10 is a diagram showing information on barley samples used as training data and test data sets in Example 5. [Figure 14] FIG. 10 is a diagram showing the learning process when cross-validation is performed using barley images from each sample series. [Figure 15] FIG. 10 is a diagram showing the learning process when cross-validation is performed using barley images from each sample series. [Figure 16] FIG. 10 is a diagram showing the results of cross-validation according to Example 5. [Figure 17] This figure shows the judgment results when an image serving as a test data set is given as input to a trained neural network trained in cross-validation. [Figure 18] This figure shows the judgment results when an image serving as a test data set is given as input to a trained neural network trained in cross-validation. [Figure 19] FIG. 10 is a diagram showing the relationship between the probability threshold for determining a grain as being highly contaminated and the sorting yield for the test data set of each sample series combination in cross-validation in Example 5. DETAILED DESCRIPTION OF THE INVENTION

[0015] [Embodiment] An embodiment of the present invention will be described in detail below. In this embodiment, a determination device will be described that can determine whether a grain to be processed contained in an image is contaminated with mycotoxins such as DON (deoxynivalenol) or NIV (nivalenol) at a level equal to or greater than a predetermined reference value or within a predetermined concentration range from a certain concentration to a certain concentration. Hereinafter, the description will be given assuming that the grain is wheat or barley as an example, but this is not limiting. As will be described later, the processing target of the determination device (learning device) may be other grains such as corn, or various other granular agricultural products.

[0016] [1. Example of the configuration of the determination device (learning device)] A description will be given of a determination device (learning device) 1 according to this embodiment. In the following description, even when the determination device 1 functions as a learning device, the name will not be differentiated.

[0017] 1 is a functional block diagram of a determination device 1 according to this embodiment. As shown in Fig. 1, the determination device 1 includes a control unit 10, a storage unit 20, an action unit 22, an imaging unit 24, a measurement unit 26, an input unit 28, and a display unit 30.

[0018] The control unit 10 is a control device that controls the entire determination device 1, and also functions as an acquisition unit 12, a determination unit 14, and a learning unit 16.

[0019] The acquisition unit 12 acquires an image including one or more grains of harvested wheat and barley, and concentration information indicating the mycotoxin concentration or contamination level for each grain of wheat and barley. The image here is not limited to an RGB image composed of data in the three wavelength bands of visible light, for example Includes wavelengths from ultraviolet to infrared Muga image· Multispectral in the visible light range It may be an image. It may also be image data composed of data of one or more wavelength ranges selected from the ultraviolet to infrared range. 。

[0020] Furthermore, the term "mycotoxin concentration (mycotoxin contamination concentration)" as used herein refers to a value calculated as the weight ratio (weight of target mycotoxin per unit weight) of the target mycotoxin contained in each target particle or multiple particles as a whole to the entire target particle (individual particle or multiple particles), and the "degree of mycotoxin contamination" indicates the degree of mycotoxin contamination, which may have a certain degree of range, but may also include the above-mentioned "mycotoxin concentration."

[0021] Figure 2 shows an example of the image. The original image in Figure 2 is an RGB color image, but it has been converted to grayscale for this purpose. The images in image group 50 in Figure 2 are images of uncontaminated grain (barley) that is not contaminated with mycotoxins, while the images in image group 52 are images of highly contaminated grain (barley) containing more than 5 ppm (μg / g) of mycotoxins (the sum of DON and NIV, which are trichothecene mycotoxins produced by Fusarium head blight fungi). In image groups 50 and 52, the top six images are images of the front side of the grain, and the bottom six images are images of the back side of the grain. Here, the side of the grain with the "grain groove" (also known as the "belly groove," a single vertical groove) is referred to as the "back side," and the side without the grain groove is referred to as the "front side." The grains of barley, wheat, and other grains have grooves (ventral grooves), which create a distinction between the front and back sides of the grain due to their structure, but the grains of corn and rice do not have grooves, so when the grains are arranged on a flat surface, there is no distinction between the front and back sides.In addition to barley and wheat, the grain family also includes oats (also called oats), rye, triticale, etc.

[0022] Note that the control unit 10 may be configured to process an image containing tens to thousands of grains as a bulk sample of wheat or barley (here, a bulk sample refers to a sample of grains in a single batch) by dividing the image into individual grain images. The images acquired by the acquisition unit 12 include training images that serve as training data for the machine learning model, and assessment images that are used to assess the mycotoxin concentration or contamination level of the wheat or barley grains. In the present disclosure, grains of wheat or barley are sometimes simply referred to as "grains," and grains of cereals and other granular agricultural products are sometimes simply referred to as "grains" or "grains." The grains may be hulled or naked two-row or six-row barley, or wheat, etc. That is, the acquisition unit 12 may acquire an image containing, for example, one or more grains of harvested barley that are hulled or peeled.

[0023] The determination unit 14 uses a trained machine learning model that receives an image acquired by the acquisition unit 12 and outputs a determination result regarding the mycotoxin concentration or contamination level in the wheat grain contained in the image to determine whether the wheat grain is contaminated with a mycotoxin equal to or greater than a predetermined standard value or within a predetermined concentration range. Here, the determination result regarding the mycotoxin concentration or contamination level in the wheat grain may include, for example, an estimated mycotoxin concentration in the wheat grain, or, if wheat grains are classified according to the contamination level within a specific range, information indicating which class the wheat grain is classified into.

[0024] The learning unit 16 uses a set of training images acquired by the acquisition unit 12 and concentration information indicating the mycotoxin concentration or contamination level of each grain of wheat as training data to train a machine learning model that receives an input image containing one or more harvested wheat grains and outputs a determination result regarding the mycotoxin concentration or contamination level of each grain. The determination method performed by the determination unit 14 and the machine learning method performed by the learning unit 16 are not limited to a specific method and may use, for example, a convolutional neural network (CNN) or a recurrent neural network (RNN). Among machine learning methods, CNN and RNN are considered to be neural networks, or even deep learning methods. Either a classification model or a regression model may be used. Input data may be preprocessed before being used as input to the machine learning model. For this type of processing, when using a neural network such as CNN, in addition to arranging the data in two or multi-dimensional ways, various data augmentation techniques can be used, such as adjusting brightness, color tone, image quality, and angle of the object, object detection for extracting the target area, background removal, and segmentation.

[0025] Furthermore, when using CNN, a convolutional layer that performs convolutional operations may be provided as one or more layers included in the neural network, and a filter operation (product-sum operation) may be performed on input data input to the layer. When performing the filter operation, processing such as padding may be used in combination, or an appropriately set stride width may be adopted.

[0026] Alternatively, for example, any one of the following machine learning techniques or a combination thereof may be used.

[0027] Support Vector Machine (SVM) Decision tree Random Forest ·K Nearest Neighbors (KNN) ·Linear Discriminant Analysis (LDA) ·Quadratic Discriminant Analysis (QDA) ·Partial Least Squares Discriminant Analysis (PLSDA) Naive Bayes ·Principal Component Analysis (PCA) Regression Analysis Ensemble Learning Clustering Inductive Logic Programming (ILP) Genetic Algorithm (GP) Bayesian Network (BN) Multilayer perceptron (MLP) Autoencoder Transformer The memory unit 20 is a storage device that stores various information, such as a parameter set that defines the above-mentioned machine learning model, information indicating a predetermined reference value that serves as a criterion for determining whether wheat grains are contaminated with mycotoxins at a predetermined level or within a predetermined concentration range, etc. The memory unit 20 also stores, for example, at least temporarily, the captured images acquired by the acquisition unit 12.

[0028] The action unit 22 is a mechanism that aligns the target wheat grains in a predetermined direction based on the control of the control unit 10. For example, the action unit 22 can be realized as an arm or roller that applies a physical action to the wheat grains that are stationary relative to the ground surface.

[0029] The photographing unit 24 is a camera mechanism that photographs images of wheat grains and the like under the control of the control unit 10. The photographing unit 24 photographs images of wheat grains aligned by the action unit 22 and supplies the images to the acquisition unit 12. The photographing unit 24 may operate as an ultraviolet camera or an infrared camera, photographing images of wheat grains in the ultraviolet or infrared region. The judgment unit 14 may perform judgment using a machine learning model trained using images of one or more grains of wheat imaged in the ultraviolet or infrared region as training data. This allows the judgment device 1 to perform more appropriate judgment processing by referencing information in the ultraviolet or infrared region that cannot be perceived by the naked eye. The action of the action unit 22 and the photographing unit 24 allows the acquisition unit 12 to acquire images of the target wheat grains aligned in a predetermined direction.

[0030] The measurement unit 26 measures or estimates the mycotoxin concentration or contamination level of the target wheat grain using a method other than chemical analysis or immunological methods, and supplies the measurement results to the acquisition unit 12. The measurement results represent concentration information indicating the mycotoxin concentration or contamination level of the wheat grain. As a method for estimating the mycotoxin contamination level of the target grain without using chemical analysis or immunological methods, for example, in the case of wheat, if a target sample is mycotoxin-contaminated, it may be possible to identify highly contaminated grains contained therein with the naked eye to some extent. Therefore, it is conceivable to estimate the mycotoxin contamination level of the target grain using the naked eye or color sorting, to the extent possible. Other possible methods include, for example, estimating the mycotoxin contamination level by quantifying the infection amount of mycotoxin-producing bacteria in the target grain.

[0031] The input unit 28 is an interface for inputting operations or information to the determination device 1. A part of the input unit 28 can be realized as a device such as a keyboard or a mouse that accepts operations to the determination device 1.

[0032] The display unit 30 is a display panel that displays text, moving images, etc. under the control of the control unit 10. The display unit 30 may be configured to implement part of the functions of the input unit 28 as a touch panel.

[0033] In another embodiment, the processing target of the determination device 1 is not limited to wheat and barley. Similar processing may be performed on other grains such as corn, rice (including unhulled rice, brown rice, and polished rice), miscellaneous grains (including buckwheat, tartary buckwheat, and Job's tears), and beans (including soybeans), as well as granular agricultural products such as nuts, seeds, and fruits of various crops, including peanuts, coffee beans, cocoa beans, nutmeg (seeds or kernels), and peppers. Furthermore, the target granular agricultural products may be either shelled or deshelled, or may be processed, such as by grinding. For example, wheat and barley may be either unhulled or polished, and barley may be malt, roasted barley for barley tea, whole barley, pressed barley, or rice grain barley (cut barley). Grains harvested before the appropriate harvest time of each target crop may also be targeted. In addition, the test may be configured to assess mycotoxins other than DON and NIV caused by Fusarium head blight, various mycotoxin contaminations other than Fusarium head blight (including aflatoxins, fumonisins, zearalenone, ochratoxin, T-2 toxin, and HT-2 toxin), contamination other than mycotoxins (including pesticide residues and harmful heavy metals such as arsenic and cadmium), diseases (including latent infections of harmful microorganisms that are difficult to discern with the naked eye), and other quality defects, either individually or in combination with multiple tests. In the case of mycotoxin contamination, individual kernels or entire kernels may be simultaneously contaminated with multiple mycotoxins. If necessary, these multiple mycotoxins may be combined for assessment. For example, the trichothecene mycotoxins DON and NIV (type B) are particularly important mycotoxins caused by Fusarium head blight in wheat. However, chemically modified derivatives of these mycotoxins (such as various acetylated forms and glycosides) may accumulate in the grain along with DON and NIV. If these substances are measured separately in association with DON or NIV in the granules used for training data, they may be added together with the individual DON or NIV as needed (and in this case, a conversion value reflecting the ratio of the molecular weight of DON or NIV to the molecular weight of each substance may be used for addition).The use of the granular agricultural products to be processed is not limited to food or feed, and may also be for seed or industrial use, etc.Furthermore, the present invention may be applied as a secondary sorting (assessment) method after removing as many target grains as possible, such as grain color sorting for wheat, which allows for some visual differentiation of grains contaminated with high levels of mycotoxins. The present invention may also be applied after non-image-based sorting, such as grain thickness sorting or specific gravity sorting, or after or simultaneously with sorting out unsuitable grains, such as foreign matter, discolored grains, and immature grains, using other methods or image-based methods. Furthermore, as opposed to assessing defective traits, the present invention may also be applied to assess various "good" quality traits. Furthermore, target grains that are not suitable for normal assessment, such as those with moisture content exceeding the appropriate range, may be detected in advance or simultaneously using the present invention or other methods, or the assessment may be corrected.

[0034] Furthermore, the target is not necessarily limited to post-harvest grains, etc. For crops such as barley (hulled barley) whose grains to be harvested are exposed when standing in the field, judgment may be made on the target grains contained in the image using images taken in the field before harvest, for example, with a camera mounted on a smartphone, drone, harvester, etc. In this case, different image processing and judgment correction may be performed than when judging the target after harvest.

[0035] Furthermore, some functions of the determination device 1, such as the photographing unit 24 and the measuring unit 26, may be realized on a separate device. In other words, the determination device 1 may be realized by a plurality of independent devices. For example, a device for performing a determination process and a device for performing a learning process may be realized as separate devices, with the devices working together. In another embodiment, the determination device 1 may not include the operating unit 22, and the user may manually perform the processing of the operating unit 22 as needed. The determination device 1 may be configured to be mounted on a sorting machine that sorts granular agricultural products, for example, or may be realized as a device separate from the sorting machine.

[0036] [2. Example of judgment process] Next, an example of a determination process will be described in which the determination device 1 refers to an image of a grain of granular agricultural product to be determined (here, an example of wheat, which may hereinafter be referred to as wheat grain) and determines whether the wheat grain is contaminated with mycotoxins at a level equal to or greater than a predetermined standard value, using a neural network (here, a regression model) as a machine learning model. Figure 3 is a flowchart showing the flow of the process in this example.

[0037] In step S101, the action unit 22 performs a process of aligning the wheat grains to be judged by the judgment device 1 in a predetermined orientation. The predetermined orientation may be any one of one or a plurality of orientations.

[0038] In step S102, the photographing unit 24 photographs the images of the wheat grains aligned by the action unit 22. The acquisition unit 12 also acquires the images photographed by the photographing unit 24.

[0039] In step S103, the determination unit 14 determines the orientation (e.g., front side, back side) of the wheat grain contained in the image acquired by the acquisition unit 12, for example, by referring to information supplied from the operation unit 22, or by the determination unit 14 itself by referring to the image, and then inputs the image into a trained neural network corresponding to the orientation of the wheat grain. This allows a suitable neural network trained using, for example, images of wheat grains with the same orientation to be used for the determination process. As mentioned above, unlike corn and rice, grains of wheat such as barley and wheat have grain grooves (ventral grooves) that are clearly visible externally, and the grain structure distinguishes between the front and back sides. Therefore, whether the grain in the image is the front side or the back side may affect learning and determination. Note that the "orientation" of the wheat grain here may include not only the front side and the back side, but also the side, tip side, base side, or various angles and directions that can be seen in the image.

[0040] In step S104, the determination unit 14 compares the determination result regarding the mycotoxin concentration in the wheat grain output from the neural network with a predetermined reference value stored in the memory unit 20 to determine whether the wheat grain is contaminated with mycotoxins at or above the predetermined reference value or within a predetermined concentration range. Also in step S104, the control unit 10 causes the display unit 30 to display information indicating the determination result.

[0041] The image input to the trained neural network by the determination unit 14 may include multiple wheat grains. In this case, the trained neural network may determine the mycotoxin concentration for each grain and output the determination result.

[0042] The above describes an example of a determination method executed by the determination device 1, which includes an acquisition step of acquiring an image containing one or more grains of harvested granular agricultural product, and a determination step of determining whether one or more grains of the granular agricultural product are contaminated with mycotoxins above a predetermined standard value or within a predetermined concentration range using a trained neural network that inputs the image and outputs a determination result regarding the mycotoxin concentration in the grains of the granular agricultural product for each grain.

[0043] In addition, as a subsequent process, for example, the determination unit 14 may determine the degree of mycotoxin contamination or estimate the mycotoxin concentration by referring to one or more of its own determination results or the outputs of multiple neural networks. Furthermore, a comprehensive determination may be made using the determination results for multiple orientations for each grain, or using the determination results of different machine learning models based on multiple types of images (e.g., multispectral images in the visible light region and images captured in the infrared region). Furthermore, the determination unit 14 may acquire images containing tens to thousands of wheat grains as a bulk sample, extract or detect individual grains in the bulk sample using techniques such as object detection, background removal, and segmentation, and determine the mycotoxin concentration or contamination level of each grain. For example, by calculating the proportion of grains determined to have each contamination level, the proportion of highly contaminated grains or the overall contamination level of the bulk sample may be determined, or the mycotoxin concentration of the entire bulk sample may be estimated. The determination unit 14 may also determine whether or not one or more individual grains of a granular agricultural product or the entire bulk sample is contaminated with mycotoxins at or above a predetermined standard value or within a predetermined concentration range.

[0044] In step S103, the judgment unit 14 may be configured to determine the type of wheat grain (for example, wheat or barley, and if it is barley, whether it is hulled or naked barley, or whether it is two-rowed or six-rowed barley) and the orientation of the wheat grain, and then input the image acquired from the acquisition unit 12 into a trained neural network corresponding to each type.

[0045] According to the configuration of this example, it is possible to improve the efficiency and accuracy of determining the mycotoxin concentration or contamination level of wheat and other grains, compared to, for example, determining by the naked eye.

[0046] <Modifications Related to Determination Processing> The determination unit 14 may be configured to determine that an individual grain of one or more grains of granular agricultural produce included in the input image is contaminated with mycotoxins at or above the predetermined standard value if the estimated mycotoxin concentration of the grain is equal to or above the predetermined standard value, or if the probability that the grain is estimated to be contaminated with mycotoxins at or above the predetermined standard value is equal to or above a set probability threshold. The probability threshold here refers to a threshold value that determines, when a target grain is determined using a trained machine learning model (in this case, a classification model), what percentage or higher the probability that the target grain is estimated to belong to a predetermined classification class is required to classify the target grain into that class. More specifically, the probability threshold here refers to a threshold value that determines, for example, what percentage or higher the probability that a target barley grain is estimated (classified) as a highly contaminated grain at or above the predetermined standard value is required to determine the target barley grain as a highly contaminated grain.

[0047] In a broader sense, the determination unit 14 determines whether or not each of one or more grains of a granular agricultural product is contaminated with a mycotoxin at a level equal to or greater than a predetermined reference value using a predetermined determination threshold. Here, the predetermined reference value and probability threshold are examples of the determination threshold. The determination threshold may be set by the determination unit 14.

[0048] In addition, in the above configuration, the machine learning model such as a neural network may be configured to receive an image containing one or more grains of granular agricultural product as input and output the probability that each grain of the one or more grains of the granular agricultural product is contaminated with mycotoxins equal to or greater than a predetermined standard value.

[0049] The determination unit 14 may also calculate a sorting yield corresponding to each of one or more determination thresholds for the set of grains to be determined. Here, the sorting yield is the proportion (ratio of grains by number or ratio of grain images) or weight ratio of grains determined by the determination unit 14 to be not contaminated with mycotoxins at a predetermined standard value or above among the set of grains to be determined, i.e., one or more grains of a granular agricultural product.

[0050] For example, the sorting yield (particle number ratio) (%) is calculated using the formula: (1 - (number of particles determined by the determination device to be highly contaminated particles / number of particles to be determined)) x 100. Also, for example, by learning particle weight data corresponding to each particle image, or data on the relative particle weight ratio (of each particle) based on the average particle weight of low-concentration contaminated particles or non-contaminated particles less than a predetermined standard value in the same sample lot as the particle image, as training data, in pairs with each particle image, it becomes possible to estimate each particle weight or the relative particle weight ratio (relative to low-concentration contaminated particles or non-contaminated particles less than a predetermined standard value) from the individual particle image to be determined. Therefore, using this estimated value, the sorting yield (weight ratio) (%) can be calculated (estimated) using a formula such as (1 - (total grain weight (estimated value) of grain images judged by the judgment device to be highly contaminated grains / total grain weight (estimated value) of images of all grains to be judged)) x 100 or (1 - (total value of relative grain weight ratio (estimated value) of grain images judged by the judgment device to be highly contaminated grains / total value of relative grain weight ratio (estimated value) of images of all grains to be judged)) x 100.Here, it is assumed that the relative grain weight ratio data will tend to be smaller than 1 for grains highly contaminated with mycotoxins, for example, in wheat, for which gravity sorting is generally considered to have a large effect on reducing mycotoxins.

[0051] Alternatively, for example, the sorting yield (particle number ratio) (%) formula can be corrected using a coefficient A assumed or estimated from the data accumulated up to that point, and the sorting yield (weight ratio) (%) can be estimated using the formula (1 - (number of particles determined by the judgment device to be highly contaminated particles x A / number of particles to be judged)) x 100. Also, by providing the judgment device 1 with a mechanism for directly measuring the weight of each particle before or after acquiring an image of each particle to be judged, it is possible to use the directly measured weight data of each particle to calculate a more accurate value for the sorting yield (weight ratio) (%) using the formula (1 - (total actual weight of particles determined by the judgment device to be highly contaminated particles / total actual weight of particles to be judged)) x 100. Such sorting yield (particle number ratio or weight ratio) values ​​can be used, for example, as a basis for determining the judgment threshold to be used for judging highly contaminated particles. Although the sorting yield based on the grain number ratio is easier to calculate than the sorting yield based on the weight ratio, it is generally believed that the sorting yield based on the weight ratio, which directly indicates the amount of yield remaining as a weight ratio after grain sorting, is often given more importance when considering the judgment threshold in actual grain sorting situations.

[0052] In addition, as described below, the judgment unit 14 may use an estimation model developed using training data, etc. accumulated to improve the judgment performance of the judgment device, etc., to estimate the mycotoxin reduction rate or the mycotoxin concentration or degree of contamination of the entire collection of remaining grains when grains judged to be contaminated with mycotoxins at or above a predetermined standard value using a predetermined judgment threshold are removed from the collection of grains to be judged as granular agricultural products, for each of one or more of the judgment thresholds or sorting yields.

[0053] [3. Learning process example] Next, an example of a learning process in which the determination device 1 uses training data to train a neural network as a machine learning model will be described, taking a case where wheat or barley is the target. Fig. 4 is a flowchart showing the flow of the process in this example.

[0054] In step S201, the action unit 22 performs a process of aligning the wheat grains in a predetermined orientation. The predetermined orientation may be any one of one or a plurality of orientations.

[0055] In step S202, the photographing unit 24 photographs an image of the wheat grains aligned by the action unit 22. The acquisition unit 12 acquires the image photographed by the photographing unit 24. After this, the process may return to step S201, and after aligning the same wheat grains in a different direction, the process of photographing an image in this step may be repeated as many times as necessary.

[0056] In step S203, the measurement unit 26 measures or estimates the mycotoxin concentration or contamination level of the wheat grain to be used in the learning process, and the acquisition unit 12 acquires the value of the mycotoxin concentration or contamination level as concentration information.

[0057] In step S204, the learning unit 16 determines the orientation (front side, back side, etc.) of the wheat grains included in the images acquired by the acquisition unit 12, for example, by referring to information supplied from the operation unit 22 or information indicating the orientation of the wheat grains input by the user via the input unit 28. The learning unit 16 then inputs the set of image and concentration information of the wheat grains acquired by the acquisition unit 12 as training data into a neural network corresponding to the orientation of the wheat grains, and trains the neural network. The neural network receives an image containing one or more grains of harvested wheat or barley, and outputs a determination result regarding the mycotoxin concentration or degree of contamination in each grain.

[0058] The image input to the neural network by the learning unit 16 may include multiple wheat grains. In this case, in addition to the image of the wheat grains and density information of each wheat grain, information such as coordinates indicating the position of each wheat grain in the image may also be input to the neural network.

[0059] The above describes an example of a learning method executed by a learning device (determination device) 1, which includes an acquisition step of acquiring an image containing one or more grains of harvested granular agricultural product and concentration information indicating the mycotoxin concentration or degree of contamination in each grain of the granular agricultural product, and a learning step of training a neural network that receives an image containing one or more grains of harvested granular agricultural product and outputs a determination result regarding the mycotoxin concentration or degree of contamination in each grain of the granular agricultural product, using the set of image and concentration information acquired in the acquisition step as training data.

[0060] In step S204, the learning unit 16 may be configured to determine the type of wheat (for example, wheat or barley, and in the case of barley, whether it is hulled or naked barley, or whether it is two-rowed or six-rowed barley) and the orientation of the wheat grain (front side, back side, etc.) by referring to information indicating the type of wheat, etc., input by the user via the input unit 28, and then input the image acquired from the acquisition unit 12 into a neural network corresponding to each type.

[0061] Furthermore, in step S204, the learning unit 16 may perform preprocessing such as cropping or resizing on the image acquired by the acquisition unit 12. Furthermore, the learning unit 16 may perform data augmentation such as inversion, enlargement, reduction, and translation on the image acquired from the acquisition unit 12, adjust brightness, color tone, image quality, angle of the object, etc., or extract a target region by object detection, background removal, segmentation, etc. Furthermore, the extraction of the target region is not necessarily limited to the entire particle, and only a partial region of the particle that is of interest may be extracted.

[0062] According to the configuration of this example, it is possible to train a neural network to improve the efficiency and accuracy of determining the mycotoxin concentration or contamination level of wheat and barley.

[0063] In addition, when grains such as wheat and barley are classified according to the degree of contamination within a specific range, the learning unit 16 may use only images of grains corresponding to one or more specific classes as training data. For example, the learning unit 16 may be configured to input images including grains of a first class whose mycotoxin concentration or contamination level is equal to or greater than a predetermined reference value and images including grains of a second class of granular agricultural product whose mycotoxin concentration or contamination level is equal to or less than a predetermined threshold set below the reference value (e.g., equal to or less than 70% of the reference value, equal to or less than half the reference value, equal to or less than one-fifth or one-tenth of the reference value, etc.) or which is uncontaminated into a machine learning model, and train the machine learning model to output a determination result, for example, a probability value, as to whether the grains included in the input images are grains contaminated with mycotoxins equal to or greater than the predetermined reference value.

[0064] Furthermore, the above-mentioned predetermined reference value and predetermined threshold value are not limited to specific values ​​and may be changeable. For example, the predetermined reference value may be 5 ppm and the predetermined threshold value may be 0.5 ppm.

[0065] In this case, by using grains contaminated with mycotoxins at or above a certain level and grains with significantly different levels of contamination for training, it is possible to favorably converge the values ​​of the parameter set that defines the machine learning model, for example, and improve the accuracy of the output value of the machine learning model. Furthermore, by using not only uncontaminated grains but also grains with low levels of contamination below a certain threshold for training, when the present invention is applied, for example, to removing highly contaminated grains from an actual mycotoxin-contaminated sample (which may include grains with various levels of contamination from non-contaminated to highly contaminated), grains with low levels of contamination at an acceptable level can be distinguished from highly contaminated grains and left in the sample, and it is thought that the sorting yield (weight ratio or grain number ratio of the sample after grain sorting treatment to the sample before sorting) can be prevented from being reduced more than necessary.

[0066] As training data used in learning such two-class classification, the threshold value of mycotoxin concentration when classifying into a highly contaminated class and a low-contaminated to non-contaminated class can be adjusted for the high concentration side (here, a predetermined reference value) and the low concentration side, respectively, or when using the trained machine learning model for discrimination (grain sorting), the probability threshold used when determining whether a grain is highly contaminated or not can be adjusted. This may enable adjustment of the balance between the mycotoxin reduction effect due to the removal of highly contaminated grains and the sorting yield (these are usually in a trade-off relationship). As an example of the configuration of the determination device 1, the determination unit 14 may make a determination regarding the mycotoxin concentration or the degree of contamination using the probability threshold that can be adjusted by a user's input to the input unit 28, etc.

[0067] Furthermore, for example, in the case of wheat in which grains contaminated with high levels of mycotoxins can be distinguished to some extent by appearance, if primary sorting is performed by color sorting or the like and then the judgment according to the present invention is applied as a secondary sorting method, more efficient sorting can be achieved by using a machine learning model trained with a threshold setting different from that used when primary sorting is not performed. Furthermore, efficient sorting can be achieved by applying the technology of the present invention to all of the multiple sorting stages, such as primary and secondary sorting, with threshold settings appropriate for each stage. Furthermore, such multiple-stage sorting using machine learning models can be performed consecutively during a single sorting process.

[0068] [Software implementation example] The control blocks of the determination device (learning device) 1 (particularly the acquisition unit 12, the determination unit 14, and the learning unit 16) may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software.

[0069] In the latter case, the determination device 1 includes a computer that executes instructions from a program, which is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium storing the program. The object of the present invention is achieved when the processor in the computer reads and executes the program from the recording medium. The processor may be, for example, a central processing unit (CPU). The recording medium may be a "non-transitory tangible medium," such as a read-only memory (ROM), tape, disk, card, semiconductor memory, or programmable logic circuit. The device may also include a random access memory (RAM) for loading the program. The program may be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). Note that one aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0070] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0071] Example 1 An example of the present disclosure is described below. Figure 5 shows information about the wheat samples used in this example and the following examples. In Figure 5, "target grain thickness fraction" indicates the range of target grain thickness. "Sample series" indicates the lot number of the corresponding "variety," and each sample is obtained under different conditions. All varieties are domestically cultivated varieties, and the varieties are distinguished by alphabets (three wheat varieties, two barley varieties).

[0072] Furthermore, "Fusa head blight infection conditions" indicates whether the grains were naturally infected with Fusarium head blight ("natural infection"), or whether Fusarium head blight infection was artificially promoted by spraying an inoculum in the field. Furthermore, "fractional mycotoxin concentration" indicates the mycotoxin concentration per unit volume in the entire sample fraction from which the corresponding "sample series" wheat grain samples were sampled. The items shown to the right of the "fractional mycotoxin concentration" indicate the number of grains with each contamination concentration. Here, mycotoxin concentration is the sum of DON and NIV measured by ELISA.

[0073] Furthermore, Figure 6 shows the images used as a data set in this example and the following examples, and shows the number of images for each contamination concentration of the wheat grain shown in Figure 5. As shown in Figures 5 and 6, for each wheat grain in Figure 5, one to three images correspond to the front and back sides of the wheat grain. All of the images were taken with a general digital camera.

[0074] Figure 6 shows, for example, that for images of wheat variety A, a total of 149 images of the front side and 148 images of the back side were prepared for each contamination concentration, and for images of barley above 5 ppm, a total of 39 images of the front side and 39 images of the back side were prepared for each sample series. Note that some of the images of the wheat grains for each contamination concentration were set aside in advance as test images to serve as a test dataset, separate from the images used for training (the images for training and testing were derived from different wheat grains). During training, the number of images between the classification classes was roughly the same or within a ratio of about 1:2, so that the data for classes with fewer images was amplified by flipping the data left and right, and the number of images for the other class was appropriately reduced.

[0075] In a determination device similar to the determination device 1 described above in the above embodiment, a neural network using AlexNet was trained using image data classified into highly contaminated barley grains exceeding 5 ppm and non-contaminated grains (ND, i.e., approximately 0 ppm), and the contamination level of each barley as training data. In this example and the examples described below, two-rowed hulled barley was used as the barley, but this is not limited thereto; similar processing can be applied to other types of barley and even other granular agricultural products such as grains. During training, the training data was divided into training data and verification data at an 8:2 ratio, and the training images were subjected to data augmentation by shifting their positions and flipping them left and right.

[0076] Figure 7 shows a screen displaying the above learning process. In Figure 7, "loss" corresponds to the value of the loss function in learning, which converges to a value close to 0 by repeating learning. Also, in Figure 7, "learning rate" refers to the degree to which each value of the parameter set that defines the neural network is adjusted in one learning session.

[0077] As shown in Figure 7, training on barley data divided into mini-batches for each epoch was repeated 16 times, and training was performed up to 10 epochs before testing. The results showed that the verification accuracy using the validation data during training was 92%, and the test accuracy for the binary choice between highly contaminated and non-contaminated grains in the test dataset (images of highly contaminated grains and non-contaminated grains that had been separated in advance from the training images) was 87% (27 / 31).In addition, the percentage of test images of highly contaminated grains with a concentration of more than 5 ppm that were accurately estimated to be highly contaminated with a probability of 90% or more (correct answer rate) was 61% (11 / 18).

[0078] Similarly, training was conducted separately up to 40 epochs, resulting in a verification accuracy of 97%, a test accuracy of 84% (26 / 31), and a correct answer rate of 67% (12 / 18) for the above-mentioned judgment results of highly contaminated particles.

[0079] In addition, a neural network using AlexNet was trained using training data from image data classified into highly contaminated grains (over 5 ppm) and low-contaminated grains (0.05-0.5 ppm), which is the acceptable level, and the contamination level of each barley. The training data was divided into mini-batches for each epoch and repeated 16 times, up to 40 epochs. The results were a validation accuracy of 90%, a test accuracy of 86% (38 / 44) for the binary choice between highly contaminated and low-contaminated grains in the test dataset (highly contaminated and low-contaminated grain images separated from the training images), and a rate of highly contaminated grains (over 5 ppm) that were accurately identified as highly contaminated with a probability of 90% or higher (correct answer rate) of 72% (13 / 18).

[0080] As mentioned above, even though it is more difficult to distinguish mycotoxin-contaminated grains by appearance with barley than with wheat, it has been confirmed that it is possible to distinguish highly contaminated grains from non-contaminated grains or grains with low (acceptable) levels of contamination with an accuracy of over 80%.

[0081] Example 2 A second example of the present disclosure will be described below. In this example, a neural network using AlexNet was trained a sufficient number of times using training data including image data classified into highly contaminated barley grains (over 5 ppm) and non- to lowly contaminated barley grains (under 0.5 ppm) from the image data shown in Fig. 6, and the contamination level of each barley grain.

[0082] In addition, a total of 61 images were used as the test dataset (which simulates actual mycotoxin-contaminated samples and includes images of contaminated grains outside the two extreme mycotoxin concentration ranges used for training): 18 images of barley with more than 5 ppm contaminated grains, 17 images of barley with 1-5 ppm contaminated grains (slightly more contaminated grains), 12 images of barley with 0.05-1 ppm contaminated grains, and 14 images of uncontaminated barley.

[0083] Explanatory diagram 56 in Figure 8 shows the test results of determining whether or not the above test data set is a "highly contaminated grain." In addition, in explanatory diagram 56 and each of the following figures, "high" corresponds to a highly contaminated grain exceeding 5 ppm, and "not" corresponds to any other type of wheat grain. Furthermore, "true" means whether or not the grain is actually highly contaminated, and "estimated" means the determination result by the determination device. In the determination, if the target barley grain is estimated to be a highly contaminated grain exceeding 5 ppm with a probability of 50% or more, the barley grain is determined to be a highly contaminated grain. In other words, the probability threshold for determining a highly contaminated grain is set to 50%.

[0084] As shown in Figure 56, of the images of barley grains that the judgment device judged to be highly contaminated, 8 were correct and 2 were false positives. Of the images of barley grains judged to be otherwise, 10 were false positives and 41 were correct. This resulted in a test accuracy (percentage of correct answers) of 80%. This result means that when this judgment device was used to sort and remove highly contaminated grains (images) from a test dataset containing images of grains with various levels of contamination, 8 of 18 highly contaminated grain images were successfully sorted and removed, and 41 of 43 images of grains with a contamination level of less than 5 ppm were retained without being removed. Although there were 2 false positives, both of these were images of grains with a slightly high concentration of contamination (1 to 5 ppm), and these grains could also be removed without any problems.

[0085] Moreover, explanatory figure 58 shows the judgment results when similar learning and judgment were performed using Xception as a neural network. In the above case, of the images of barley grains judged by the judgment device to be highly contaminated grains, 8 were correct and 3 were incorrectly detected. Of the images of barley grains judged to be otherwise, 10 were incorrectly judged and 40 were correct. This resulted in a test accuracy of 79%. In addition, 2 of the 3 incorrectly detected images in this result were images of grains contaminated with a slightly high concentration (1 to 5 ppm).

[0086] Moreover, explanatory diagram 60 shows the judgment results when similar learning and judgment were performed using VGG16 as a neural network. In the above case, of the images of barley grains that the judgment device judged to be highly contaminated grains, 8 images were correct and 9 images were incorrectly detected, and of the images of barley grains that were judged to be other than those, 10 images were incorrectly judged and 34 images were correct. As a result, the test accuracy was 69%.

[0087] Moreover, explanatory diagram 62 shows the judgment results when similar learning and judgment were performed using ResNet50 as a neural network. In the above case, of the images of barley grains that the judgment device judged to be highly contaminated grains, 16 images were correct and 16 images were incorrectly detected, and of the images of barley grains that the judgment device judged to be non- to low-contaminated grains, 2 images were incorrectly detected and 27 images were correct. As a result, the test accuracy was 70%.

[0088] As described above, in this embodiment corresponding to Figure 8, the learning results for image classification of barley grains highly contaminated with more than 5 ppm and non- to low-contaminated grains less than 0.5 ppm converged regardless of whether AlexNet, Xception, VGG16, or ResNet50 was used, and the test accuracy for determining highly contaminated grains in a test dataset containing images of grains with various mycotoxin contamination concentrations was 69 to 80%, with relatively high test accuracy being obtained when AlexNet or Xception was used.

[0089] Example 3 A third example of the present disclosure will be described below. In this example, a neural network using AlexNet was trained a sufficient number of times using approximately 176 image data pieces, each of which was classified into highly contaminated wheat grains (over 5 ppm) and non-low-contaminated wheat grains (under 0.5 ppm) from the image data shown in Figure 6, and the contamination level of each wheat grain as training data.

[0090] In addition, a total of 102 images were used as the test dataset (which simulates actual mycotoxin-contaminated samples and includes images of contaminated grains outside the two extreme mycotoxin concentration ranges used for training): 23 images of wheat grains with more than 5 ppm of mycotoxin contamination (highly contaminated grains), 29 images of wheat grains with 1 to 5 ppm of mycotoxin contamination (slightly more contaminated grains), 24 images of wheat grains with 0.05 to 1 ppm of mycotoxin contamination, and 26 images of uncontaminated wheat.

[0091] Graph 64 in Figure 9 shows the test results of determining whether the test data set is a "highly contaminated grain." However, in the determination shown in Graph 64, the probability threshold for determining whether a grain is a highly contaminated grain was set to 50%, as in Example 2 for barley. As a result, of the images of wheat grains determined by the determination device to be highly contaminated, 19 were correct and 21 were incorrectly detected. Of the images of wheat grains determined to be otherwise, 4 were incorrectly detected and 58 were correct. This resulted in a test accuracy of 75%. Note that 16 of the 21 incorrectly detected images were grains contaminated with a slightly high concentration of 1 to 5 ppm.

[0092] Furthermore, Figure 66 shows the results of a similar learning and judgment experiment using Xception as a neural network. The probability threshold for determining whether a grain is highly contaminated was set to 50% in the judgment shown in Figure 66. As a result, of the images of wheat grains that the judgment device determined to be highly contaminated, 21 were correct and 25 were incorrectly detected. Of the images of wheat grains that were otherwise determined, 2 were incorrectly detected and 54 were correct. This resulted in a test accuracy of 74%. Of the 25 incorrectly detected images, 17 were grains with a slightly higher concentration of contamination, between 1 and 5 ppm.

[0093] Furthermore, Figure 68 shows the results of the test when AlexNet was used as a neural network for learning and judgment, and the probability threshold for determining whether a grain was highly contaminated was set to 97% instead of 50%. In this case, of the images of wheat grains that the judgment device judged to be highly contaminated, 16 were correct and 8 were incorrectly detected. Of the images of wheat grains that were judged to be otherwise, 7 were incorrectly detected and 71 were correct. This resulted in a test accuracy of 85%. All of the 8 incorrectly detected images were grains with a slightly high concentration of contaminants, ranging from 1 to 5 ppm.

[0094] Furthermore, Figure 70 shows the results of the test when Xception was used as a neural network for learning and judgment, and the probability threshold for determining whether a grain was highly contaminated was set to 85% instead of 50%. In this case, of the images of wheat grains that the judgment device judged to be highly contaminated, 15 were correct and 5 were incorrectly detected. Of the images of wheat grains that were judged to be otherwise, 8 were incorrectly detected and 74 were correct. This resulted in a test accuracy of 87%. Of the 5 incorrectly detected images, 4 were grains with a slightly higher concentration of contaminants, ranging from 1 to 5 ppm.

[0095] That is, in this example (wheat), the accuracy of the determination could be improved by adjusting the probability threshold for determining that the grain is highly contaminated.

[0096] Example 4 A fourth embodiment of the present disclosure will be described below. In this embodiment, a configuration using a neural network according to the orientation of barley grains in an image input to a determination device will be described. In this embodiment, the same training data and test data set as in Example 2 were used as described below. In addition, the probability threshold for determining that a grain is highly contaminated was set to 50%.

[0097] Figure 72 in Figure 10 shows the results of a test where an AlexNet-based neural network was trained a sufficient number of times using a mixture of images of the front and back sides of barley grains as training data, and the images of the front and back sides of the barley grains were used as the test data set. The test accuracy of the test shown in Figure 72 was 82%. One false positive image in Figure 72 was a contaminated grain with a slightly high concentration of 1 to 5 ppm. Figure 76 shows the results of the test where an image of the back side of the barley grains was used as the test data set in the above case. The test accuracy of the test shown in Figure 76 was 79%.

[0098] FIG. 74 shows the results of a neural network using AlexNet, which was trained a sufficient number of times using only images of the front side of barley as training data. The test accuracy of the results shown in FIG. 74 was 82%.

[0099] FIG. 78 shows the results of a neural network using AlexNet, which was trained a sufficient number of times using only images of the underside of barley as training data. The test accuracy of the results shown in FIG. 78 was 65%.

[0100] As shown above, in the results of training with AlexNet in Figure 10, the test accuracy for the front-side images was higher than the test accuracy for the back-side images in both cases where the training data included a mixture of front-side and back-side images of barley, and where only the front-side and back-side images were used as training data.

[0101] Moreover, diagram 80 in FIG. 11 shows the judgment results when a neural network using Xception was trained a sufficient number of times using a mixture of images of the front and back sides of barley as training data, and the images of the front side of the barley were used as the test data set. The test accuracy of the judgment shown in diagram 80 was 74%. Moreover, diagram 84 shows the judgment results when images of the back side of the barley were used as the test data set in the above case. The test accuracy of the judgment shown in diagram 84 was 83%.

[0102] Figure 82 shows the results of a neural network trained using Xception with only images of the front side of barley as training data, and the test data set was the front side of barley. The test accuracy of the results shown in Figure 82 was 63%.

[0103] Furthermore, Figure 86 shows the results of a neural network using Xception, which was trained a sufficient number of times using only images of the underside of barley as training data, when images of the underside of barley were used as the test data set. The test accuracy of the judgment shown in Figure 86 was 79%. Of the four false positive images in Figure 86, three were contaminated grains with a slightly high concentration of 1 to 5 ppm.

[0104] As mentioned above, in the results of training with Xception in Figure 11, when both images of the front and back of barley grains were mixed in the training data and when only images of the front and back of the grains were used as training data, the test accuracy for the back images was higher than the test accuracy for the front images, which was the opposite of the results when training with AlexNet in Figure 10. In other words, it was suggested that the accuracy of the classification can be improved by using neural networks that are suitable for the front and back of the grains.

[0105] Furthermore, Fig. 88 in Fig. 12 shows the results of a case where a neural network using AlexNet and a neural network using Xception were trained a sufficient number of times using a mixture of images of the front and back sides of barley as training data, and the images of the front side of the barley were judged using the neural network using AlexNet, and the images of the back side were judged using the neural network using Xception. Fig. 88 corresponds to the sum of the judgment results shown in Fig. 72 and Fig. 84. The test accuracy of the judgment shown in Fig. 88 was 82%. One false positive image in this result was an image of contaminated grains with a slightly high concentration of 1 to 5 ppm.

[0106] FIG. 90 shows the results of a neural network using AlexNet trained a sufficient number of times using only images of the front side of barley as training data, and a neural network using Xception trained a sufficient number of times using only images of the back side of barley as training data. The images of the front side of barley were judged using the neural network using AlexNet, and the images of the back side were judged using the neural network using Xception. FIG. 90 corresponds to the sum of the judgment results shown in FIG. 74 and FIG. 86. The test accuracy of the judgment shown in FIG. 90 was 80%. Three of the four false positive images in this result were images of contaminated grains with a moderately high concentration of 1 to 5 ppm.

[0107] Furthermore, explanatory diagrams 92 and 94 are used for comparison with the determination results shown in explanatory diagram 88. Explained diagram 92 shows the determination results when images of the front and back sides of barley are used as test data sets after a neural network using AlexNet is trained a sufficient number of times using a mixture of images of the front and back sides of barley as training data. Explained diagram 92 corresponds to the sum of the determination results shown in explanatory diagram 72 and explanatory diagram 76, and to explanatory diagram 56. The test accuracy of the determination shown in explanatory diagram 92 was 80%.

[0108] FIG. 94 shows the results of a test where images of the front and back of barley were used as test data sets after training a neural network using Xception a sufficient number of times using a mixture of training data (images of the front and back of barley). FIG. 94 corresponds to the sum of the results shown in FIG. 80 and FIG. 84, as well as FIG. 58. The test accuracy of the results shown in FIG. 94 was 79%.

[0109] As mentioned above, the results shown in Figure 12 are thought to be unclear due to the small number of images (samples) used for training and testing. However, when different networks were used to judge the images of the front and back of the barley, the test accuracy (80-82%) tended to be higher than the test accuracy (79-80%) when no distinction was made between the front and back.

[0110] Example 5 A fifth example of the present disclosure will be described below. In this example, the results of cross-validation performed on a barley sample series including part of the sample series used in Example 1 using a determination device similar to the determination device 1 described in the above embodiment will be described. Figure 13 shows information about the barley samples used as the training data and test data set in this example.

[0111] In Figure 13, "sample series number" indicates a number to distinguish each sample. Although some items differ between Figure 13 and Figure 5, the barley samples with sample series numbers 1 to 3 in Figure 13 correspond to the barley samples in sample series T1, T2, and U01 in Figure 5, respectively.

[0112] In Figure 13, "Sample series," "Variety," "Fusa blight infection conditions," and "Grain thickness" have the same meanings as the items in Figure 5. Also, the meaning of the items indicating the number of wheat grains at each contamination level for four rows from "Total number of grains in sample" is the same as in Figure 5, but in Figure 13, the contamination levels are distinguished using thresholds different from those in Figure 5.

[0113] In Figure 13, "original mycotoxin concentration estimated from data for each grain" indicates the mycotoxin concentration of the entire sample series, calculated backward from the mycotoxin concentration measured for each grain of barley included in the sample series. Furthermore, "digital camera model" indicates the model of digital camera used to capture images of the barley of each sample series number. For example, the images of barley of sample series numbers 4 and 5 were captured using the same model of digital camera Y. Furthermore, "number of images of front and back" indicates the number of images of the front and back of each barley grain used in the training of this example. During training, the training images were subjected to data augmentation, including positional shifting, horizontal flipping, enlargement / reduction, and color tone change.

[0114] The mycotoxin concentrations in barley sample series numbers 4 to 6 were measured under different conditions than those in barley sample series numbers 1 to 3. The mycotoxins in barley sample series numbers 1 to 3 were measured by ELISA, while the mycotoxins in barley sample series numbers 4 to 6 were measured by LC-MS / MS. Furthermore, in barley sample series numbers 4 to 6, DON-3-glucoside, a glycoside of DON that is sometimes measured separately from DON, was converted into DON to calculate the mycotoxin concentration (DON + NIV).

[0115] In this example, a six-fold cross-validation was performed in which training was performed using data from five of the six sample series, and validation and testing were performed using data from the remaining independent sample series. Images of highly contaminated barley grains (over 5 ppm) and images of non- to low-contaminated barley grains (less than 0.5 ppm) were used as training and validation data for two-class classification. However, as described below, the test dataset assumed the actual use of this technology, and images of barley grains with medium levels of contamination (0.5 to 5 ppm) were also used. Furthermore, during training, the number of images between the classification classes in each sample series was approximately the same or within a ratio of approximately 1:2. Data was amplified by left-right inversion for classes with fewer images, and the number of images in the other class was appropriately reduced. However, images of barley grains with levels of contamination less than 0.5 ppm were used without reduction for validation data.

[0116] 14 and 15 show the learning process when cross-validation was performed using barley images from sample series 1 to 6 in Fig. 13. In the cross-validation, a neural network using AlexNet was trained using training data and validation data, which were training data.

[0117] 14 and 15, graphs 101 to 106 show the accuracy during training and validation, respectively. Graphs 111 to 116 correspond to the loss function values ​​during training and validation, respectively. The horizontal axis of each graph represents the number of epochs.

[0118] Furthermore, in Figures 14 and 15, descriptions such as "23456train_1val" indicate combinations of sample series in cross-validation, and correspond to the "training_validation dataset" in Figure 16, etc., described below. For example, "23456train_1val" in Figure 14 indicates that learning was performed using images of sample series numbers 2 to 6 as training data and images of sample series number 1 as validation data. Furthermore, "13456train_2val" indicates that learning was performed using images of sample series numbers 1 and 3 to 6 as training data and images of sample series number 2 as validation data. Furthermore, other combinations of sample series in cross-validation are similarly described.

[0119] Figure 16 shows the results of the cross-validation shown in Figures 14 and 15. Details will be described later, but the accuracy, precision, recall, F1 value, etc. corresponding to each combination of sample series are listed with rounded values. In Figure 16, "TP" (True Positive) indicates the number of images of barley that were actually highly contaminated and that the judgment device judged as highly contaminated grains. "FP" (False Positive) indicates the number of images of barley that were actually not highly contaminated and that the judgment device judged as highly contaminated grains. "FN" (False Negative) indicates the number of images of barley that were actually highly contaminated and that the judgment device judged as not being highly contaminated grains. "TN" (True Negative) indicates the number of images of barley that were not actually highly contaminated and that the judgment device judged as not being highly contaminated grains. Larger values ​​of TP and TN are desirable, and smaller values ​​of FP and FN are desirable.

[0120] "Accuracy" indicates the percentage of times the judgment device correctly judges whether an object is a highly contaminated particle or not. The "accuracy" value is calculated using the formula (TP + TN) / (TP + FP + FN + TN). "Precision" indicates the percentage of images that the judgment device judged to be highly contaminated particles, but that were actually highly contaminated particles. The "precision" value is calculated using the formula (TP) / (TP + FP). "Recall" indicates the percentage of images that the judgment device judged to be highly contaminated particles, but that were actually highly contaminated particles. The "recall" value is calculated using the formula (TP) / (TP + FN). "F1 value" is a value that indicates the harmonic mean of the precision and recall. The "F1 value" is calculated using the formula (2 × precision × recall) / (precision + recall).

[0121] 17 and 18 show the judgment results when images serving as a test data set were given as input to a trained neural network trained in the above-mentioned cross-validation. Here, all grains from each sample series, i.e., one front and one back image each of highly contaminated barley grains exceeding 5 ppm, medium-contaminated grains of 0.5 to 5 ppm, and non- to low-contaminated barley grains of less than 0.5 ppm, were used as test data sets to perform two-class classification of whether the grains were highly contaminated or not.

[0122] In Figure 17, "Number of Test Images" indicates the number of barley images used in the test dataset. The barley sample series number used for the images in the test dataset is the same as the barley sample series number used for the validation data. For example, the trained neural network corresponding to the row where "Learning_Validation Dataset" is "23456train_1val" receives the barley image of sample series number 1 as the test dataset. The test dataset also includes one image of the front side and one image of the back side of each barley grain. For example, 108 test images corresponds to 54 grains of barley. Data augmentation, such as positional translation and left-right flipping, has not been performed on the images in the test dataset. Some items, such as "Number of Test Images," are also listed in Figure 18.

[0123] Furthermore, the "mycotoxin concentration estimated from individual grain data" indicates the mycotoxin concentration of the entire sample series, calculated backward from the mycotoxin concentration measured for each grain of barley in the validation and test datasets. Note that this value was calculated directly from the data for each grain (44–60 grains) in each sample series and may not necessarily match the mycotoxin concentrations of the sample fractions from which each sample was sampled (Figures 5 and 6) due to sampling error. The "probability threshold for determining high-concentration grains" indicates the threshold for determining the probability that a target grain image is estimated to be a highly contaminated grain. For example, if the probability threshold value is 0.5, the determination device will determine a grain image as a highly contaminated grain if the target grain image is estimated to be a highly contaminated grain with a probability of 50% or higher. The "number of images determined to be high-concentration grains" indicates the number of images determined to be highly contaminated grains by the determination device. "Concentration after sorting removal" indicates the mycotoxin concentration of the entire grain image remaining after removing images determined to be highly contaminated grains from all grain images (test dataset) of the sample series used for validation data. "Sorting yield" indicates the weight ratio (weight ratio) of the remaining grain images after removing images determined to be highly contaminated grains by the judgment device to the weight of the entire grain image before sorting. "Mycotoxin reduction rate" indicates the percentage reduction in mycotoxin concentration from the entire grain image of the sample series used for validation data after removing images determined to be highly contaminated grains from the entire grain image of the sample series (test dataset). In addition, the "average when sorting yield is approximately 80%" in the last line indicates the average value of each item when a sorting yield of approximately 80% is used, as shown in the underlined part. The results shown in Figure 17 show that, although there were differences depending on the combination of cross-validation datasets, an average reduction in mycotoxin concentration of approximately 50% was confirmed when a sorting yield of approximately 80% was used.

[0124] Figure 18 includes a section on the significance test for mycotoxin concentrations per particle between images of particles judged as highly contaminated and the remaining particle images excluding the highly contaminated particles in the test dataset of the sample series used for validation data. In Figure 18, "Average concentration per particle of highly contaminated particles" indicates the average mycotoxin concentration per particle for images of particles judged as highly contaminated in the test dataset of the sample series used for validation data. "Average concentration per particle after removing highly contaminated particles" indicates the average mycotoxin concentration per particle for the remaining particle images excluding images of highly contaminated particles in the test dataset of the sample series used for validation data. In the "Significance test," "***" indicates a significant difference at the 0.1% significance level, "**" indicates a significant difference at the 1% significance level, "*" indicates a significant difference at the 5% significance level, and "ns" indicates no significant difference at the 5% significance level.

[0125] Until now, there have been no effective postharvest sorting methods for barley other than grain thickness sorting and gravity sorting. However, the present example suggests that the identification device can remove highly contaminated grains difficult to distinguish with the naked eye from mycotoxin-contaminated barley samples after grain thickness sorting (here, grains 2.6 mm or larger), thereby reducing mycotoxin concentrations. As described above, in this example, despite the relatively small number of grains used as training data and images taken with a standard digital camera, the effectiveness of grain sorting was verified using a test dataset containing images of grains with various degrees of contamination derived from samples independent of the training data. While this example focused on barley, mycotoxins can also accumulate in wheat grains that appear healthy or have unclear damage. Therefore, even in mycotoxin-contaminated wheat samples containing such contaminated grains, the use of an identification device similar to this example may potentially improve the mycotoxin reduction effect of grain sorting. The identification device's performance is expected to improve with increased training data, consideration of the wavelength range constituting the images, and development and improvement of machine learning models.

[0126] Figure 19 also shows the relationship between the probability threshold for determining a grain as highly contaminated and the sorting yield for the test data set of each of the sample series combinations described above. The sorting yield here is a value in terms of the grain (image) number ratio, but for barley samples with a uniform grain thickness, this value can be considered as an approximation of the sorting yield in terms of weight ratio. Figure 19 also clearly shows the 80% sorting yield line described with reference to Figures 17 and 18.

[0127] 19 also shows that, for example, in the combination of sample series "13456train_2val" and "12456train_3val," the barley of sample series number 2 and the barley of sample series number 3, which are test data sets, have a sorting yield of 80% or more even when the probability threshold is set to 0.5 (equivalent to 50%). On the other hand, for example, in the combination of sample series number 4 and barley of sample series number 6, in order to achieve a sorting yield of 80% or more, the probability threshold needs to be set to 0.8 (80%) or more.

[0128] Furthermore, the relationship between the aforementioned probability threshold and the sorting yield (ratio of grain (image) numbers or weight ratio) can be calculated or estimated by the judgment device by referring to the barley grain images of each sample series. As an example, the sorting yield (ratio of grain (image) numbers) (%) is calculated using the formula (1-(number of images judged by the judgment device to be images of highly contaminated grains / number of images of barley grains in the target sample series)) x 100. Furthermore, by training each grain image with corresponding grain weight data, or data on the relative grain weight ratio based on the average grain weight of low-contamination grains or non-contaminated grains below a predetermined standard value in the same sample lot, it becomes possible to estimate each grain weight or relative grain weight ratio from each grain image being processed. Therefore, using this estimated value, the sorting yield (weight ratio) (%) can be calculated (estimated) using the formula: (1 - (total grain weight (estimated value) of each image that the judgment device judges to be a highly contaminated grain / total grain weight (estimated value) of all images of barley grains in the target sample series)) x 100, or (1 - (total value of relative grain weight ratio (estimated value) of each image that the judgment device judges to be a highly contaminated grain / total value of relative grain weight ratio (estimated value) of all images of barley grains in the target sample series)) x 100. Alternatively, as mentioned above, under specified conditions, the sorting yield based on the grain (image) number ratio and the sorting yield based on the weight ratio can be treated as approximate values ​​of each other.

[0129] For example, this judgment device can refer to images of a certain amount of grains extracted in advance from a barley sample lot that is to be sorted and removed from contaminated grains, and obtain information on the relationship between the probability threshold for estimating highly contaminated grains and the sorting yield (grain number ratio or weight ratio), thereby determining the probability threshold to be used for determining highly contaminated grains in the barley sample lot, using the corresponding sorting yield value as a judgment criteria.

[0130] If necessary, information on the relationship between the probability threshold and the sorting yield can be obtained separately for each grain orientation, such as the front side or back side, and different probability thresholds can be used to determine whether the grain is highly contaminated or not, depending on the grain orientation.

[0131] In addition, by accumulating data such as images of individual grains and mycotoxin concentrations from various sample series obtained under various conditions, the system can use these data for training purposes to improve the system's performance. Data on the relationship between probability thresholds and sorting yields for various sample series, as well as the corresponding mycotoxin reduction rates and mycotoxin concentrations after removal of highly contaminated grains, are also being updated and accumulated, as shown in Figure 19. By developing an estimation model using this data, a certain amount of grain extracted from a newly targeted barley sample can be referenced in the system (which provides information on the relationship between the probability threshold and sorting yield for that sample lot) to estimate the original mycotoxin concentration or contamination level of that sample lot, as well as the mycotoxin reduction rate corresponding to the selected probability threshold and sorting yield, and the mycotoxin concentration or contamination level after removal of highly contaminated grains. This information can be used to determine the probability threshold and sorting yield to be used to identify highly contaminated grains in that barley sample lot. In developing this estimation model, techniques such as generalized linear models, Bayesian inference, and machine learning can be used. Even for samples for which mycotoxin data for each grain is not necessarily available, data on mycotoxin concentrations before sorting of contaminated grains and after sorting using a predetermined probability threshold or sorting yield can be used to develop and improve the estimation model. Various additional information about each sample, such as information on the type of target crop (e.g., for barley, two-row barley, six-row barley, naked barley, etc.), variety, and various cultivation conditions (cultivation area, weather conditions, dominant fungal species, pest control conditions, etc.), can also be used to develop and improve the estimation model.

[0132] In Example 5, a neural network classification model is used in the determination device. However, when a regression model is used, the "probability threshold" used in this example as the determination threshold for high-concentration mycotoxin-contaminated granules can be replaced with the "estimated mycotoxin concentration." Other appropriate determination thresholds can also be used depending on the type of machine learning model used in the determination device. Machine learning models that do not have a determination threshold for determining whether a granule is highly mycotoxin-contaminated or not are not envisioned. For example, in a classification model that returns a unique determination result for each granule (only a binary result), the reference value for determining "high-concentration mycotoxin-contaminated granules," i.e., the threshold (e.g., 5 ppm) that is the lower limit of the mycotoxin concentration or contamination level of high-concentration mycotoxin-contaminated granules used in the model's training data, can be considered to be the determination threshold for that model. In this way, in many cases, the reference value for determining "high-concentration mycotoxin-contaminated granules" can be directly considered to be the determination threshold. [Explanation of symbols]

[0133] 1. Judgment device (learning device) 10 Control Unit 12 Acquisition Department 14 Judgment section 16 Learning Department 20 Memory section 22 Acting part 24 Filming Department 26 Measuring part

Claims

1. An acquisition unit that acquires an image including one or more grains of wheat; a judgment unit that uses a trained machine learning model that receives the image and outputs judgment results for each grain regarding the trichothecene mycotoxin concentration or degree of contamination in the grain of wheat or barley, and (1) judges whether each grain or all of the grain of wheat or barley is contaminated with trichothecene mycotoxins at or above a predetermined standard value or within a predetermined concentration range, or (2) judges the trichothecene mycotoxin concentration or degree of contamination in each grain or all of the grain of wheat or barley; A determination device comprising:

2. the acquisition unit acquires an image including a plurality of grains of wheat or barley; The determination unit (1) determines whether the whole grain of the wheat or barley is contaminated with trichothecene mycotoxins at or above a predetermined standard value or within a predetermined concentration range based on the determination of each grain of the wheat or barley, or (2) determines the overall trichothecene mycotoxin concentration or contamination level based on the determination of each grain of the wheat or barley.

2. The determination device according to claim 1.

3. 3. The determination device according to claim 1, wherein the acquisition unit acquires an image including one or more grains of barley.

4. The concentration or degree of contamination of the trichothecene mycotoxin is the sum of multiple substances including deoxynivalenol and nivalenol.

4. The determination device according to claim 1, wherein the determination device comprises: a first detecting means for detecting a first error;

5. The acquisition unit: An image including one or more grains extracted from a wheat lot having a trichothecene mycotoxin concentration in the range of 1.1 ppm to 7.8 ppm is obtained.

5. The determination device according to claim 1, wherein the determination device comprises: a first electrode;

6. A determination device described in any one of claims 1 to 5, characterized in that a neural network is used as the machine learning model.

7. A determination device described in any one of claims 1 to 6, characterized in that the image is an image composed of data in the visible light range.

8. The determination device described in Claim 7, characterized in that the image is an RGB image.

9. The determination unit determining whether or not each of the one or more grains of wheat or barley contained in the input image is contaminated with a predetermined standard value or more of trichothecene mycotoxins using a predetermined determination threshold value; The system further has a function of calculating or estimating the sorting yield, which is the percentage of the number or weight of grains that are determined to be not contaminated with trichothecene mycotoxins at or above the predetermined standard value, for each of the plurality of judgment threshold values.

9. The determination device according to claim 1, wherein the determination device comprises: a first electrode;

10. The determination unit determining whether or not each of the one or more grains of wheat or barley contained in the input image is contaminated with a predetermined standard value or more of trichothecene mycotoxins using a predetermined determination threshold value; The system further has a function of estimating the trichothecene mycotoxin reduction rate or the trichothecene mycotoxin concentration or degree of contamination of the entire remaining grain set for each of the plurality of judgment thresholds when grains judged to be contaminated with trichothecene mycotoxins at or above the predetermined standard value are removed from the grain set to be judged.

10. The determination device according to claim 1, wherein the determination device comprises:

11. An acquisition unit that acquires an image including one or more grains of wheat and barley, and concentration information indicating the trichothecene mycotoxin concentration or contamination level of each grain of wheat and barley; a learning unit that uses the set of image and concentration information acquired by the acquisition unit as training data to train a machine learning model that receives an image containing one or more grains of wheat or barley and outputs a determination result regarding the trichothecene mycotoxin concentration or contamination level in the grains of wheat or barley, for each grain; The learning unit An image including grains of a first class of wheat or barley whose trichothecene mycotoxin concentration or contamination level is equal to or greater than a predetermined standard value, and an image including grains of a second class of wheat or barley whose trichothecene mycotoxin concentration or contamination level is equal to or less than a predetermined threshold value set below the standard value or which is uncontaminated, are input into the machine learning model. A learning device characterized by:

12. The predetermined reference value is a value of 1 ppm or more, and the predetermined threshold value is a value of 0.5 ppm or less. The learning device according to claim 11 .

13. The learning unit The machine learning model is trained to output a probability value indicating whether or not a grain of wheat or the like included in an input image is contaminated with a trichothecene mycotoxin at a level equal to or greater than the predetermined reference value. The learning device according to claim 11 .

14. An acquisition step of acquiring an image including one or more grains of wheat or barley; A trained machine learning model is used to input the image and output a judgment result for each grain regarding the trichothecene mycotoxin concentration or degree of contamination in the grain of wheat or barley. This model (1) judges whether each grain or all of one or more grains of wheat or barley are contaminated with trichothecene mycotoxins at or above a predetermined standard value or within a predetermined concentration range, or (2) judges the trichothecene mycotoxin concentration or degree of contamination in each grain or all of one or more grains of wheat or barley. A determination method comprising:

15. An acquisition step of acquiring an image including one or more grains of wheat and barley, and concentration information indicating the trichothecene mycotoxin concentration or contamination level of each grain of wheat and barley; a learning step in which a machine learning model is trained using the set of image and concentration information acquired in the acquisition step as training data, to input an image containing one or more grains of wheat or barley and output a determination result regarding the trichothecene mycotoxin concentration or contamination level in the grains of wheat or barley, for each grain. Including, In the learning step, An image including grains of a first class of wheat or barley whose trichothecene mycotoxin concentration or contamination level is equal to or greater than a predetermined standard value, and an image including grains of a second class of wheat or barley whose trichothecene mycotoxin concentration or contamination level is equal to or less than a predetermined threshold value set below the standard value or which is uncontaminated, are input into the machine learning model. A learning method characterized by:

16. A control program for causing a computer to function as the determination device according to claim 1, the control program causing a computer to function as the acquisition unit and the determination unit.

17. A control program for causing a computer to function as the learning device according to claim 11, the control program causing a computer to function as the acquisition unit and the learning unit.

Images