Method for generating agricultural product state determination model, agricultural product state determination device, and agricultural product state determination method

Abstract

To provide a method for generating an agricultural product state determination model, an agricultural product state determination device, and an agricultural product state determination method with which accurate selection can be performed.SOLUTION: A method for generating an agricultural product state determination model includes the steps of: generating three-dimensional shape data obtained by three-dimensionally constructing a known agricultural product image obtained by picking up an image of a known agricultural product in a known state; creating a plurality of two-dimensional known agricultural product images on the basis of the three-dimensional shape data; and performing machine learning using learning data that uses the two-dimensional known agricultural product images as input data and the state of the known agricultural product as output data, to generate an agricultural product state determination model.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method for generating a crop condition determination model, a crop condition determination device, and a crop condition determination method. [Background technology]

[0002] Agricultural crops are often grown in a natural environment, and various shapes and blemishes occur. For this reason, agricultural crops, such as sweet potatoes for fresh produce, are graded according to shape, size, weight, etc. at the stage of shipping, and are also judged for the shape of the tuberous root, the presence or absence of pests, damage, peeling, bruising, and other blemishes, and are then sorted according to the presence or absence of blemishes.

[0003] Furthermore, Japanese sweet potatoes are highly regarded overseas due to their high quality and excellent taste, and many are exported from Japan to other countries. Many of the sweet potatoes exported overseas rot during sea transport in winter, and in some cases, approximately 25% of the sweet potatoes rot. In order to prevent sweet potatoes from rotting during transport, for example, it is effective to select sweet potatoes that are at high risk of spoilage and are not suitable for long-term storage or export before the start of transport and exclude them from export.

[0004] Sweet potatoes to be sorted that are at high risk of spoilage due to damage are sorted, for example, by visual inspection by an operator. The operator sorts the sweet potatoes that he or she visually inspects based on his or her own experience, for example, but sorting requires skill based on a wealth of experience. In addition, since the operator needs to distinguish the sweet potatoes to be sorted from among the sweet potatoes transported in large quantities, it is difficult to sort them quickly and accurately.

[0005] On the other hand, there is a technique for selecting agricultural products to be selected, for example, by capturing an external image of the agricultural product being transported by a conveyer, evaluating the external quality of the agricultural product based on the captured external image, and selecting the agricultural product based on the evaluation result (see, for example, Patent Document 1). By using this technique, agricultural products are selected without relying on the worker's visual inspection, so that the agricultural product can be selected regardless of the worker's level of skill, etc. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-151613 A Summary of the Invention [Problem to be solved by the invention]

[0007] In order to evaluate and sort agricultural products based on the appearance images of the agricultural products, for example, it is possible to use a trained model generated using training data. In order to generate a trained model that can sort agricultural products with high accuracy, a large amount of training data is required. However, it is difficult to prepare a large amount of training data obtained by photographing agricultural products. For this reason, it is difficult to generate a trained model for sorting agricultural products with high accuracy.

[0008] Therefore, the problem that the present invention aims to solve is to provide a method for generating a crop condition determination model that enables accurate selection, a crop condition determination device, and a crop condition determination method. [Means for solving the problem]

[0009] A method for generating a crop condition determination model that solves the above-mentioned problem includes the steps of: generating three-dimensional shape data by constructing in three dimensions known crop images obtained by imaging known crops whose conditions are known, generating a plurality of two-dimensional images of known crops based on the three-dimensional shape data, and performing machine learning using learning data in which the two-dimensional images of the known crops are input data and the conditions of the known crops are output data, thereby generating a crop condition determination model.

[0010] The known crop images may include images of the known crop captured from a plurality of different directions.

[0011] The three-dimensional shape data may be generated by three-dimensionally constructing the known agricultural crop image converted into a point cloud.

[0012] The known agricultural crop may also include sweet potato.

[0013] In addition, the status may include at least one of a shape status relating to the shape of the known agricultural product, a damage level-related status relating to determining damage to the known agricultural product, or a shipping-related status relating to whether or not the known agricultural product can be shipped as a commercial product.

[0014] The condition may also include at least one of the following that occurs in the known agricultural crop: "grooves," "holes," "lenticels," "breaks," "peeling," "gouging," "pins," "skin veins," or "bruises."

[0015] In addition, the crop condition determination device that solves the above problem is a crop determination device that includes an acquisition unit that acquires a target crop image, which is an image of the target crop whose condition is to be determined, and a determination unit that determines the condition of the target crop based on output data obtained by inputting the acquired target crop image into a crop condition determination model generated by the above-mentioned crop condition determination model generation method.

[0016] The system may further include an imaging device that captures an image of the target crop, and the imaging device may transmit the captured image of the target crop to the acquisition unit.

[0017] The agricultural production system may further include a conveyor that transports the target crop, and the imaging device may capture an image of the target crop transported by the conveyor.

[0018] In addition, the crop condition determination method that solves the above problem is a crop condition determination method that acquires a target crop image, which is an image of a target crop whose condition is to be determined, and determines the condition of the target crop based on output data obtained by inputting the acquired target crop image into a crop condition determination model generated by the above-mentioned crop condition determination model generation method. Effect of the Invention

[0019] According to the cultivated crop harvesting method and harvesting apparatus of the present invention, it is possible to provide a method for generating a crop condition determination model, a crop condition determination apparatus, and a crop condition determination method that enable accurate selection. [Brief description of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating an example of a configuration of a state determination model generating device 1. FIG. [Diagram 2] 10 is a flowchart illustrating an example of a procedure of a method for generating a state determination model. [Diagram 3] FIG. 2 is a diagram showing an example of the contents of first model generation information 31. [Figure 4] 13 is a diagram showing an example of the contents of second model generation information 32. FIG. [Diagram 5] FIG. 2 is a diagram conceptually illustrating the function of a surface diagnostic model 151. [Figure 6] FIG. 2 is a diagram conceptually illustrating the function of a shape diagnostic model 152. [Figure 7] FIG. 2 is a diagram showing an example of the configuration of a sorting device 2. [Figure 8] FIG. 1 is a diagram showing the process from harvesting to shipping of harvested sweet potato HSP. [Figure 9] 4 is a flowchart showing an example of a process of the sorting device 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, a method for generating a crop condition determination model, a crop condition determination device, and a crop condition determination method according to the present invention will be described with reference to the drawings.

[0022] The method for generating a crop determination model according to the embodiment is a method for generating a trained model (hereinafter, a state determination model) for determining the state of a crop, for example, a sweet potato. The state determination model generating device generates a crop state determination model using a known crop image obtained by capturing an image of a known crop whose state is known. The state of the known crop is known, for example, by visual inspection by a person who performs the task of generating the state determination model (hereinafter, a generator).

[0023] The crop state determination device is a device that determines the state of the crop using a state determination model generated by a state determination model generation device. In the following, the crop is assumed to be a harvested sweet potato, and the state determination model generation device and the method for generating the crop determination model are described, followed by the description of the crop state determination device and the crop state determination method. Note that the crop to be subjected to state determination may be a crop other than sweet potato, and may be, for example, vegetables such as root vegetables and fruit vegetables, or fruit trees.

[0024] The state of the agricultural product includes, for example, at least one of a shape state related to the shape of the agricultural product, a damage level related state related to the determination of damage to the agricultural product, and a shipping related state related to whether the agricultural product can be shipped as a commodity. In the embodiment, whether the agricultural product can be shipped as a commodity is determined based on the shape and the level of damage to the agricultural product.

[0025] (Method of generating crop identification model) 1 is a diagram showing an example of the configuration of a state judgment model generating device 1. The state judgment model generating device 1 includes, for example, an imaging device 10 and a control device 20. The imaging device 10 includes, for example, a swivel device 11, a mounting base 12, a camera 13, and a lighting device 14. The swivel device 11 is movable in a first direction W1 about a vertical axis. The mounting base 12 is mounted on the swivel device 11 and rotates in accordance with the rotational operation of the swivel device 11. The swivel device 11 may rotate based on a rotation drive signal from the control device 20, or may rotate by operating a switch provided on the swivel device 11.

[0026] A sweet potato SP to be imaged is placed on the placement table 12. The sweet potato SP placed on the placement table 12 can be replaced by, for example, rotating it in a second direction W2 about a horizontal axis by the producer's hand. A driving device such as a robot arm that rotates the sweet potato SP placed on the placement table 12 in the second direction W2 may be provided, and the sweet potato SP may be rotated in the second direction W2 by the driving device.

[0027] The camera 13 is disposed, for example, at a position staring at the sweet potato SP on the mounting table 12, and is capable of capturing an image of the sweet potato SP. The camera 13 captures an image of the sweet potato SP based on an imaging signal transmitted by the control device 20. The camera 13 transmits the captured image of the sweet potato SP to the control device 20. The camera 13 includes images of the sweet potato SP captured from a plurality of different directions.

[0028] The lighting device 14 is disposed above the sweet potato SP on the mounting table 12, and is turned on based on an illumination signal transmitted by the control device 20 when the sweet potato SP is photographed, to illuminate the sweet potato SP on the mounting table 12. The lighting device 14 is configured, for example, with a plurality of LEDs arranged in an array. The camera 13 and the lighting device 14 may be movable, and for example, the camera 13 and the lighting device 14 may cooperate with each other around a horizontal axis perpendicular to the rotation axis in the second direction W2, or may be independently rotatable.

[0029] Instead of the embodiment in which camera 13 is rotatable, multiple cameras 13 may be installed at multiple positions to which camera 13 moves when rotating (multiple positions around a horizontal axis perpendicular to the rotation axis in the second direction W2) and used simultaneously. In this case, the multiple cameras 13 may each be connected to control device 20 via a Universal Serial Bus (USB). By installing multiple cameras 13 at multiple positions around a horizontal axis perpendicular to the rotation axis in the second direction W2, the operation of rotating camera 13 (flip operation) can be reduced.

[0030] The control device 20 includes, for example, an input unit 21, a storage unit 22, and a control unit 23. The input unit 21 is a device that can be operated by the creator, such as a keyboard. The input unit 21 may be, for example, a mouse or a touch panel. The touch panel has the functions of an input unit and a display unit that displays information. The input unit 21 accepts instructions according to operations by the creator and outputs the instructions to the control unit 23.

[0031] The storage unit 22 is configured, for example, by a hard disk drive (HDD) or a flash memory. The storage unit 22 may be a drive device for DVD or CD-ROM, and may be configured by loading a storage medium such as a DVD or CD-ROM into a reading device of the drive device. The storage unit 22 stores model generation information for generating a condition determination model and a condition determination model generated by the control unit 23. The model generation information includes first model generation information for generating a scratch condition model and second model generation information for generating a shape diagnosis model.

[0032] The control unit 23 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). The control unit 23 includes, for example, a drive control unit 24 and a model generation unit 25. The drive control unit 24 drives and controls the imaging device 10 in response to an instruction from an operator that is received by the input unit 21 and output.

[0033] For example, in response to a rotation drive instruction from the operator, the drive control unit 24 transmits a rotation drive signal to the turning device 11 to rotate the turning device 11. In response to an image capture instruction from the operator, the drive control unit 24 transmits an image capture signal to the camera 13 and transmits an illumination on signal to the lighting device 14 to cause the camera 13 to capture an image of the sweet potato SP while turning on the lighting device 14.

[0034] The model generation unit 25 generates a condition determination model including a surface diagnosis model and a shape diagnosis model based on a generated image generated from a captured image of the sweet potato SP transmitted by the camera 13 and model generation information stored in the storage unit 22. The surface diagnosis model is a model that diagnoses the surface of the sweet potato SP and determines the presence and type of scratches on the surface. The shape diagnosis model is a model that determines the three-dimensional shape of the sweet potato. The procedure for generating the condition determination model will be described later. The model generation unit 25 stores the generated condition determination model in the storage unit 22.

[0035] Next, a procedure for generating a state judgment model will be described. Fig. 2 is a flow chart showing an example of the procedure for a method for generating a state judgment model. When generating a state judgment model, first, a generator makes a plan for generating three-dimensional shape data of a sweet potato, and determines an imaging direction of a two-dimensional image of the sweet potato required for generating three-dimensional shape data (step S101).

[0036] When generating the three-dimensional shape data of the sweet potato SP, the producer visually checks the surface of the sweet potato SP and generates the confirmed state of the sweet potato SP as model generation information by linking it to the ID assigned to the individual sweet potato SP. The producer operates the input unit 21 of the control device 20 to store the generated model generation information in the storage unit 22.

[0037] For example, the generator checks the state of blemishes on the sweet potato SP to generate first model generation information 31, and classifies the shape of the sweet potato SP to generate second model generation information 32. Every time the generator performs work to generate three-dimensional shape data for each sweet potato SP, the generator visually checks the blemishes and shape of the sweet potato SP and updates the first model generation information 31 and the second model generation information 32.

[0038] 3 is a diagram showing an example of the contents of the first model generation information 31. The first model generation information 31 has an individual ID given to each individual of the sweet potato SP, and information on the presence or absence of a blemish on the surface of the sweet potato SP linked to each individual ID. Types of blemishes on the surface of the sweet potato SP included in the first model generation information 31 include, for example, symptoms such as "stripes," "holes," "lenticels," "breaks," "peeling," "gouges," "spikes," "skin veins," and "bruises."

[0039] 4 is a diagram showing an example of the contents of the second model generation information 32. The second model generation information 32 has an individual ID given to each individual of the sweet potato SP, and information on the shape of each sweet potato SP linked to each individual ID. The shapes classified in the second model generation information 32 include "salable shape," "rod shape," "round shape," and "abnormal shape."

[0040] A "saleable shape" is the general shape of sweet potato SP, for example, a long shape that is thick in the middle. A "rod-shaped shape" is a shape that is thinner in the middle and thinner overall, becoming rod-like, compared to a "saleable shape". A "round shape" is a shape that is short and generally closer to a sphere. An "abnormal shape" is a deformed shape that cannot be classified as a "saleable shape", "rod-shaped shape", or "round shape", and there are various shapes. Shapes other than the "saleable shape" are shapes that are not suitable for sale as sweet potato SP.

[0041] Next, the generator operates the input unit 21 to input an image capturing instruction, and captures an external image of the sweet potato SP with the imaging device 10 (step S103). The imaging device 10 transmits the captured image of the external image of the sweet potato SP to the control device 20. Next, the generator determines whether or not the entire peripheral image of the sweet potato planned in step S101 has been captured (step S105).

[0042] If it is determined that the panoramic image has not been captured, the generator inputs a rotation drive command to the input unit 21, rotates the rotation device 11 in the first direction W1, or rotates the sweet potato SP in the second direction W2, thereby adjusting the capture direction of the sweet potato SP (step S107), and returns the process to step S103 to capture the panoramic image of the sweet potato SP. Thereafter, the processes of steps S103 to S107 are repeated until the panoramic image is captured. The determination as to whether the panoramic image of the sweet potato SP has been captured may be performed by the control device 20.

[0043] In step S105, if the generator determines that the panoramic image has been captured, the control device 20 generates three-dimensional shape data by constructing the generated image of the sweet potato SP in three dimensions in the model generation unit 25 in accordance with the generator's instructions (step S109). The three-dimensional shape data of the sweet potato SP is generated as a point cloud image that is converted into a point cloud by a virtual camera on the metaverse, for example. Next, the model generation unit 25 generates an avatar of the sweet potato SP based on the generated point cloud image (step S111).

[0044] Next, the model generation unit 25 generates a two-dimensional image (hereinafter, generated image) of the sweet potato SP by rendering the generated three-dimensional shape data (step S113). The generated image is a two-dimensional image generated by rendering the three-dimensional shape data. The generated image is an example of a known agricultural crop two-dimensional image. Next, the model generation unit 25 determines whether a sufficient number of generated images have been generated to generate a state determination model (step S115).

[0045] If it is determined that a sufficient number of generated images for generating a state judgment model have not been generated, the model generation unit 25 returns the process to step S113 and repeats the processes of steps S113 to S115 until a sufficient number of generated images for generating a state judgment model are generated. Since the model generation unit 25 can generate different generated images simply by slightly changing the rendering portion of the three-dimensional shape data, the model generation unit 25 can easily generate a large number of generated images.

[0046] In step S115, if it is determined that a sufficient number of generated images have been generated to generate a state determination model, the model generation unit 25 performs machine learning using the generated generated images and the model generation information stored in the storage unit 22 to generate a state determination model (step S117). The model generation unit 25 generates a surface diagnosis model 151 and a shape diagnosis model 152 as the state determination model (step S119). In this manner, the state determination model is generated.

[0047] The surface diagnosis model 151 and the shape diagnosis model 152 will be described below. Fig. 5 is a diagram conceptually showing the function of the surface diagnosis model 151. The surface diagnosis model 151 includes, for example, an input layer, an intermediate layer, and an output layer. Input data is input to the input layer, and output data is output from the output layer. The intermediate layer generates output data output from the output layer based on the input data input to the input layer.

[0048] The input data is, for example, a plurality of generated images (generated image A, generated image B, generated image C, ... generated image N) generated by rendering three-dimensional shape data. The output data is, for example, information on the presence or absence of "stripes," "holes," "grains," "folds," "peeling," "gouges," "pins," "veins," or "bruises" included in the first model generation information. The intermediate layer has, for example, a multi-layered neural network connecting the input layer and the output layer.

[0049] FIG. 6 is a diagram conceptually illustrating the function of the shape diagnostic model 152. The shape diagnostic model 152 includes an input layer, an intermediate layer, and an output layer, similar to the surface diagnostic model 151. The input data input to the input layer is, for example, a plurality of generated images generated by rendering three-dimensional shape data similar to the surface diagnostic model 151, and the data output from the output layer is, for example, any one of the shapes "salable shape", "rod shape", "round shape", and "abnormal shape" included in the second model generation information. The intermediate layer has, for example, a multi-layered neural network connecting the input layer and the output layer, similar to the surface diagnostic model 151.

[0050] A specific example of generating the surface diagnostic model 151 will be described below. First, the generator captured images of 263 individual sweet potatoes SP and generated a three-dimensional model for each individual. Next, the generator memorized the symptoms observed in each of the 263 individuals. Note that the symptoms observed in one individual often included multiple types.

[0051] Next, the producer generated a surface diagnosis model 151 for symptoms that can be seen all around the sweet potato SP. The surface diagnosis model 151 may output all symptoms, but in order to prevent the processing volume from becoming too large and the costs from increasing, the surface diagnosis model 151 was generated by focusing on one symptom, specifically, the "lenticels."

[0052] The number of generated images used as learning data (input data) for each individual was 648. The generated images were rotated 180° in 10° increments in the first direction W1, and 360° in 10° increments in the second direction W2, and images were captured at each angle position, and the images were rendered. Of the 263 individuals, 35 individuals had lenticels and 228 individuals did not. In addition, some images of sweet potatoes SP with lenticels and some images of sweet potatoes SP without lenticels were used, and about 6,000 images each with and without lenticels were used to reduce memory usage.

[0053] The neural network in the surface diagnosis model 151 was a CNN (Convolutional Neural Network) network. First, when it was trained with VGG16, over-learning occurred, so a network with a shallower phase, consisting of seven convolution layers and three fully connected layers, was created to generate the surface diagnosis model 151. Here, generated images equivalent to 80% of all generated images were used as training data, and the remaining 20% ​​were used as test data to perform a confirmation test using the surface diagnosis model 151, and the accuracy rate for the test data used was 90%.

[0054] In order to determine whether a sweet potato SP has a blemish, a visual check of the surface of the sweet potato SP may be used. For example, an image determined to have a blemish may be processed using Grad-CAM (gradient weighted class activation mapping) to determine whether the blemish is present in a particular area, and the luminance of the area may be different from that of other areas. By visually observing this difference, the area where the blemish is present may be visually checked.

[0055] For the shape diagnosis model 152, diagnosis can be performed more simply than for the surface diagnosis model 151, so the resolution of the input data may be set lower than the resolution of the input data used to generate the surface diagnosis model 151. In addition, when input data of about 5,000 sheets was used, the shape diagnosis model 152 was expected to have an accuracy of 90% or more.

[0056] (Crop condition determination device) Next, a crop condition determination device (hereinafter, condition determination device) and a crop condition determination method will be described. Fig. 7 is a diagram showing an example of the configuration of a sorting device 2. The sorting device 2 includes, for example, a conveyor 50, a first camera 61, a second camera 62, a third camera 63, a PoE (Power over Ethernet) hub 70, and a condition determination device 100. The sorting device 2 may function as the condition determination device.

[0057] The conveyor 50 sequentially transports a plurality of harvested sweet potatoes (hereinafter, harvested sweet potatoes) HSP. The harvested sweet potatoes HSP are brought in to the base end of the conveyor 50. The brought in harvested sweet potatoes HSP are sequentially loaded onto the conveyor 50 by, for example, a worker and transported. The middle and end of the conveyor 50 serve as a sorting section. In the sorting section, the harvested sweet potatoes HSP are sorted by the worker according to the presence or absence of damage and the type.

[0058] The first camera 61, the second camera 62, and the third camera 63 are, for example, a conveyor 50, and are provided at positions staring at the harvested sweet potatoes HSP transported by the conveyor 50. The first camera 61, the second camera 62, and the third camera 63 capture images of the harvested sweet potatoes HSP transported by the conveyor 50 from different positions. The first camera 61, the second camera 62, and the third camera 63 are connected to the state determination device 100 via the PoE hub 70.

[0059] The PoE hub 70 is connected, for example, wirelessly or by wire, to the first camera 61, the second camera 62, the third camera 63, and the status determination device 100. The first camera 61, the second camera 62, and the third camera 63 transmit captured images including the captured harvested sweet potato HSP to the status determination device 100 via the PoE hub 70. The first camera 61, the second camera 62, and the third camera 63 are examples of imaging devices. The harvested sweet potato HSP is an example of a target crop whose status is determined by the status determination device 100.

[0060] The first camera 61, the second camera 62, and the third camera 63 may be connected to the state determination device 100 without going through the PoE hub 70. In this case, the first camera 61, the second camera 62, and the third camera 63 may be directly connected to the state determination layer 100 by wiring alone, or may be connected to the state determination device 100 via USB. The number of cameras that capture the harvested sweet potato HSP is three, the first camera 61, the second camera 62, and the third camera 63, but may be increased or decreased depending on the purpose of the capture. For example, when the harvest scale is large, the harvested sweet potato HSP to be captured is large in quantity, or the transport speed of the harvested sweet potato HSP is high, the number of cameras that capture the harvested sweet potato HSP may be further increased.

[0061] The state determination device 100 includes, for example, a communication interface 110, an input interface 120, a display 130, a processing circuit 140, and a storage unit 150. The communication interface 110 is an interface capable of communicating with the first camera 61, the second camera 62, and the third camera 63 via the PoE hub 70, for example, wirelessly or by wire. The communication interface 110 receives captured images transmitted by the first camera 61, the second camera 62, and the third camera 63. The communication interface 110 outputs the received captured images to the processing circuit 140. The captured images including the image of the harvested sweet potato HSP are an example of a target crop image.

[0062] The input interface 120 is realized by, for example, a mouse, a keyboard, a touch panel, a drag ball, a switch, a button, a joystick, a camera, an infrared sensor, a microphone, etc. The input interface 120 may be realized by a display device (for example, a tablet terminal) capable of wireless communication with the state determination device 100.

[0063] The display 130 displays various information. The display 130 is, for example, a liquid crystal display, a CRT (Cathode Ray Tube), an organic EL (Electroluminescence) display, etc. The display 130 may be a desktop type, or may be a display device (for example, a tablet terminal) capable of wireless communication with the processing circuit 140.

[0064] The processing circuit 140 includes, for example, an acquisition unit 141 and a determination unit 142. The processing circuit 140 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium (non-transient storage medium) such as a DVD or a CD-ROM, and may be installed by mounting the storage medium in a drive device.

[0065] The acquisition unit 141 acquires captured images including the harvested sweet potato HSP that are transmitted by the first camera 61, the second camera 62, and the third camera 63 and output by the communication interface 110. The acquisition unit 141 performs image processing on the acquired captured images, and extracts sweet potato images that are images of the harvested sweet potato HSP included in the captured images.

[0066] The first camera 61, the second camera 62, and the third camera 63 may capture the same harvested sweet potato HSP. The acquisition unit 141 identifies sweet potato images that capture the same harvested sweet potato HSP among the multiple harvested sweet potato HSPs captured by the first camera 61, the second camera 62, and the third camera 63.

[0067] The determination unit 142 determines the state of the harvested sweet potato HSP using the sweet potato image acquired by the acquisition unit 141 and the state determination model generated by the state determination model generating device 1. For example, the determination unit 142 determines the state of damage to the harvested sweet potato HSP by inputting the sweet potato image as input data to the surface diagnosis model 151. For example, the determination unit 142 determines the shape of the harvested sweet potato HSP by inputting the sweet potato image as input data to the shape diagnosis model 152.

[0068] The display control unit 143 controls the display 130. For example, the display control unit 143 causes the display 130 to display the sweet potato image acquired by the acquisition unit 141 and the captured image including the harvested sweet potato HSP. Furthermore, the display control unit 143 causes the display 130 to display the damage and shape of the harvested sweet potato HSP determined by the determination unit 142 together with the sweet potato image and the captured image.

[0069] The storage unit 150 is configured, for example, by a HDD, a flash memory, etc. The storage unit 150 may be a drive device for DVD or CD-ROM, and may be configured by loading a storage medium such as a DVD or CD-ROM into a reading device of the drive device. The storage unit 150 stores a surface diagnosis model 151 and a shape diagnosis model 152 generated by the condition determination model generating device 1.

[0070] In the sorting device 2, an operator sorts the harvested sweet potatoes HSP at the end of the conveyor 50. The display 130 is provided, for example, near the end of the conveyor 50 and is located in a position that is easy for the operator to see. The operator sorts the harvested sweet potatoes HSP while looking at the judgment result of the harvested sweet potatoes HSP displayed on the display 130.

[0071] The workers sort the harvested sweet potato HSP according to whether it is in a sellable shape or not, and for the harvested sweet potato HSP that is not in a sellable shape, they sort the harvested sweet potato HSP by its shape. For the harvested sweet potato HSP that is in a sellable shape, they sort it into those with and without blemishes, and for the ones with blemishes, they further sort it according to the type of blemish.

[0072] Next, the procedure up to collection of harvested sweet potato HSP will be described. Figure 8 is a diagram showing the process from harvesting to shipping harvested sweet potato HSP. After being harvested, the harvested sweet potato HSP is stored in a storehouse for a certain period of time (P0). The harvested sweet potato HSP taken out of the storehouse is sorted and shipped, for example, by the fifth day.

[0073] First, on the first day, the harvested sweet potato HSP is roughly sorted and adjusted (root cut) (P11). Next, the root-cut harvested sweet potato HSP is washed (P12), sterilized (P13), and then stored in a curing storehouse to begin curing (P14). The above processing is carried out on the first day. For example, high-temperature curing is used for curing.

[0074] Curing is carried out for several days. After curing, the harvested sweet potatoes HSP are sorted by condition using a sorting device 2 (P21). After that, the harvested sweet potatoes HSP are packaged by the sorted type (P22) and shipped by being loaded onto a truck (P23).

[0075] The state determination device 100 in the sorting device 2 determines the state of the harvested sweet potatoes HSP transported by the conveyor 50, and an operator sorts the harvested sweet potatoes HSP based on the determination result of the state determination device 100. Next, the process of the sorting device 2 will be described. Figure 9 is a flowchart showing an example of the process of the sorting device 2.

[0076] The sorting device 2 first transports the harvested sweet potatoes HSP loaded onto the conveyor 50 by an operator (step S201). While the conveyor 50 is transporting the harvested sweet potatoes HSP, the first camera 61, the second camera 62, and the third camera 63 each capture images of the harvested sweet potatoes HSP being transported (step S203). The first camera 61, the second camera 62, and the third camera 63 transmit the captured images to the state determination device 100.

[0077] Next, the state determining device 100 causes the acquiring unit 141 to extract sweet potato images from among the captured images transmitted by the first camera 61, the second camera 62, and the third camera 63 (step S205).

[0078] Next, the determination unit 142 specifies the three-dimensional shape of the harvested sweet potato HSP for each of the extracted sweet potato images based on the output data outputted by inputting the sweet potato image as input data to the shape diagnosis model 152 (step S207). As a result of determining the three-dimensional shape of the harvested sweet potato HSP, the determination unit 142 determines whether the three-dimensional shape of the harvested sweet potato HSP is a sellable shape (step S209).

[0079] If the determination unit 142 determines that the three-dimensional shape of the harvested sweet potato HSP is not a sellable shape, the three-dimensional shape of the harvested sweet potato HSP is a "rod shape," a "round shape," or an "abnormal shape," none of which are suitable for sale. In this case, the display control unit 143 causes the three-dimensional shape of the harvested sweet potato HSP determined by the determination unit 142 to be displayed on the display 130 together with the sweet potato image (step S211). The worker identifies the harvested sweet potato HSP to be sorted while looking at the sweet potato image displayed on the display 130, and sorts the harvested sweet potato HSP by shape according to the displayed content.

[0080] When it is determined that the three-dimensional shape of the harvested sweet potato HSP is not a sellable shape, the determination unit 142 determines whether or not the harvested sweet potato HSP has any scratches based on the output data outputted by inputting the sweet potato image as input data to the surface diagnosis model 151 (step S213). The determination unit 142 determines whether or not the surface of the harvested sweet potato HSP has any scratches based on the specific result (step S215).

[0081] When the determination unit 142 determines that there is no damage on the surface of the harvested sweet potato HSP, the display control unit 143 displays on the display 130, together with the sweet potato image, that the harvested sweet potato HSP to be sorted determined by the determination unit 142 is a sellable product (step S217). On the other hand, when it is determined that there is damage on the surface of the harvested sweet potato HSP, the determination unit 142 identifies the symptoms of the damage (step S219). After that, the display control unit 143 displays on the display 130 the symptoms of the damage determined by the determination unit 142 together with the sweet potato image (step S221). The worker distinguishes the harvested sweet potato HSP to be sorted while looking at the sweet potato image displayed on the display 130, and sorts the harvested sweet potato HSP by shape according to the displayed contents. In this way, the process shown in FIG. 9 is completed.

[0082] In the state determination model generating device 1 of the embodiment, a surface diagnosis model 151 and a shape diagnosis model 152 are generated as state determination models based on generated images obtained by constructing a two-dimensional image of a sweet potato in three dimensions and rendering the three-dimensional shape data. Since the generated images for generating the state determination model are obtained by rendering the three-dimensional shape data, a large amount of generated images can be generated depending on the rendering mode. Since the state determination model is generated using the generated images as training data, a large amount of training data can be easily generated. Therefore, according to the state determination model generating device 1 of the embodiment, a farm crop state determination model capable of performing accurate sorting can be generated. In addition, since the state determination model generated by the sorting device 2 (state determination device 100) of the embodiment and the state determination model generated by the state determination model generating device 1 of the embodiment is used to perform state determination of the sweet potato, the sweet potato can be sorted with high accuracy.

[0083] In the above embodiment, the surface diagnostic model 151 determines whether or not there is a scratch on the surface of the sweet potato SP, but future spoilage of the sweet potato SP may be predicted based on the scratch on the surface, particularly the symptom of a "bruise." The prediction of spoilage of the sweet potato SP will be described below.

[0084] When a "bruise" is found on a sweet potato SP, the part where the "bruise" has occurred may gradually become rotten. Even if the sweet potato SP is fine at the time of shipment, for example, in a situation where the transportation time is long, for example, when it is exported to a foreign country, the spoilage may progress over a long period of time, and the quality as a product may decrease. Therefore, for example, in the condition determination model generation device 1, machine learning may be performed using training data in which the input data is an image of a sweet potato with a "bruise" and the output data is the presence or absence of spoilage that makes the product worthless, and a spoilage diagnosis model may be generated as a condition determination model. In this case, the condition determination device 100 may predict spoilage of the sweet potato SP using the spoilage diagnosis model.

[0085] When predicting spoilage of sweet potatoes SP using the spoilage diagnostic model, judgment may be performed in combination with the shape diagnostic model 152 and the surface diagnostic model 151. In this case, for example, sweet potato images obtained by images captured by the first camera 61, the second camera 62, and the third camera 63 in the sorting device 2 may be divided and used for either judgment using the shape diagnostic model 152 and the surface diagnostic model 151 or prediction using the spoilage diagnostic model.

[0086] For example, the first camera 61 captures an image of the sweet potato SP from a viewpoint from above the conveyor 50, and the second camera 62 and the third camera 63 capture images of the sweet potato SP from a viewpoint oblique to the conveyor 50. In this case, the image captured by the first camera 61 may be used for judgment using the shape diagnostic model 152 and the surface diagnostic model 151, and the images captured by the second camera 62 and the third camera 63 may be used for prediction of spoilage of the sweet potato SP using the spoilage diagnostic model. In this way, spoilage of the sweet potato SP can also be predicted.

[0087] In the above embodiment, the sorting device 2 classifies the harvested sweet potatoes HSP based on the scratches on the surface of the harvested sweet potatoes HSP, but instead of or in addition to the scratches on the surface, the harvested sweet potatoes HSP may be classified according to the type of symptoms that have occurred on the harvested sweet potatoes HSP. In this case, the condition determination model generating device 1 may generate a condition determination model with the output data being the type of symptoms that have occurred on the sweet potatoes SP. Symptoms that have occurred on the sweet potatoes SP include pest damage, such as "nematode damage," "chafer damage," and "wireworm damage." Also, instead of or in addition to predicting spoilage, the feasibility of export may be determined.

[0088] In addition, the condition determination model may be generated according to, for example, shipping standards set by various associations. For example, when the shipping standards include items such as sweet potato size, quality classification, and adjustment, a trained model generated by machine learning using training data using correct answer data of output data for each item may be used as the condition determination model.

[0089] The state determination model may be used in a manner other than the manner in which the conveyor 50 and the cameras 61 to 63 are provided as in the sorting device 2. For example, the state determination model may be installed as an application program in a smartphone carried by an operator, and sweet potatoes photographed by the smartphone may be sorted. Alternatively, a control device in which the state determination model is installed may be incorporated in camera-equipped goggles worn by an operator, and sweet potatoes photographed by the camera of the camera-equipped goggles may be sorted. [Explanation of symbols]

[0090] 1...state determination model generating device, 2...sorting device, 10...imaging device, 11...rotating device, 12...mounting table, 13...camera, 14...lighting device, 20...control device, 21...input section, 22...memory section, 23...control section, 24...drive control section, 25...model generating section, 31...first model generating information, 32...second model generating information, 50...conveyor, 61...first camera, 62...second camera, 63...third camera, 70...PoE hub, 100...state determination device, 110...communication interface, 120...input interface, 130...display, 140...processing circuit, 141...acquisition section, 142...determination section, 143...display control section, 150...memory section, 151...surface diagnosis model, 152...shape diagnosis model, HSP...harvested sweet potato, SP...sweet potato.

Claims

1. The computer A step of generating three-dimensional shape data by three-dimensionally constructing a known crop image obtained by imaging a known crop whose state is known; generating a plurality of two-dimensional images of known agricultural crops based on the three-dimensional shape data; and performing machine learning using learning data in which the known two-dimensional image of the crop is input data and the state of the known crop is output data, thereby generating a crop state determination model. A method for generating a crop condition determination model.

2. The known crop image includes images of the known crop captured from a plurality of different directions. A method for generating a crop condition determination model according to claim 1 .

3. The three-dimensional shape data is generated by three-dimensionally constructing the known agricultural crop image converted into a point cloud. A method for generating a crop condition determination model according to claim 1 .

4. The known agricultural crops include sweet potato; A method for generating a crop condition determination model according to claim 1 .

5. The state includes at least one of a shape state related to the shape of the known agricultural product, a damage degree related state related to a determination of damage to the known agricultural product, or a shipping related state related to whether or not the known agricultural product can be shipped as a commodity. The method for generating a crop condition determination model according to claim 4.

6. The injury-related condition includes at least one of "groove", "hole", "lenticel", "break", "peeling", "gouging", "yarapin", "skin vein" and "bruise" occurring in the known agricultural crop; A method for generating a crop condition determination model according to claim 5.

7. An acquisition unit that acquires a target crop image, which is an image of a target crop that is a subject of a state judgment; and a determination unit that determines a state of the target crop based on output data obtained by inputting the acquired target crop image into a crop state determination model generated by the crop state determination model generation method according to any one of claims 1 to 6. Crop condition determination device.

8. Further, an imaging device for capturing the target crop image is provided. The imaging device transmits the captured target crop image to the acquisition unit. The crop condition determining device according to claim 7.

9. Further comprising a conveyor for transporting the target crop; The imaging device images the target crop being transported by the conveyor. The crop condition determining device according to claim 8.

10. The computer Acquire a target crop image, which is an image of a target crop for which a state is to be determined; The acquired target crop image is inputted into a crop state determination model generated by the method for generating a crop state determination model according to any one of claims 1 to 6, and the state of the target crop is determined based on output data obtained. Method for determining crop condition.

Images