Grade determination system, holding box, grade determination method and grade determination program

Abstract

To provide a grade determination system, s method, s program, and a holding box which can accurately determine grade of grape even not by a skilled person.SOLUTION: A grade determination system is provided in which a holding box for hanging and holding grape bunch as grade determination object, a display device and a server are connected to each other via communication channels. A control part of the server comprises: a picked-up image acquisition part for acquiring a picked-up image obtained by imaging grape bunch; a weight information acquisition part for acquiring weight information of the grape bunch; a feature information acquisition part acquiring feature information indicating grain size of grape, density aggregation of grains in bunch, bunch color, based on picked-up image; a grade information acquisition part for, when information indicating size of grains included in the grape bunch, density aggregation of the grains in the bunch, and colour and weight of the bunch, inputting the acquired feature information and weight information into a first learning model trained to output grade information indicating grade of the bunch to acquire grade information outputted from the first learning model; and a grade information output part for outputting the acquired grade information.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a grading system, a holding box, a grading method, and a grading program. [Background technology]

[0002] In recent years, consumer demand for grapes such as Shine Muscat has been increasing. Such grapes have a high unit price and are a highly profitable crop for farmers, so grading them before shipping is a very important process in order to maintain the quality and brand of the grapes.

[0003] As a technology related to grading, a technology has been proposed in which light is irradiated onto agricultural products (e.g., strawberries, cherry tomatoes, cherries) from a light-projecting means, the transmitted light is detected by a light-receiving means, and the absorbance of the agricultural products is measured and analyzed to determine the internal quality (grade) of the agricultural products, such as their sugar content (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-325281 Summary of the Invention [Problem to be solved by the invention]

[0005] Grading grapes collected at sorting facilities (collection points) after harvest is left to a small number of experienced agricultural workers, but labor shortages due to factors such as aging are becoming a major problem. Specifically, at the site of grape grading, which requires skilled labor, many of the experienced workers are reaching retirement age due to the declining birthrate and aging population, while the number of young workers newly taking on agricultural work is decreasing. Furthermore, since training skilled workers requires many years of experience and is difficult to do in a short period of time, passing on the skilled work that has been carried out by experienced workers is a major issue. The skilled skills possessed by skilled workers are called tacit knowledge. Tacit knowledge includes many skills that cannot be expressed in words, such as intuition and knacks, and it is extremely difficult to translate it into logic and control machines.

[0006] The main object of the present invention is to provide a grading system, a holding box, a grading method, and a grading program that enable even non-experts to accurately determine the grade of grapes. [Means for solving the problem]

[0007] According to an exemplary embodiment of the present invention, there is provided a grading system having the following configuration. [1] A grading system for grading a grape bunch, comprising an image acquisition unit, a weight information acquisition unit, a characteristic information acquisition unit, a grade information acquisition unit, and a grade information output unit, wherein the image acquisition unit acquires an image of the grape bunch having a plurality of grapes, the weight information acquisition unit acquires weight information indicating a detection result of the weight of the grape bunch, and the characteristic information acquisition unit determines the size of the grapes, the density of the grapes in the grape bunch, and the weight of the grapes based on the acquired image. A grading system that acquires feature information indicating the color of a grape bunch, and the grade information acquisition unit inputs the acquired feature information and weight information into a first learning model that has been trained to output grade information indicating the grade of the grape bunch when information indicating the size of the grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, to acquire grade information output from the first learning model, and the grade information output unit outputs the acquired grade information.

[0008] Various embodiments of the present invention will be described below as examples, and the embodiments shown below can be combined with each other. [2] A grading system as described in [1], further comprising an image generation unit that generates, based on the acquired captured image, a grape bunch image showing the grape bunch included in the captured image and a grape berry image showing the plurality of grapes included in the captured image, and the characteristic information acquisition unit acquires, based on the generated grape bunch image and grape berry image, information showing the size of the grapes, the density of the grapes in the grape bunch, and the color of the grape bunch as the characteristic information. [3] A grading system as described in [2], wherein the characteristic information acquisition unit calculates the area of ​​the grape region as the size of the grape based on the generated grape image. [4] A grading system according to [2] or [3], wherein the characteristic information acquisition unit calculates the density set by subtracting a second area representing the areas of the grapes from a first area representing the area of ​​the grapes based on the generated grape bunch image and grape berry image, and dividing the result by the second area. [5] A grading system according to any one of [2] to [4], wherein the feature information acquisition unit inputs the generated grape bunch image into a second learning model that has been trained to output color information indicating the color of the grape bunch when an image showing a grape bunch is input, and acquires the color information output from the second learning model. [6] A grading system according to any one of [1] to [5], further comprising a holding box, an imaging unit, a weight detection unit, and a display device, wherein the holding box is configured to be able to hold the bunches of grapes in a suspended state by a holding mechanism, the holding box having a bottom, a ceiling, and pillars connecting a plurality of corners of the bottom to the ceiling, the imaging unit is attached to the pillars and generates the captured image by imaging the bunches of grapes held by the holding mechanism, the weight detection unit is integrated with the holding mechanism and attached to the ceiling, and outputs the weight information in response to detecting the weight of the bunches of grapes held by the holding mechanism, and the display device displays the grade information output by the grade information output unit. [7] [6] A grading system as described in [6], wherein the bottom, ceiling, and column of the retaining box are constructed in a skeletal manner using multiple frames. [8] A holding box comprising an imaging unit, a weight detection unit, and a display device, wherein the holding box has a bottom, a ceiling, and pillars connecting multiple corners of the bottom to the ceiling, which are constructed in a skeletal form using multiple frames, and is configured to be able to hold grape bunches in a suspended state using a holding mechanism; the imaging unit is attached to the pillars and generates an image by capturing an image of the grape bunches held by the holding mechanism; the weight detection unit is attached to the ceiling and integrated with the holding mechanism, and outputs weight information in response to detecting the weight of the grape bunches held by the holding mechanism; and the display device is capable of displaying information indicating the grade of the grape bunches. [9] [8] The holding box further includes a control unit capable of calculating information representing the grade, wherein the control unit acquires, based on the captured image, feature information indicating the size of the grapes, the density of the grapes in the grape bunch, and the color of the grape bunch, inputs the acquired feature information and weight information into a first learning model that is trained to output grade information indicating the grade of the grape bunch when information indicating the size of the grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, to acquire grade information output from the first learning model, and outputs the acquired grade information to the display device.

[10] A grade estimation method for determining the grade of a grape bunch, comprising: an image acquisition step, a weight information acquisition step, a characteristic information acquisition step, a grade information acquisition step, and a grade information output step, wherein the image acquisition step acquires an image of a grape bunch having a plurality of grapes; the weight information acquisition step acquires weight information indicating a detection result of the weight of the grape bunch; and the characteristic information acquisition step acquires, based on the acquired image, the size of the grapes, the density and aggregation of the grapes in the grape bunch, and acquiring characteristic information indicating the color of the grape bunch, and in the grade information acquisition step, inputting the acquired characteristic information and weight information into a first learning model that has been trained to output grade information indicating the grade of the grape bunch when information indicating the size of the grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, thereby acquiring grade information output from the first learning model, and in the grade information output step, outputting the acquired grade information.

[11] A grading program that causes a processor to execute a process of acquiring, based on an image of a grape bunch having a plurality of grapes, feature information indicating the size of the grapes, the density of the grapes in the grape bunch, and the color of the grape bunch, inputting the feature information and weight information indicating the detection result of the weight of the grape bunch into a first learning model that has been trained to output grade information indicating the grade of the grape bunch when information indicating the size of the grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, acquiring grade information output from the first learning model, and outputting the acquired grade information. [Effects of the Invention]

[0009] According to the present invention, even an unskilled person can accurately determine the grade of grapes. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing the configuration of a grade determination system according to an embodiment of the present invention; [Figure 2] 2A and 2B are diagrams illustrating the configuration of a holding box and a display device according to the present embodiment. [Figure 3] FIG. 2 is a block diagram showing the hardware configuration of the display device according to the present embodiment. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of a display device according to the present embodiment. [Figure 5] FIG. 2 is a block diagram showing the hardware configuration of a server according to the present embodiment. [Figure 6] FIG. 2 is a block diagram illustrating an example of a functional configuration of a server according to the present embodiment. [Figure 7] 10 is a flowchart illustrating an example of processing performed by the display device according to the present embodiment. [Figure 8] 10 is a flowchart illustrating an example of processing by a server according to the present embodiment. [Figure 9] FIG. 10 is a block diagram showing a modified example of the functional configuration of the display device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described with reference to the accompanying drawings, in which the same reference numerals denote the same or similar components.

[0012] (Configuration of Grade Judgment System 1) FIG. 1 shows the configuration of a grading system 1 according to this embodiment. The grading system 1 shown in FIG. 1 provides a user (e.g., a farmer) with a grading service that grades grapes collected at a sorting facility (collection site) after harvest. As shown in FIG. 1, the grading system 1 includes a holding box 2, a display device 3, and a server 5. The display device 3 and the server 5 are communicatively connected to each other via a communication line 4 that may be the Internet, an intranet, a wireless local area network (LAN), mobile communications, or an appropriate combination of these communication means. A weight sensor 25 (functioning as a weight detection unit) and a camera 26 (functioning as an imaging unit) attached to the holding box 2 are communicatively connected to the display device 3 via a sensor cable (not shown). To facilitate user transport, the holding box 2 may be foldable on a pillar 22 by employing a common mechanism, and the display device 3, weight sensor 25, and camera 26 may also be detachable from the pillar 22.

[0013] FIG. 2 is a diagram illustrating the configuration of the holding box 2 and the display device 3 according to this embodiment. As shown in FIG. 2, the holding box 2 includes a bottom 21, a ceiling 23, and pillars 22A, 22B, 22C, and 22D connecting multiple corners (four corners in the illustrated example) of the bottom 21 to the ceiling 23. When there is no need to distinguish between the pillars 22A, 22B, 22C, and 22D, the pillars 22A, 22B, 22C, and 22D are each referred to as a pillar 22. In this embodiment, the holding box 2 includes the bottom 21, the ceiling 23, and the pillars 22 configured in a hexahedral skeleton shape using multiple frames. That is, the holding box 2 has no side surfaces on the front, rear, left, or right sides. This allows a user to easily reach into the holding box 2 between the pillars 22A, 22B, 22C, and 22D.

[0014] The holding box 2 is configured to hold a branch 31 of the grape bunch 30 (having multiple grapes) to be graded, with a holding mechanism 24 (e.g., a gripper) suspending the grape bunch 30. The holding mechanism 24 has a known mechanism that opens and closes using the weight of the grape bunch 30, opening when a user lifts the grape bunch 30 upward and closing when the user releases the grape bunch 30. An example of such a known mechanism is the mechanism disclosed in the paper "Development of a Simple Photography Booth for Shine Muscat Grapes to Create Training Data for Deep Learning" by Akihiro Usui (Yamanashi University), Hiromitsu Nishizaki (Yamanashi University), Mao Xiaoyang (Yamanashi University), and Koji Makino (Yamanashi University) for the 22nd System Integration Conference (SI2021).

[0015] The weight sensor 25 is integrated with the holding mechanism 24 and attached to the ceiling 23 of the holding box 2, and detects the weight of the grape bunches 30 held by the holding mechanism 24. In this embodiment, the weight sensor 25 is configured to automatically detect the weight of the grape bunches 30 simply by holding the grape bunches 30 in the holding mechanism 24. The weight sensor 25 then outputs weight information indicating the detected weight of the grape bunches 30 (detection result) to the display device 3.

[0016] Cameras 26A, 26B, 26C, and 26D are attached to posts 22A, 22B, 22C, and 22D, respectively, and generate captured images by simultaneously (almost simultaneously) capturing images of grape bunches 30 held by holding mechanism 24 from multiple (four in the illustrated example) directions. Cameras 26A, 26B, 26C, and 26D then output the generated captured images to display device 3. When it is not necessary to distinguish between cameras 26A, 26B, 26C, and 26D, cameras 26A, 26B, 26C, and 26D will each be referred to as camera 26.

[0017] The display device 3 is an information processing terminal such as a tablet terminal, and provides a rating determination service to a user. By operating the display device 3, the user can log in to the rating determination service and execute processing corresponding to the rating determination service. The display device 3 may be any terminal, such as a mobile phone, a personal computer (PC), a notebook PC, a personal digital assistant (PDA), or a home game console.

[0018] (Hardware configuration of display device 3) 3 is a block diagram showing the hardware configuration of the display device 3 in this embodiment. As shown in FIG. 3, the display device 3 includes a control unit 11, a storage unit 12, a communication unit 13, a display unit 14, a speaker 15, and an operation button 16.

[0019] The control unit 11 is, for example, a CPU (Central Processing Unit), a microprocessor, a DSP (Digital Signal Processor), or the like, and controls the overall operation of the display device 3.

[0020] A part of the storage unit 12 is configured with, for example, a RAM (Random Access Memory) or a DRAM (Dynamic Random Access Memory), and is used as a work area when the control unit 11 executes processes based on various programs.

[0021] Furthermore, a part of the storage unit 12 is, for example, a non-volatile memory such as a ROM (Read Only Memory) or an HDD (Hard Disk Drive), and stores various data and programs used in the processing of the control unit 11. The storage unit 12 can hold a database including one or more tables for recording various information, processing results, and the like.

[0022] The programs stored in the storage unit 12 include, for example, an OS (Operating System) for realizing the basic functions of the user terminal 20, drivers for controlling various hardware, programs for realizing various functions, and the like.

[0023] The communication unit 13 is, for example, a network interface controller (NIC), and has a function of connecting to the communication line 4. Instead of or together with the NIC, the communication unit 13 may have a function of connecting to a wireless local area network (LAN), a function of connecting to a wireless wide area network (WAN), a function of enabling short-range wireless communication such as Bluetooth (registered trademark), infrared communication, etc. The display device 3 is connected to a server 5, etc. via the communication line 4, and can transmit and receive various data to and from the server 5, etc.

[0024] The display unit 14 is a touch panel display or the like, and is capable of displaying images and the like and accepting operations by the user.

[0025] The speaker 15 outputs various sounds under the control of the control unit 11. The operation buttons 16 are provided on, for example, the side surface of the display device 3, and include a power button for starting or stopping the display device 3, a button for adjusting the volume of the sound output from the speaker 15, and the like.

[0026] The control unit 11, memory unit 12, communication unit 13, display unit 14, speaker 15, operation buttons 16, weight sensor 25, and camera 26 are electrically connected to one another via a system bus 17. Therefore, the control unit 11 can access the memory unit 12, display images on the display unit 14, grasp the operation status of the touch panel display (display unit 14) and operation buttons 16 by the user, output sound from the speaker 15, control imaging of the camera 26, and access various communication networks, the server 5, etc. via the communication unit 13.

[0027] Fig. 4 is a block diagram showing an example of the functional configuration of the control unit 11 provided in the display device 3 in this embodiment. As shown in Fig. 4, the control unit 11 has, as its functional configuration, a captured image acquisition unit 11a, a weight information acquisition unit 11b, a transmission control unit 11c, a reception control unit 11d, and a display control unit 11e. Note that the control unit 11 of the display device 3 generally has various functions in addition to those described above, but here we will only explain the functions that are characteristic of the grading service provided by the grading system 1 in this embodiment, and will not illustrate or explain other known functions, etc.

[0028] The captured image acquisition unit 11a acquires image information of a captured image generated and output by the camera 26 (hereinafter also referred to as captured image information).

[0029] The weight information acquiring unit 11b acquires weight information detected by the weight sensor 25.

[0030] The transmission control unit 11c controls the communication unit 13 to transmit to the server 5 the captured image information acquired by the captured image acquisition unit 11a and the weight information acquired by the weight information acquisition unit 11b.

[0031] The reception control unit 11d controls the communication unit 13 so as to receive the rating information transmitted from the server 5.

[0032] The display control unit 11e controls the display unit 14 to display the rating information received by the reception control unit 11d.

[0033] (Server 5 hardware configuration) 5 is a block diagram showing the hardware configuration of the server 5 in this embodiment. As shown in FIG. 5, the server 5 is configured to include a control unit 41, a storage unit 42, a communication unit 43, an operation input unit 44, and a display unit 45.

[0034] The control unit 41 is, for example, a CPU (Central Processing Unit), a microprocessor, a DSP (Digital Signal Processor), or the like, and controls the overall operation of the server 5. The control unit 41 functions as a processor.

[0035] A part of the storage unit 42 is configured, for example, with RAM (Random Access Memory) or DRAM (Dynamic Random Access Memory), and is used as a work area when the control unit 41 executes processes based on various programs.

[0036] Furthermore, a part of the storage unit 42 is, for example, a non-volatile memory such as a ROM (Read Only Memory) or an HDD (Hard Disk Drive), and stores various data and programs used in the processing of the control unit 41. The storage unit 42 can hold a database including one or more tables for recording various information, processing results, and the like.

[0037] The programs stored in the memory unit 42 include, for example, an OS (Operating System) for realizing the basic functions of the server 5, drivers for controlling various hardware, programs for realizing various functions, etc., and include a program that functions as a grade determination program.

[0038] The communication unit 43 is, for example, a network interface controller (NIC), and has a function of connecting to the communication line 4. Instead of or together with the NIC, the communication unit 43 may have a function of connecting to a wireless local area network (LAN), a function of connecting to a wireless wide area network (WAN), a function of enabling short-range wireless communication such as Bluetooth (registered trademark), infrared communication, etc. The server 5 is connected to the display device 3 etc. via the communication line 4, and can transmit and receive various data to and from the display device 3 etc.

[0039] The operation input unit 44 is composed of a keyboard, a mouse, etc., and accepts input of various operations by a user (e.g., a grade determination service provider) who uses the server 5. The display unit 45 is, for example, a liquid crystal display device, etc., and displays various images.

[0040] The control unit 41, the storage unit 42, the communication unit 43, the operation input unit 44, and the display unit 45 are electrically connected to one another via a system bus 46. Therefore, the control unit 41 can access the storage unit 42, display images on the display unit 45, grasp the operation state of the operation input unit 44 by the user, and access various communication networks and the display device 3 via the communication unit 43.

[0041] Fig. 6 is a block diagram showing an example of the functional configuration of the control unit 41 provided in the server 5 in this embodiment. As shown in Fig. 6, the control unit 41 has, as its functional configuration, a reception control unit 41a, an image generation unit 41b, a characteristic information acquisition unit 41c, a grade information acquisition unit 41d, and a transmission control unit 41e. Note that the control unit 41 of the server 5 generally has various functions in addition to those described above, but here we will only explain the functions that are characteristic of the grade determination service provided by the grade determination system 1 in this embodiment, and will not illustrate or explain other known functions, etc.

[0042] The reception control unit 41 a controls the communication unit 43 to receive the captured image information and weight information transmitted from the display device 3 .

[0043] Based on the captured image information received by the reception control unit 41a, the image generation unit 41b generates a grape bunch image showing a grape bunch 30 included in the captured image represented by the captured image information and a grape berry image showing multiple grapes in the grape bunch 30 included in the captured image. In this embodiment, the image generation unit 41b performs instance segmentation processing on the captured image (processing to identify object regions and perform region division and object type recognition for each individual object) to remove background images other than the grape bunch 30 in the captured image, thereby generating a grape bunch image and a grape berry image. The instance segmentation processing is performed, for example, by inputting the captured image received by the reception control unit 41a into a segmentation model (DNN: Deep Neural Network) that has been trained to generate a grape bunch image showing the grape bunch included in the image and a grape berry image showing multiple grapes in the grape bunch included in the image when an image showing a grape bunch is input, and acquiring the grape bunch image and grape berry image output (generated) from the segmentation model.

[0044] Based on the captured image received by the reception control unit 41a, the feature information acquisition unit 41c acquires feature information (feature vector) indicating the size of the grapes, the density of the grapes in the grape bunch 30 (representing the degree of density of the individual grapes in one grape bunch 30), and the color of the grape bunch 30. In this embodiment, the feature information acquisition unit 41c acquires, as feature information, information indicating the size (average) of the grapes, the density of the grapes in the grape bunch 30, and the color of the grape bunch 30 based on the grape bunch image and grape image generated by the image generation unit 41b.

[0045] Based on the grape image generated by the image generating unit 41b, the characteristic information acquiring unit 41c calculates the average area (unit: pixels) of the regions of each grape as the (average) size of the grape.

[0046] Based on the grape bunch image and grape image generated by the image generation unit 41b, the characteristic information acquisition unit 41c calculates the density set of grapes in the grape bunch 30 by subtracting a second area (unit: pixel) representing the area of ​​the plurality of grapes from a first area (unit: pixel) representing the area of ​​the grape bunch 30 to obtain an area (blank area in the grape bunch 30) and dividing the area by the second area (area of ​​the plurality of grapes). Note that the smaller the blank area in the grape bunch 30, the more grapes are packed into the grape bunch 30, and the higher the grade of the grape bunch 30.

[0047] The feature information acquisition unit 41c inputs the grape bunch image generated by the image generation unit 41b into a second learning model that has been trained to output color information indicating the color of the grape bunch when an image showing a grape bunch is input, and acquires color information output from the second learning model. The second learning model is, for example, a color estimation model (DNN) stored in the storage unit 42 and trained using images of grape bunches labeled with colors by grape producers as training data. In this embodiment, the color of the grape bunch 30 is represented by a number, for example, 1 to 5. As described above, in this embodiment, the size of the grapes in the grape bunch 30, the density of the grapes in the grape bunch 30, and the color of the grape bunch 30, which are parameters necessary for grading, can be acquired without depending on the lighting environment of the site (e.g., a sorting facility).

[0048] The grade information acquisition unit 41d inputs the characteristic information acquired by the characteristic information acquisition unit 41c and the weight information received by the reception control unit 41a into a first learning model that has been trained to output grade information indicating the grade of the grape bunch (e.g., any of exceptional (special selection), excellent, good, good, or no mark) when information indicating the size of the grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, and acquires grade information (e.g., information indicating the grade of grape bunch 30 as "excellent") output from the first learning model. The first learning model is, for example, a grade estimation model stored in the memory unit 42 and trained using standard grading results for grapes as training data.

[0049] In this embodiment, since each piece of information indicated in the feature information and weight information has a different range, the grade information acquisition unit 41d performs a normalization process as preprocessing to convert each piece of information indicated in the feature information and weight information into a unified range before inputting the feature information and weight information into the first learning model.

[0050] The transmission control unit 41e controls the communication unit 43 so as to transmit the grade information acquired by the grade information acquisition unit 41d to the display device 3. The transmission control unit 41e functions as a grade information output unit.

[0051] (Processing performed by the display device 3) FIG. 7 is a flowchart showing an example of processing performed by the display device 3 when providing a rating determination service in this embodiment.

[0052] First, the captured image acquisition unit 11a acquires a captured image generated and output by the camera 26 (step S100).

[0053] Next, the weight information acquiring unit 11b acquires the weight information detected and output by the weight sensor 25 (step S110).

[0054] Next, the transmission control unit 11c controls the communication unit 13 to transmit the captured image (captured image information) acquired by the captured image acquisition unit 11a and the weight information acquired by the weight information acquisition unit 11b to the server 5 (step S120).

[0055] Next, the reception control unit 11d determines whether or not it has received the rating information transmitted from the server 5 (step S130). If the determination result shows that it has not received the rating information transmitted from the server 5 (NO in step S130), the process returns to before step S130.

[0056] On the other hand, if the rating information transmitted from the server 5 is received (YES in step S130), the display control unit 11e controls the display unit 14 to display the rating information received by the reception control unit 11d (step S140). When the process of step S140 is completed, the display device 3 ends the process shown in FIG.

[0057] (Processing performed by Server 5) 8 is a flowchart showing an example of processing (corresponding to the "class determination method" of the present invention) performed by the server 5 when providing a class determination service in this embodiment. The processing of step S200 in Fig. 8 starts when the transmission control unit 11c transmits to the server 5 the captured image (captured image information) acquired by the captured image acquisition unit 11a and the weight information acquired by the weight information acquisition unit 11b in the processing of step S120 shown in Fig. 7.

[0058] First, the reception control unit 41a controls the communication unit 43 to receive the captured image and weight information transmitted from the display device 3 (step S200).

[0059] Next, based on the captured image received by the reception control unit 41a, the image generation unit 41b generates a grape bunch image showing the grape bunch 30 included in the captured image and a grape berry image showing multiple grapes in the grape bunch 30 included in the captured image (step S210).

[0060] Next, the characteristic information acquisition unit 41c calculates the area of ​​the grape region as the size of the grape based on the grape image generated by the image generation unit 41b (step S220).

[0061] Next, based on the grape bunch image and grape berry image generated by the image generation unit 41b, the feature information acquisition unit 41c calculates the density set of grape berries in the grape bunch 30 by dividing the area (blank area within the grape bunch 30) obtained by subtracting the second area representing the area of ​​the plurality of grape berries from the first area representing the area of ​​the grape bunch 30 by the second area (area of ​​the plurality of grape berries) (step S230).

[0062] Next, the feature information acquisition unit 41c inputs the grape bunch image generated by the image generation unit 41b to the second learning model and acquires the color information output from the second learning model (step S240).

[0063] Next, the grade information acquisition unit 41d inputs the characteristic information acquired by the characteristic information acquisition unit 41c and the weight information received by the receiving control unit 41a into the first learning model and acquires the grade information output from the first learning model (step S250).

[0064] Finally, the transmission control unit 41e controls the communication unit 43 to transmit the grade information acquired by the grade information acquisition unit 41d to the display device 3 (step S260). When the process of step S260 is completed, the server 5 ends the process shown in FIG.

[0065] (Effects of this embodiment) As described in detail above, in this embodiment, the grading system 1 includes a captured image acquisition unit 11a (display device 3), a weight information acquisition unit 11b (display device 3), a characteristic information acquisition unit 41c (server 5), a grade information acquisition unit 41d (server 5), and a grade information output unit (a transmission control unit 41e of the server 5). The captured image acquisition unit 11a acquires a captured image of a grape bunch 30 having a plurality of grapes. The weight information acquisition unit 11b acquires weight information indicating the detection result of the weight of the grape bunch 30. The characteristic information acquisition unit 41c acquires characteristic information indicating the size of the grapes, the density of the grapes in the grape bunch 30, and the color of the grape bunch 30 based on the acquired captured image. The grade information acquisition unit 41d inputs the acquired feature information and weight information into a first learning model that has been trained to output grade information indicating the grade of a grape bunch when information indicating the size of grapes in the grape bunch 30, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, and acquires the grade information output from the first learning model. The grade information output unit outputs the acquired grade information.

[0066] According to this embodiment configured as described above, even an unexpert can accurately determine the grade of grapes by inputting feature information acquired based on an image of the grape bunch 30 and weight information indicating the measurement results of the weight of the grape bunch 30 and checking the grade information output from the first learning model.

[0067] Furthermore, according to this embodiment, the holding mechanism 24 that opens and closes by the weight of the grape bunches 30 allows the grape bunches 30 to be attached and detached safely and easily. Furthermore, the weight sensor 25 is integrated with the holding mechanism 24 and attached to the ceiling 23 of the holding box 2, allowing for simultaneous (or substantially simultaneous) detection of the weight of the grape bunches 30 and photographing of the grape bunches 30 from four directions, thereby enabling rapid grading of the grape bunches 30. Furthermore, the bottom 21, ceiling 23, and columns 22 of the holding box 2 are configured using multiple frames to form a hexahedral skeleton, i.e., the box has no sides on the front, rear, left, or right sides. Therefore, the grape bunches 30 can be attached and detached from any direction, including the front, rear, left, or right, and can be easily bagged for shipping with the bunches 30 still attached, significantly reducing the grading work time.

[0068] In the above embodiment, an example has been described in which a captured image is generated by simultaneously capturing images of grape bunches 30 held by holding mechanism 24 from multiple (four in the illustrated example) directions, but the present invention, which has been described using the above embodiment as an example, is not limited to this. For example, a captured image may be generated by using only one camera 26, rotating holding mechanism 24, and capturing images of grape bunches 30 held by holding mechanism 24 sequentially from multiple (four in the illustrated example) directions.

[0069] Furthermore, in the above embodiment, the grade information acquisition unit 41d may adjust the feature information and weight information (e.g., add or subtract) via a user's operation on the operation button 16 of the display device 3 before inputting the feature information and weight information to the first learning model. This allows the grade information acquisition unit 41d to appropriately adjust the parameter standards required for grading the grape cluster 30, such as the size of the grapes in the grape cluster 30, the density of the grapes in the grape cluster 30, the color of the grape cluster 30, and the weight of the grape cluster 30, to adapt to the grape shipping season and market conditions. This adaptability is important for responding to differences in grape characteristics and market needs, for example, between different regions, seasons, and environmental conditions. Furthermore, even if the grape harvest throughout the entire grape-producing region changes due to bad weather, disasters, etc., the grade can be determined based on the actual market environment.

[0070] Furthermore, in the above embodiment, the grading system 1 may include the holding box 2 and the display device 3, but not the server 5. In this case, the holding box 2 may include a control unit that includes the image generation unit 41b, characteristic information acquisition unit 41c, and grade information acquisition unit 41d that were included in the control unit 41 of the server 5, either integrally with the display device 3 or separately. With this configuration, the grading of grapes can be performed by the holding box 2 alone at the sorting site (collection site) without the intervention of the server 5.

[0071] 9 is a block diagram showing a modified functional configuration of the display device 3 when the holding box 2 is provided with a control unit integrally with the display device 3, the control unit including the image generation unit 41b, the characteristic information acquisition unit 41c, and the grade information acquisition unit 41d that were provided in the control unit 41 of the server 5. As shown in FIG. 9, the control unit 11 of the display device 3 is provided with, as functional components, a captured image acquisition unit 11a, a weight information acquisition unit 11b, a display control unit 11e, an image generation unit 11f, a characteristic information acquisition unit 11g, and a grade information acquisition unit 11h.

[0072] The captured image acquiring unit 11a acquires captured image information generated and output by the camera 26. The weight information acquiring unit 11b acquires weight information detected by the weight sensor 25 and output.

[0073] The image generation unit 11f generates, based on the captured image information acquired by the captured image acquisition unit 11a, a grape bunch image showing the grape bunch 30 included in the captured image and a grape berry image showing multiple grape berries in the grape bunch 30 included in the captured image.

[0074] The feature information acquisition unit 11g calculates the area of ​​the grape region as the size of the grape based on the grape image generated by the image generation unit 41b. The feature information acquisition unit 41c calculates the density of grapes in the grape cluster 30 based on the grape bunch image and grape image generated by the image generation unit 41b by dividing the area (blank area in the grape cluster 30) obtained by subtracting a second area representing the region of multiple grapes from a first area representing the region of the grape cluster 30 by the second area (the region of multiple grapes). The feature information acquisition unit 41c inputs the grape cluster image generated by the image generation unit 41b to a second learning model and acquires color information output from the second learning model.

[0075] The grade information acquisition unit 11h inputs the feature information acquired by the feature information acquisition unit 11g and the weight information acquired by the weight information acquisition unit 11b into the first learning model and acquires the grade information output from the first learning model.

[0076] The display control unit 11e controls the display unit 14 so as to display a screen in a display mode (GUI: Graphical User Interface) according to the grade information acquired by the grade information acquisition unit 11h.

[0077] Although the above embodiment describes an example in which the grade information is displayed as an output mode of the grade information, the present invention is not limited to this. Examples of output modes of the grade information include audible output by playing an audio signal through the speaker 15 of the display device 3, visual output by displaying an image on the display unit 14 of the display device 3, visual output by generating light by controlling the light color, lighting / flashing pattern, light intensity, etc. on the display device 3, tactile output by generating vibrations using the vibration function of the display device 3 by controlling the vibration pattern, intensity, etc., and olfactory output by firing air with a predetermined odor, such as an irritating odor, into the user's nasal cavity using a swingable air cannon on the display device 3. Note that audio signals include human voice messages, buzzer sounds, chimes, and alarm sounds. The image displayed on the display unit 14 includes at least one element, such as a figure, computer graphics, a photograph, letters, numbers, and symbols, and may also include a combination of two or more elements. Furthermore, the image may be either monochrome or color, and may be either still or moving.

[0078] In the above embodiment, a mechanism for automatically attaching and detaching grape bunches 30 to and from holding box 2 may be added, thereby enabling grading system 1 to perform grading processing fully automatically.

[0079] <Modification> In the above-described embodiment, a system configuration has been described in which an operator determines the grade of grape bunches to be graded while replacing each bunch one by one using the holding box 2. In contrast, in the following modified example, a system configuration will be described in which the above-described holding box 2 is not used. Specifically, for example, a system according to one modification includes a hanging mechanism that is motor-driven and can change the orientation of the hanging grape bunches, and a light and a camera installed at a position around the hanging mechanism. The system controls the hanging mechanism to rotate the grape bunches to different orientations and take pictures of them, thereby obtaining multiple images to be used for grading. In this modification, the same procedure as in the above-described embodiment can be adopted for image processing related to class determination after the plurality of images are acquired. Furthermore, for example, a system according to another modified example includes a conveying line that can transport grape bunches in a suspended state and has multiple hanging mechanisms arranged along the length that can change the orientation of the grape bunches by motor drive, and multiple sets of lights and cameras at different positions along the conveying line. The system then transports the hanging grape bunches on a transport line, photographs the grape bunches at a first position, rotates the grape bunches by a predetermined angle, and photographs the grape bunches at a second position different from the first position. The system performs this operation sequentially for the number of images required to grade one grape bunch. In this case, the number of pairs of lights and cameras required may be, for example, the same as the number of images required to grade the grape bunches. The orientation of the lights and cameras arranged along the conveyor line may be on one side of the direction of travel of the grape bunches on the conveyor line, or on both sides. In this case, the angle of rotation of the grape bunches may be appropriately set according to the number and positions of the sets of lights and cameras arranged along the conveyor line. In this modified example, the same procedure as in the above-described embodiment can be adopted for image processing related to class determination after the plurality of images are acquired. With this system configuration, multiple images for use in grading can be acquired without stopping the transport of each hanging bunch of grapes. In any of the above-mentioned modified examples, the line operations such as hanging the grape bunches from the mechanism, bagging the grape bunches after grading, removing the bagged grape bunches from the mechanism, and, if necessary, packing the grape bunches into gift boxes for shipping prepared according to grade, may be automated by using automatic devices such as so-called material handling robots.

[0080] Furthermore, the above-described embodiments and modifications are merely examples of specific embodiments of the present invention, and the technical scope of the present invention should not be construed as being limited by these. In other words, the present invention can be embodied in various forms without departing from the gist or main features thereof. [Explanation of symbols]

[0081] 1: Classification system, 2: Holding box, 3: Display device, 4: Communication line, 5: Server, 11, 41: Control unit, 11a: Image acquisition unit, 11b: Weight information acquisition unit, 11c: Transmission control unit, 11d: Reception control unit, 11e: Display control unit, 11f, 41b: Image generation unit, 11g, 41c: Feature information acquisition unit, 11h, 41d: Class information acquisition unit, 12, 42: Memory unit, 13, 43: communication unit, 14, 45: display unit, 15: speaker, 16: operation button, 17, 46: system bus, 21: bottom, 22, 22A, 22B, 22C, 22D: pillars, 23: ceiling, 24: holding mechanism, 25: weight sensor, 26, 26A, 26B, 26C, 26D: camera, 30: grape bunch, 31: branch, 41a: reception control unit, 41e: transmission control unit, 44: operation input unit

Claims

1. A grading system for grading grape bunches, comprising: The system includes an image acquisition unit, a weight information acquisition unit, a feature information acquisition unit, a grade information acquisition unit, and a grade information output unit, the captured image acquisition unit acquires a captured image of the grape bunch having a plurality of grapes, the weight information acquisition unit acquires weight information indicating a detection result of the weight of the grape bunch; the characteristic information acquisition unit acquires characteristic information indicating a size of the grapes, a density of the grapes in the grape cluster, and a color of the grape cluster based on the acquired captured image; the grade information acquisition unit inputs the acquired feature information and weight information into a first learning model that has been trained to output grade information indicating the grade of a grape bunch when information indicating the size of grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, and acquires the grade information output from the first learning model; The grade information output unit outputs the acquired grade information. Grading system.

2. further comprising an image generating unit; the image generation unit generates, based on the acquired captured image, a grape bunch image showing the grape bunch included in the captured image and a grape berry image showing the plurality of grape berries included in the captured image; the characteristic information acquisition unit acquires, as the characteristic information, information indicating a size of the grapes, a density of the grapes in the grape bunch, and a color of the grape bunch based on the generated grape bunch image and grape individual image; The grading system of claim 1 .

3. The feature information acquisition unit calculates the area of ​​the grape region as the size of the grape based on the generated grape image. The grading system of claim 2 .

4. the feature information acquisition unit calculates the density set by subtracting a second area representing the regions of the plurality of grapes from a first area representing the region of the grape bunch, based on the generated grape bunch image and grape berry image, and dividing the resultant area by the second area; The grading system of claim 2 .

5. the feature information acquisition unit inputs the generated grape bunch image into a second learning model that has been trained to output color information indicating the color of the grape bunch when an image showing a grape bunch is input, and acquires the color information output from the second learning model; The grading system of claim 2 .

6. Further comprising a holding box, an imaging unit, a weight detection unit, and a display device, the holding box is configured to be able to hold the grape bunches in a suspended state by a holding mechanism, the holding box is configured to have a bottom, a ceiling, and pillars connecting a plurality of corners of the bottom to the ceiling, the imaging unit is attached to the column and generates the captured image by capturing an image of the grape bunch held by the holding mechanism; the weight detection unit is integrated with the holding mechanism and attached to the ceiling, and outputs the weight information in response to detecting a weight of the grape bunches held by the holding mechanism; The display device displays the grade information output by the grade information output unit. The grading system of claim 1 .

7. The holding box has a bottom, a ceiling, and a column that are configured in a skeletal form using a plurality of frames. The grading system of claim 6.

8. A holding box including an imaging unit, a weight detection unit, and a display device, The holding box is The bottom, the ceiling, and the columns connecting the corners of the bottom and the ceiling are configured in a skeletal form using a plurality of frames, The holding mechanism is configured to hold the grape bunches in a suspended state, the imaging unit is attached to the column and generates an image by imaging the grape bunch held by the holding mechanism; the weight detection unit is integrated with the holding mechanism and attached to the ceiling, and outputs weight information in response to detecting the weight of the grape bunches held by the holding mechanism; The display device is capable of displaying information representing the grade of the grape bunch. Holding box.

9. Further, a control unit capable of calculating information representing the grade is provided, The control unit acquiring, based on the captured image, characteristic information indicating a size of the grapes, a density of the grapes in the grape cluster, and a color of the grape cluster; a first learning model that is trained to output grade information indicating the grade of a grape bunch when information indicating the size of grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, and the acquired feature information and weight information are input to the first learning model to obtain grade information output from the first learning model; outputting the acquired grade information to the display device; The holding box of claim 8.

10. A method for estimating the grade of a grape bunch, comprising: The method includes an image acquisition step, a weight information acquisition step, a feature information acquisition step, a grade information acquisition step, and a grade information output step, In the image acquisition step, an image of a grape bunch having a plurality of grapes is acquired, In the weight information acquisition step, weight information indicating a detection result of the weight of the grape bunch is acquired, In the characteristic information acquisition step, characteristic information indicating a size of the grapes, a density of the grapes in the grape cluster, and a color of the grape cluster is acquired based on the acquired captured image; In the grade information acquisition step, the acquired feature information and weight information are input into a first learning model that has been trained to output grade information indicating the grade of a grape bunch when information indicating the size of grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, and the grade information output from the first learning model is acquired; In the grade information output step, the acquired grade information is output. Grading method.

11. The processor a captured image of a grape bunch having a plurality of grapes; and based on the captured image, characteristic information indicating the size of the grapes, the density of the grapes in the grape bunch, and the color of the grape bunch is acquired; a first learning model that has been trained to output grade information indicating the grade of a grape bunch when information indicating the size of grapes in the grape bunch, the density of the grapes in the grape bunch, the color of the grape bunch, and the weight of the grape bunch is input, and the feature information and weight information indicating the detection result of the weight of the grape bunch are input to the first learning model to obtain grade information output from the first learning model; outputting the acquired grade information; A grading program that executes the process.

Images