Melon grading system, melon grading method, program for melon grading system, and recording medium

Abstract

To provide a melon grade determination system capable of consistently performing melon grade determination with accuracy equivalent to that of a skilled producer.SOLUTION: A melon grade determination system includes an imaging unit and a server. The imaging unit includes: a photograph data acquisition part for taking a 360° photograph of the exterior of the melon; and a transmission part for transmitting the photograph data acquired by the photograph data acquisition part to the server. The server includes: a reception part for receiving the photograph data transmitted by the transmission part; an image conversion part for converting the photograph data received by the reception part into whole circumference image data of the melon; a determination part that determines the grade of the melon from the whole circumference image data converted by the image conversion part, on the basis of a deep learning model trained to transform melon contour feature vector to melon mesh feature vector using data annotated by a skilled melon producer; and a notification control part for performing control to issue a notification on the grade of the melon determined by the determination part.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This disclosure relates to a melon grading system, a melon grading method, a program for the melon grading system, and a recording medium. [Background technology]

[0002] A sorting device for Prince melons that determines the grade of Prince melons is known (see, for example, Patent Document 1). The Prince melon sorting device disclosed in Patent Document 1 includes a determination means that calculates the maximum cross-sectional area of ​​the Prince melon and the area of ​​the green streaky pattern present on its surface, and compares the calculation results with predetermined set values ​​to determine the grade of the Prince melon based on its appearance and size. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 1970-197074 [Overview of the project] [Problems that the invention aims to solve]

[0004] Melons, known as a luxury fruit, place great importance on their external appearance, including the fruit's visual appeal. During shipping, they are graded based on factors such as shape, color, netting pattern, and the presence or absence of blemishes. Generally, this visual inspection is carried out by experienced producers (skilled growers). Therefore, the grading relies heavily on the experience and subjectivity of the skilled growers, making it difficult to maintain consistency among multiple producers. Furthermore, the manual nature of the process is labor-intensive and inefficient.

[0005] Therefore, one of the objectives is to provide a melon grading system that can consistently determine the grade of melons with the same accuracy as that of skilled producers. [Means for solving the problem]

[0006] The melon grading system according to this disclosure is a melon grading system for determining the grade of a melon, comprising: an imaging unit for imaging the appearance of a melon; and a server for determining the grade of the melon based on the appearance of the melon imaged by the imaging unit. The imaging unit includes an imaging data acquisition unit for imaging the appearance of a melon in 360° and acquiring imaging data; and a transmission unit for transmitting the imaging data acquired by the imaging data acquisition unit to the server. The server includes a receiving unit for receiving the imaging data transmitted by the transmission unit; an image conversion unit for converting the imaging data received by the receiving unit into full-circumference image data of a melon; a grading unit for determining the grade of a melon from the full-circumference image data of a melon converted by the image conversion unit, based on a deep learning model that has learned to convert melon contour feature vectors and melon mesh feature vectors using data annotated by a melon grading expert; and a notification control unit for controlling the system to notify the grade of the melon determined by the grading unit. [Effects of the Invention]

[0007] According to the above melon grading system, melon grading can be performed with the same accuracy and consistency as that of a skilled producer. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a conceptual diagram showing the appearance of the melon grading system in Embodiment 1. [Figure 2] Figure 2 is a block diagram showing the configuration of the melon grading system shown in Figure 1. [Figure 3] Figure 3 shows the appearance of melons M, which are divided into four grades: A, B, C, and D. [Figure 4] Figure 4 is a magnified view of the surface of melons M, which are divided into four grades: A, B, C, and D. [Figure 5]Fig. 5 is a flowchart showing a representative configuration when grading melons using the melon grading system. [Figure 6] Fig. 6 is a diagram showing a state in which received captured data is converted into a contour image and a net pattern image. [Figure 7] Fig. 7 is a diagram showing a learning architecture of a melon grading model. [Figure 8] Fig. 8 is a diagram showing a calculation state of contour information. [Figure 9] Figs. 9(a) and 9(b) are diagrams showing embedding using class-specific margins. [Figure 10] Fig. 10 is a diagram showing an example of an image generated by data augmentation at the time point when 250 epochs of learning were performed. DETAILED DESCRIPTION OF EMBODIMENTS

[0009] SUMMARY OF EMBODIMENTS The melon grading system according to the present disclosure is a melon grading system for grading melons, comprising: an imaging unit that images the appearance of a melon; and a server that grades the melon based on the appearance of the melon imaged by the imaging unit. The imaging unit includes: an imaging data acquisition unit that acquires imaging data by imaging the appearance of the melon 360 degrees; and a transmission unit that transmits the imaging data acquired by the imaging data acquisition unit to the server. The server includes: a reception unit that receives the imaging data transmitted by the transmission unit; an image conversion unit that converts the imaging data received by the reception unit into full-circumference image data of the melon; and a determination unit that determines a grade of the melon from the full-circumference image data of the melon converted by the image conversion unit based on a deep learning model that has learned conversion into a melon contour feature vector and a melon net pattern feature vector using data annotated by an expert skilled in melon grade determination; and a notification control unit that performs control to notify the grade of the melon determined by the determination unit.

[0010] According to the melon grade determination system according to the present disclosure, it includes a determination unit having the above configuration. Since the determination unit performs determination based on a deep learning model that has learned conversion into melon contour feature vectors and melon net pattern feature vectors using data annotated by experts skilled in melon grade determination, the determination result is consistent and has an accuracy comparable to that of determination by skilled producers. Therefore, according to the above melon grade determination system, melon grade determination can be performed consistently with the same accuracy as that of skilled producers.

[0011] In the above melon grade determination system, the determination unit may determine the melon grade by determining a similarity or a distance threshold from full-circumference image data of the melon. This enables more accurate determination.

[0012] In the above melon grade determination system, a notification control unit may perform control to notify the melon grade determined by the determination unit by at least one of displaying the grade on a display and outputting a sound. This enables notification of the determined grade via at least one of visual and auditory senses, allowing the determined grade to be easily recognized.

[0013] In the above melon grade determination system, an imaging unit is rotatable 360°, and may include at least one of imaging by a rotary table on which the melon is placed and imaging of the external appearance of the melon by a camera rotatable with respect to the melon. This improves efficiency when 360° imaging of the external appearance of the melon is performed by rotating at least one of the rotary table and the camera, and improves convenience.

[0014] In the above-described melon grading system, the imaging unit may include an RGB camera. The image acquisition unit may acquire 360° video data of the melon's appearance using the RGB camera. By doing so, high-precision full-color imaging becomes possible, and images more desirable for grading can be obtained. Therefore, the accuracy of the grading can be further improved.

[0015] The melon grading method according to this disclosure is a melon grading method for determining the grade of a melon, and includes the steps of: acquiring photographic data by taking a 360° photograph of the melon's appearance; converting the photographic data into full-circumference image data of the melon; determining the grade of the melon from the converted full-circumference image data of the melon based on a deep learning model that has learned to convert the melon's contour feature vector to the melon's mesh feature vector using data annotated by a skilled melon grader; and controlling the system to notify the determined grade of the melon.

[0016] According to the above-described method for determining the grading of melons, it is possible to determine the grading of melons with the same accuracy and consistency as that of a skilled producer.

[0017] The program for a melon grading system according to this disclosure includes: an image data acquisition unit that takes 360° images of the melon's appearance and acquires image data; and a transmission unit that transmits the image data acquired by the image data acquisition unit to the server. The program for a melon grading system used in a melon grading system that determines the grade of a melon, comprising: an image unit that takes images of the melon's appearance; and a server that determines the grade of the melon based on the appearance of the melon captured by the image unit, wherein the server receives the image data transmitted by the transmission unit. This is a program for a melon grading system, which functions as a receiving unit, an image conversion unit that converts the captured data received by the receiving unit into full-circumference image data of the melon, a grading unit that determines the grade of the melon from the full-circumference image data of the melon converted by the image conversion unit, based on a deep learning model that has learned to convert the melon's contour feature vectors into the melon's mesh feature vectors using data annotated by an expert in melon grading, and a notification control unit that controls the system to notify the grade of the melon determined by the grading unit.

[0018] According to the above melon grading system program, melon grading can be performed with the same accuracy and consistency as that of a skilled producer.

[0019] The recording medium according to this disclosure includes an image data acquisition unit that captures 360° images of the melon's exterior and acquires image data, and a transmission unit that transmits the image data acquired by the image data acquisition unit to the server. The recording medium is used in a melon grading system that determines the grade of a melon, and includes an image unit that captures images of the melon's exterior and a server that determines the grade of the melon based on the images of the melon's exterior captured by the image unit. The recording medium is computer-readable, and the server is connected to a receiving unit that receives the image data transmitted by the transmission unit, and the receiving unit receives the image data. This is a computer-readable recording medium that records a program for a melon grading system, which functions as an image conversion unit that converts the aforementioned captured data into full-circumference image data of the melon, a grading unit that determines the grade of the melon from the full-circumference image data of the melon converted by the image conversion unit based on a deep learning model that has learned to convert the melon's contour feature vector to the melon's mesh feature vector using data annotated by an expert in grading melons, and a notification control unit that controls the system to notify the grade of the melon determined by the grading unit.

[0020] According to the above recording medium, melon grading can be performed with the same accuracy and consistency as that of a skilled producer.

[0021] [Specific examples of embodiments] Next, an example of a specific embodiment of the melon grading system of this disclosure will be described with reference to the drawings. In the following drawings, the same or corresponding parts are given the same reference numerals, and their descriptions will not be repeated.

[0022] (Embodiment 1) The configuration of the melon grading system in Embodiment 1 of this disclosure will now be described. Figure 1 is a conceptual diagram showing the external appearance of the melon grading system in Embodiment 1. Figure 2 is a block diagram showing the configuration of the melon grading system shown in Figure 1.

[0023] Referring to Figures 1 and 2, the melon grading system 10 in Embodiment 1 takes a photograph of the melon M's appearance and determines the melon's grade based on the captured image.

[0024] The melon grading system 10 includes an imaging unit 11 and a server 21. The imaging unit 11 includes an imaging-side control unit 12 that controls the imaging unit 11 itself, an imaging-side network interface unit 13 for connecting the imaging unit 11 to external devices, a memory 14 for storing data in the imaging unit 11, an RGB camera 15 for photographing the appearance of the melon M, an LED light 16 for illuminating the melon M to be photographed, and a rotatable rotating table 17 on which the melon M to be photographed is placed. The memory 14 and the RGB camera 15 function as an imaging data acquisition unit that photographs the appearance of the melon M in 360° and acquires imaging data. The imaging unit 11 is configured to communicate with the server 21. The imaging-side network interface unit 13 functions as a transmission unit for sending data and a reception unit for receiving data. The imaging unit 11 and the server 21 may be connected, for example, via a network, or they may be connected by wired or wireless connections. In this embodiment, the imaging unit 11 is configured to exchange data with the server 21 via Wi-Fi connection.

[0025] The imaging unit 11 includes a box 18 for housing the melon M when photographing it. The box 18 is roughly rectangular in shape. The box 18 has an opening 19 in the upper wall. Through this opening 19, the object to be photographed, i.e., the melon M to be graded, is placed inside and outside the box 18. That is, the opening 19 is provided in a size and position that allows the user to put the melon M in and out. The inner wall of the box 18 is black. That is, the box 18 constitutes a darkroom. The opening 19 is closed when photographing the exterior of the melon M. An RGB camera 15, an LED light 16, and a rotating table 17 are installed in predetermined locations inside the box 18.

[0026] During shooting, the melon M is placed on the rotating table 17. The LED lighting 16 can illuminate the melon M placed on the rotating table 17. The RGB camera 15 is installed inside the box 18 and can capture the exterior of the melon M placed on the rotating table 17. During shooting, the rotating table 17 rotates (rotates on its axis) while the melon M placed on the rotating table 17 is illuminated by the LED lighting 16. The RGB camera 15 is a so-called fixed-point camera, and the exterior of the melon M is captured in 360° as the rotating table 17 rotates. The rotating table 17 also functions as a mass meter, and the mass of the melon M placed on the rotating table 17 can also be measured. Alternatively, instead of the rotating table 17 rotating, the RGB camera 15 could be configured to rotate to capture a 360° view of the melon M's exterior. Of course, both could be configured to rotate. Alternatively, instead of video, still images could be taken in sequence to capture a 360° view of Melon M. In this case, for example, the 360° full-circumference image data of Melon M could be derived from a total of eight photographic images taken by rotating it 45° at a time.

[0027] Next, the configuration of server 21 will be described. Server 21 includes a server control unit 22 that controls server 21 itself, a server network interface unit 23 for connecting server 21 to external devices, and a server hard disk 24 for storing data in server 21. The server network interface unit 23 functions as a transmission unit for sending data and a reception unit for receiving data. The server control unit 22 also includes an image conversion unit 25, a determination unit 26, and a notification control unit 27. The image conversion unit 25 converts the melon image data received by the server network interface unit 23 as a reception unit, i.e., 360° video data of the melon's appearance, into full-circumference image data of the melon. The determination unit 26 uses data annotated by an expert in grading melons M to determine the melon's grade, and based on a deep learning model that has learned to convert melon contour feature vectors and melon mesh feature vectors, it determines the similarity from the full-circumference image data of the melon converted by the image conversion unit 25 and determines the grade of the melon. The notification control unit 27 controls the system to notify the grade of the melon determined by the determination unit 26. These components will be described in detail later.

[0028] The melon grading system 10 further includes a display 29. The display 29 is connected to a server 21 and can display data from the server 21.

[0029] Next, we will briefly explain the grades of melon M determined by the melon grading system 10. Figure 3 shows the appearance of melon M divided into four grades: A, B, C, and D. Figure 4 shows a magnified view of the surface of melon M divided into four grades: A, B, C, and D. Referring to Figures 3 and 4, Grade A is high quality, with a uniform netting density and little distortion in the shape of the fruit. Grade B is medium quality, with noticeable irregularities in the netting, but little distortion in the shape of the fruit. Grade C is low quality, with distortions in the shape of the fruit, such as being flattened, elongated, or convex. Grade D is low quality, with scratches, dirt, black mold, and significant irregularities in the netting on the fruit. Note that Grade C is judged by shape, so the netting is not considered.

[0030] In this grading system, the priority order is D > C > B > A, with lower quality grades taking precedence (for example, a melon with characteristics of C and D would be graded D). Grades A, B, and D are determined by the characteristics of the netting, while grade C is determined solely by the characteristics of the fruit's shape. There are no quantitative criteria for judging the density or distortion of the netting.

[0031] Next, a method for determining the grade of a melon using the melon grading system 10 in this embodiment will be described. Figure 5 is a flowchart showing a typical configuration when determining the grade of a melon using the melon grading system 10.

[0032] Referring to Figure 5, first, a user requesting the grade of melon M places melon M on the rotating table 17 inside box 18 using the opening 19. Then, they close the opening 19 and press the start button to begin determining the grade of melon M. When the start button is detected to have been pressed (in Figure 5, step S11, YES; the following steps are omitted), the determination of the grade of melon M begins. Specifically, the following processes are performed.

[0033] First, upon detecting that the start button has been pressed, the camera begins to photograph the exterior of Melon M (S12). Specifically, the LED lighting 16 is turned on to illuminate the surface of Melon M. Then, the rotating table 17 begins to rotate, and the RGB camera 15 begins to capture video.

[0034] The rotary table 17 is rotated 360° to photograph the exterior of the melon M. Then, images of the entire circumference of the melon M's exterior, i.e., 360°, are taken, and the shooting is completed (S13). The memory 14, which functions as an image acquisition unit, takes 360° images of the melon's exterior and acquires the shooting data, i.e., video data. After that, the acquired video data is transmitted to the server 21 by the shooting-side network interface unit 13 (S14).

[0035] Next, the processing on the server 21 side will be explained. The server 21 receives the captured data transmitted from the imaging unit 11 via the server network interface unit 23 (S15). Then, it processes the received captured data. Specifically, first, the image conversion unit 25 converts the received captured data into a contour image and a mesh image (S16).

[0036] Figure 6 shows the state after the received image data has been converted into a contour image and a mesh image. Referring to Figure 6, a total of 8 frames are extracted as images from the received 360° panoramic video data 31 so that the orientation of the fruit is at 45° intervals. The extracted frame images are binarized so that the background region and the fruit region can be distinguished. Then, by applying a shrinkage process and an expansion process 20 times each to the fruit region, only the thin stem portion is removed from the fruit region image to generate a contour image 32.

[0037] Furthermore, for the mesh image, the first frame is extracted as an image from the 360° panoramic video data 31. The extracted frame image is then binarized to distinguish between the background image and the fruit region. Next, elliptic fitting is performed on the binarized image to obtain information on the center coordinates, height, and width of the fruit. Then, a region with a width of 1 pixel and height is extracted from the center coordinates. This is done for all 599 frames, and the images extracted from each frame are stitched together horizontally to generate an image with a width of 599 pixels. Note that this method is the same as the method for generating a simple panoramic image. Finally, the generated image is resized to 112 pixels vertically and 448 pixels horizontally to become the mesh image 33.

[0038] In this way, the image conversion unit 25 converts the received 360° video data into a contour image 32 and a mesh image 33. Next, the grade of the melon M is determined from the converted contour image 32 and mesh image 33 (S17).

[0039] First, let's explain the learning architecture of the melon grading model. Figure 7 shows the learning architecture of the melon grading model. Referring to Figure 7, the melon grading model is broadly composed of a part that converts contour images into contour image feature vectors and a part that converts mesh images into mesh feature vectors. Finally, both feature vectors are combined, linearly transformed, and then L2 normalized to obtain the feature vector for one melon. Other features related to grading may also be combined here. Note that if the correct label for data x of one melon is grade 4, then y ∈ {0, 1, 2, 3}, but separately, labels are also assigned to the contour image and mesh image. Contour image x Shape Label y Shape ∈{0,1} indicates whether the image is grade C or not, and the mesh image x Net The label yNet{0,1,2} represents the grade of the image (A, B, or D).

[0040] Embedding contour images into contour image vectors uses statistical values ​​of contour information calculated from the contour images. Figure 8 shows the state of contour information calculation. As shown in Figure 8, the width of the fruit indicated by arrow 42, the height indicated by arrow 43, the aspect ratio calculated from width / height, and the length from the fruit center 44 to the contour 45 at 10° intervals indicated by arrows 46a, 46b, 46c, 46d, and 46e are calculated from the contour image 41, and the average, minimum, maximum, and standard deviation of all 8 images are calculated. Other statistical values ​​related to grade determination may be combined here. Note that arrow 46a indicates the length of part 1, arrow 46b indicates the length of part 2, arrow 46c indicates the length of part 3, arrow 46d indicates the length of part 4, and arrow 46e indicates the length of part 36. The parts correspond to the positions when the melon M is divided into 36 sections in the circumferential direction. Next, from all the calculated statistics, seven statistics effective for determining C-grade yShape with distorted fruit shapes are selected using a recursive feature reduction method such as Random Forest. Table 1 shows the selected statistics calculated from the contour images. Underlined items in Table 1 indicate features selected during feature selection. This feature selection is performed before training using ArcFace, etc., and thereafter, the Embedding module in Figure 7 only calculates the seven contour features from the input contour image. [Table 1]

[0041] The goal of extracting mesh feature vectors is not to extract features from the entire image from a mesh image, but rather to detect local, class-specific features of a few to tens of pixels that represent the image's grade. Therefore, an encoder constructed using a Convolutional Neural Network (CNN) is trained using MC-Loss (Mutual-Channel Loss) as the loss function. However, the loss function is not limited to MC-Loss. First, the input mesh image is converted into a feature map by the encoder. At this time, instead of using the highly abstract feature map output by the final layer of the CNN's convolutional layers, the output of an intermediate convolutional layer with a lower level of abstraction is used to utilize local features. For example, when using ResNet50, instead of using the 7x7 feature map of the Conv5 block, the 14x14 feature map of the preceding Conv4 block is used.

[0042] The transformed feature map is given by F∈R, where height H, width W, and number of channels N. W×H×N Let's assume that when using MC-Loss, we need to set N = C × ε. Here, C is the number of classification classes, and ε is the number of channels in the feature map for each class. As a result, the feature map corresponding to class i can be expressed as Fi ∈ Rε × W × H, i = 0, 1, ..., C-1. In MC-Loss, the loss for this feature map is defined as follows.

number

number

number

[0043] Ldiv(F) is the loss relating to the diversity of extracted features, and is represented by Equation 4. [Formula] Here, the function h(·) is defined as follows. [Formula] This Ldiv gives a bonus when the per-class feature maps extracted by the encoder do not overlap in the channel direction. Specifically, as shown in Equation 1, a value multiplied by parameter λ is subtracted from the loss. This loss function acts to make the encoder learn to extract diverse features with as little overlap as possible.

[0044] Thereafter, an Activation Map is generated. The Activation Map aggregates feature maps to visualize which region in the image space contributes features to the estimation result of the model. This is known as one of explainable artificial intelligence (XAI) techniques. In Class activation map (CAM), Activation Map:A∈R W×H×CThis is calculated using the feature map output by the encoder and the weights of the fully connected layer. In this system, after applying a 1x1 kernel-size maximum pooling to the feature map Fi of each class output by the encoder, it is calculated by calculating Softmax between classes. The Activation Map is a map that represents the contribution of each class on the feature space W x H such that the sum of the channel directions is 1.0. Furthermore, the Global Activation Map (GA∈R) is obtained by aggregating the Activation Maps of each class. W×H This is then obtained by applying maximum pooling to A again. Therefore, GA can be described as a map that represents parts where the values ​​of features specific to a particular class are large, and parts where the values ​​of parts that could be any class are small.

number

number

[0045] The contour feature vector and the mesh feature vector are concatenated, and after a linear transformation, the L2 normalized vector is obtained as the melon feature vector F. ALL ∈R N+7 Let's assume this. We embed this feature vector into an angular distance space using ArcFace, with vector similarity as the index. The ArcFace loss function can be expressed by the following equation.

number

number

[0046] Finally, the MC-Loss loss and the ArcFace loss are multiplied by the hyperparameter α = {0.0~1.0} and the sum is used as the overall model loss L for training.

number

number

[0047] The determination based on maximum similarity is as follows: The average feature vector REF of each class is obtained in advance using the training data. i=0,1,…,C-1 We will calculate the following: Feature vector Fn of the melon to be judged. ALL and average feature vector REF i The cosine similarity is calculated, and the class of the most similar average feature vector is defined as the melon's grade yn'.

number

number

number

[0048] The grade of the melon is determined in this manner. After the determination, the notification control unit 27 controls the system to notify the determined grade of the melon. Specifically, the grade is notified by displaying it on the display 29 connected to the server 21 (S18).

[0049] This melon grading system 10 includes a grading unit 26 with the above configuration. Since the grading unit 26 performs grading based on a deep learning model, its grading results are consistent and have an accuracy comparable to that of a skilled producer. Therefore, this melon grading system 10 can perform melon grading with the same accuracy and consistency as a skilled producer.

[0050] According to this embodiment, the determination unit 26 determines the grade of the melon by determining the similarity or distance threshold from the full-circumference image data of the melon. Therefore, the determination can be made with greater accuracy.

[0051] According to this embodiment, the notification control unit 27 is controlled to notify the user by displaying the grade of the melon determined by the determination unit 26 using the display 29. Therefore, the determined grade can be visually notified, and the determined grade can be easily understood.

[0052] According to this embodiment, the imaging unit 11 is 360° rotatable and includes a rotating table 17 on which the melon is placed. Therefore, efficiency can be improved when photographing the melon's exterior in 360°, and convenience can be enhanced. The imaging unit 11 is 360° rotatable and may include at least one of the following: imaging using the rotating table 17 on which the melon is placed, and imaging of the melon's exterior using a camera that is rotatable relative to the melon. By doing so, efficiency can be improved when photographing the melon's exterior in 360° by rotating at least one of the rotating table 17 and the camera, and convenience can be enhanced. The camera may be capable of capturing monochrome images. Furthermore, while rotating the rotating table 17, the camera may also be rotated, for example, in the opposite direction to the rotating table 17 for imaging.

[0053] According to this embodiment, the imaging unit 11 includes an RGB camera 15. The memory 14, acting as the video acquisition unit, acquires 360° video data of the melon's appearance using the RGB camera. This enables high-precision full-color imaging, allowing for the acquisition of images more suitable for grading. Consequently, the accuracy of the grading can be further improved.

[0054] (Other embodiments) In the above embodiment, the notification control unit notifies by displaying the determined grade of melon M on a display. However, the notification control unit is not limited to this and may also notify the grade of melon M by voice. That is, the notification control unit may be controlled to notify the grade of the melon determined by the determination unit by displaying it on a display and by emitting a sound, or by at least one of the other. In this way, the determined grade can be notified visually and audibly, making it easy to grasp the determined grade. In addition to display, the notification may also be made by voice. Furthermore, the notification control unit may be controlled to store the determination result on the server hard disk along with the notification.

[0055] Furthermore, the melon grading method according to this disclosure is a melon grading method for determining the grade of a melon, and includes the steps of: acquiring photographic data by taking a 360° photograph of the melon's appearance; converting the photographic data into full-circumference image data of the melon; determining the grade of the melon from the converted full-circumference image data of the melon based on a deep learning model that has learned to convert the melon's contour feature vector to the melon's mesh feature vector using data annotated by a skilled melon grader; and controlling the system to notify the determined grade of the melon.

[0056] According to the above-described method for determining the grading of melons, it is possible to determine the grading of melons with the same accuracy and consistency as that of a skilled producer.

[0057] Furthermore, the program for a melon grading system according to this disclosure includes: an image data acquisition unit that takes 360° images of the melon's appearance and acquires image data; and a transmission unit that transmits the image data acquired by the image data acquisition unit to the server. The program for a melon grading system used in a melon grading system that determines the grade of a melon, comprising: an image unit that takes images of the melon's appearance; and a server that determines the grade of the melon based on the appearance of the melon captured by the image unit, wherein the server receives the image data transmitted by the transmission unit. This is a program for a melon grading system that functions as a receiving unit that receives data, an image conversion unit that converts the captured data received by the receiving unit into full-circumference image data of the melon, a grading unit that determines the grade of the melon from the full-circumference image data of the melon converted by the image conversion unit, based on a deep learning model that has learned to convert the melon's contour feature vector to the melon's mesh feature vector using data annotated by an expert in melon grading, and a notification control unit that controls the system to notify the grade of the melon determined by the grading unit.

[0058] According to the above melon grading system program, melon grading can be performed with the same accuracy and consistency as that of a skilled producer.

[0059] Furthermore, the recording medium according to this disclosure includes an image data acquisition unit that takes 360° images of the melon's exterior and acquires image data, and a transmission unit that transmits the image data acquired by the image data acquisition unit to the server, and is used in a melon grading system that determines the grade of a melon, comprising an image unit that takes images of the melon's exterior and a server that determines the grade of the melon based on the image of the melon taken by the image unit, and is a computer-readable recording medium, wherein the server is connected to a receiving unit that receives the image data transmitted by the transmission unit, and the receiving unit receives This is a computer-readable recording medium that records a program for a melon grading system, which functions as an image conversion unit that converts the transmitted image data into full-circumference image data of the melon, a grading unit that determines the grade of the melon from the full-circumference image data of the melon converted by the image conversion unit based on a deep learning model that has learned to convert the melon's contour feature vector to the melon's mesh feature vector using data annotated by an expert in grading melons, and a notification control unit that controls the system to notify the grade of the melon determined by the grading unit.

[0060] According to the above recording medium, melon grading can be performed with the same accuracy and consistency as that of a skilled producer. [Examples]

[0061] The dataset used in the basic experiment consisted of 360° video data of 122 melons of each grade and 80 unannotated melons, captured by the above-mentioned camera unit 11. Data acquisition was carried out 18 times from 2021 to December 2021, and the grade labels were annotated by the same experienced producer.

[0062] The grade classification model used in this experiment utilizes the seven underlined features shown in Table 1 as contour features, and the encoder used to extract mesh features is a ResNet50 pre-trained on ImageNet. For MC-Loss settings, the input image size to the encoder was a 114×448×3 color image, with hyperparameters ε=5 and λ=10. The encoder adds one convolutional layer after the Conv4 block of ResNet50, outputting a 7×28×15 feature map. For ArcFace settings, the hyperparameters were s=10, m=[0.8,0.4,0.4,0.4], and α=2. Training was performed using SDG with momentum=0.9, patch size 32, and learning rate 0.001, with e -0.01 The model was trained for 250 epochs with attenuation applied. The grade determination process for the test data was performed using the maximum similarity method described above.

[0063] An experiment was conducted to assess the accuracy of grade determination using this system. In this experiment, data from 122 melons of each grade were used, with N∈{32, 52, 72, 92} melons per grade used as training data and 30 melons per grade used as test data. First, grade determination was performed using the training data. This was repeated three times using cross-validation to examine the accuracy rate for different numbers of data points used for training. For comparison, experiments were also conducted using standard ResNet50 and ViT-B / 16, and using the judgment of skilled human producers. In the skilled producer experiment, participants judged the grade by viewing a full-circumference image of the melon. In the experiments using ResNet50 and ViT-B / 16, both used pre-trained ImageNet models, inputting the mesh image and concatenating the contour feature vectors after GAP, and fine-tuning was performed as a general classification problem.

[0064] As a result of the training, we confirmed the convergence of losses in all models. Table 2 shows the accuracy rates for different numbers of training data. The highest accuracy rate was 82.1%, achieved when the proposed model (N=92) was trained with AGDA enabled. This is comparable to the 84.9% accuracy achieved by skilled human producers, confirming that the model can determine grades with near-human accuracy. Furthermore, even with a small number of data points (N=32), the proposed model was found to be more accurate than the comparison model. [Table 2]

[0065] We conducted an experiment to investigate the relationship between feature vector embedding using class-specific margins and grade classification accuracy. Using N ∈ {32, 92} of training data, we varied the margin settings from no margin (m=[0.0,0.0,0.0,0.0]) to all-class margin (m=[0.8,0.8,0.8,0.8]) and trained the model without ADGA. This was repeated three times using cross-validation, and the accuracy for each margin setting was calculated. [Table 3] Table 3 shows the accuracy rates for each class margin. Regardless of the amount of training data, the highest accuracy rate was achieved when the margin was set large only for grade A and half for the other grades. When there was no margin, the discriminative ability between classes was low and the accuracy rate was low. Conversely, when the margin was too large, the discriminative ability between classes for the training data was high, but the test data tended to be distributed farther away from the training data, and the accuracy of grade determination also decreased. When the margin was set large only for grade A and half for the others, the training data and test data were distributed relatively similarly.

[0066] The effectiveness of AGDA and the validity of the data-augmented images will be confirmed. Figure 10 shows an example of an image generated with data augmentation after 250 epochs of training. Figure 10 shows an example of an image generated with data augmentation after 250 epochs of training. In Figure 10, three images 61, 62, and 63 are shown, and the dotted lines 64, 65, 66, and 67 in Figure 10 represent regions synthesized using Activation Maps, respectively.

[0067] From the experimental results in Table 2 above, comparing the results of the proposed model with and without AGDA, it was confirmed that applying data augmentation improved the accuracy rate by an average of 0.9%. Furthermore, from Figure 10, it was confirmed that the generated images were natural mesh images that reflected the feature representation of the original labels.

[0068] To verify how well the grading results obtained using the grading model matched the judgments of experienced producers, a qualitative evaluation of the model's judgments was conducted. From the results of the above experiment, it was found that even experienced producers who had annotated the melons made errors of about 15% when they re-graded them at a later date. Some melons included individuals that even experienced producers were unsure which grade to classify. Therefore, in addition to evaluating the accuracy rate using the dataset, a qualitative evaluation was also conducted. Experienced producers were presented with video data of the melons to be graded and the grading results of the grading model, and were asked to evaluate the validity of the model's judgment on a four-point scale.

[0069] 1. The judgments are in agreement. 2. They don't match, but it's acceptable. 3. They don't match and are unacceptable. 4. Clearly wrong (irrelevant). [Table 4]

[0070] Table 4 shows the results of two experienced growers evaluating the model's judgments for each of 80 unannotated fruits. The agreement rate was calculated as the percentage of fruits where the experienced growers rated "1," and the acceptance rate as the percentage where they rated "1 or 2." The agreement rate was 71.3% and the acceptance rate was 83.8%. Furthermore, the acceptance rate for Grade D was 100%, confirming that the model's judgment was consistent with that of the experienced growers.

[0071] The embodiments disclosed herein should be understood to be illustrative in all respects and not restrictive in any way. The scope of the present invention is defined by the claims and is intended to include all modifications in the sense and scope equivalent to the claims. [Explanation of Symbols]

[0072] 10 Melon grading system, 11 Imaging unit, 12 Imaging side control unit, 13 Imaging side network interface unit, 14 Memory, 15 RGB camera, 16 LED lighting, 17 Rotating table, 18 Box, 19 Opening, 21 Server, 22 Server control unit, 23 Server network interface unit, Server hard disk, 25 Image conversion unit, 26 Judgment unit, 27 Notification control unit, 29 Display, 31 Full-circumference video data, 32,41 Contour image, 33 Mesh image, 42,43,46a,46b,46c,46d,46e Arrow, 44 Fruit center, 45 Contour, 51,52,53,54,55,56 Code, 61,62,63 Image, 64,65,66,67 Dotted line.

Claims

1. A melon grading system for determining the grade of melons, The system comprises a camera unit that photographs the appearance of a melon, and a server that determines the grade of the melon based on the appearance of the melon photographed by the camera unit. The aforementioned imaging unit is A shooting data acquisition unit that captures shooting data by taking a 360° photograph of the melon's exterior, The system includes a transmission unit that transmits the image data acquired by the image data acquisition unit to the server, The aforementioned server, A receiving unit that receives the image data transmitted by the transmitting unit, An image conversion unit converts the image data received by the receiving unit into image data of the entire circumference of the melon, Based on a deep learning model that has learned to convert the melon's contour feature vector to the melon's mesh feature vector using data annotated by an expert in grading the melon, the grading unit obtains a feature vector by concatenating the melon's contour feature vector and the melon's mesh feature vector from the full-circumference image data of the melon converted by the image conversion unit, and determines the grade of the melon from the feature vector. A melon grading system, comprising: a notification control unit that controls the system to notify the grade of the melon determined by the determination unit; and a notification control unit.

2. The melon grading system according to claim 1, wherein the determination unit determines the grade of the melon from the circumferential image data of the melon by similarity, or by comparing the cosine distance between the melon's feature vector and the average feature vector with a threshold value for each class.

3. The melon grading system according to claim 1 or 2, wherein the notification control unit controls the system to notify the grade of the melon determined by the determination unit by displaying it on a display and by emitting sound.

4. The melon grading system according to claim 1 or 2, wherein the imaging unit is 360° rotatable and includes at least one of imaging using a rotating table on which the melon is placed and imaging the appearance of the melon using a camera rotatable relative to the melon.

5. The aforementioned imaging unit includes an RGB camera, The melon grading system according to claim 1 or 2, wherein the shooting data acquisition unit acquires 360° shooting data of the melon's appearance using the RGB camera.

6. The melon grading system according to claim 1 or claim 2, wherein the feature vector is obtained by linear transformation followed by L2 normalization.

7. A method for determining the grade of melons, The process involves taking a 360° photograph of the melon's exterior and acquiring the photographic data. A step of converting the aforementioned shooting data into image data of the entire circumference of the melon, The process involves using data annotated by an expert in grading melons to learn how to convert the melon's contour feature vector to the melon's mesh feature vector, then obtaining a feature vector by concatenating the melon's contour feature vector and the melon's mesh feature vector from the converted full-circumference image data of the melon, and determining the melon's grade from the feature vector, and A method for determining the grade of a melon, comprising the step of controlling the system to notify the determined grade of the melon.

8. A program for a melon grading system used in a melon grading system that includes: an image data acquisition unit that takes 360° images of the melon's exterior and acquires image data; a transmission unit that transmits the image data acquired by the image data acquisition unit to a server; an image unit that takes images of the melon's exterior; and a server that determines the grade of the melon based on the image of the melon taken by the image unit, the program for a melon grading system used in a melon grading system, the program for a melon grading system that determines the grade of the melon, the program for a melon grading system that includes: an image data acquisition unit that takes 360° images of the melon's exterior and acquires image data; a transmission unit that transmits the image data acquired by the image data acquisition unit to a server; an image unit that takes images of the melon's exterior; and a server that determines the grade of the melon based on the image of the melon taken by the image unit. The aforementioned server, A receiving unit that receives the image data transmitted by the transmitting unit, An image conversion unit converts the captured data received by the receiving unit into image data of the entire circumference of the melon. Based on a deep learning model that has learned to convert the melon's contour feature vector to the melon's mesh feature vector using data annotated by an expert in grading the melon, the image conversion unit obtains a feature vector by concatenating the melon's contour feature vector and the melon's mesh feature vector from the full-circumference image data of the melon converted by the image conversion unit, and determines the grade of the melon from the feature vector, and A program for a melon grading system that functions as a notification control unit that controls the system to notify the grade of the melon determined by the determination unit.

9. A computer-readable recording medium used in a melon grading system for determining the grade of a melon, comprising: an image data acquisition unit that takes 360° images of the melon's exterior and acquires image data; a transmission unit that transmits the image data acquired by the image data acquisition unit to a server, an image unit that takes images of the melon's exterior; and a server that determines the grade of the melon based on the image of the melon taken by the image unit, The aforementioned server, A receiving unit that receives the image data transmitted by the transmitting unit, An image conversion unit converts the captured data received by the receiving unit into image data of the entire circumference of the melon. Based on a deep learning model that has learned to convert the melon's contour feature vector to the melon's mesh feature vector using data annotated by an expert in grading the melon, the image conversion unit obtains a feature vector by concatenating the melon's contour feature vector and the melon's mesh feature vector from the full-circumference image data of the melon converted by the image conversion unit, and determines the grade of the melon from the feature vector, and A computer-readable recording medium containing a program for a melon grading system, which functions as a notification control unit that controls the system to notify the grade of the melon determined by the determination unit.

Images