Information processor, information processing method, method for producing learning model, and program

Abstract

To provide a technique capable of appropriately determining a laver grade while saving manpower.SOLUTION: An information processor that determines a laver grade acquires a score Q based on brightness information E1 of an image 212 obtained by imaging a laver (laver image) and color information E2 of the image 212. The color information E2 shows color characteristic in a direction connecting red and blue in a predetermined color space (e.g., YCbCr color space). Further, the information processor determines a laver grade on the basis of the score Q.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing technique for determining the grade of seaweed. [Background technology]

[0002] There is technology to determine the grade of seaweed. Currently, humans mainly judge the grade of seaweed by looking at its appearance. More specifically, humans (inspectors) judge the grade of seaweed by distinguishing the color of the seaweed, etc.

[0003] In addition, a technology has been proposed to mechanize the grading of seaweed (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2003-047445 A Summary of the Invention [Problem to be solved by the invention]

[0005] Considering the burden on inspectors, etc., it is preferable to proceed with mechanization of grading (reducing the number of workers required).

[0006] Incidentally, as described above, when an inspector (human) judges the grade of seaweed, the inspector takes into consideration the color of the seaweed.

[0007] On the other hand, in the above-mentioned prior art (Patent Document 1), although the uniformity of color tone is utilized, the color of the seaweed itself is not utilized for the judgment.

[0008] Considering that humans also take the color of seaweed into consideration when judging the grade of seaweed, in order for a machine to make an appropriate judgment close to that of a human when judging the grade of seaweed, it is preferable for the machine to appropriately take the color of the seaweed into consideration.

[0009] Therefore, an objective of this invention is to provide a technology that can appropriately determine the grade of seaweed while reducing the number of workers required. [Means for solving the problem]

[0010] In order to solve the above problem, the information processing device of the present invention is characterized by having a control unit that calculates a score for determining the grade of the seaweed based on brightness information of an image of the seaweed and color information of the image that indicates color characteristics in a direction connecting red and blue in a specified color space.

[0011] The predetermined color space may be any one of a YCbCr color space, a YUV color space, a YPbPr color space, an RGB color space, and a Lab color space.

[0012] The luminance information and the color information may be calculated based on average values ​​obtained by averaging a plurality of pixel values ​​in the image for each channel in the predetermined color space.

[0013] The brightness information may be information on a first principal component of the image based on the results of a principal component analysis of the average values ​​obtained for a plurality of images of seaweed, and the color information may be information on a second principal component of the image based on the results of the principal component analysis.

[0014] The score for the image may be obtained based on the analysis result of an interaction linear analysis that takes into account the interaction between two explanatory variables, the luminance information and the color information.

[0015] The control unit may calculate the score based also on an index value related to a degree of focus of the image.

[0016] The score for the image may be an evaluation value output by inputting the luminance information and color information for the image into a learning model trained based on the luminance information, the color information, and grade correct answer data for a plurality of learning images.

[0017] The learning model is a first learning model, and the control unit calculates an improved score for determining the grade of the seaweed based on a second learning model, and the second learning model is a learning model that takes as input the score output from the first learning model for an input image and an index value related to the degree of focus of the input image, and outputs the improved score, and is a learning model trained using multiple teacher data, and the control unit may input the score output from the first learning model for the image to be inferred and the index value related to the degree of focus of the image into the second learning model, and obtain the improved score output from the second learning model.

[0018] In order to solve the above problem, the information processing method of the present invention is characterized by comprising a step of: a) calculating a score for determining the grade of the seaweed based on luminance information of an image of the seaweed and color information of the image, the color information indicating color characteristics in a direction connecting red and blue in a predetermined color space.

[0019] In order to solve the above problems, the present invention provides a program for causing a computer to execute the above information processing method.

[0020] In order to solve the above problems, the present invention provides a method for producing a learning model for determining the grade of seaweed, which is characterized by comprising a step of learning the learning model based on multiple teacher data, using as input brightness information of an image of seaweed and color information of the image which indicates color characteristics in a direction connecting red and blue in a predetermined color space, and outputting a score related to the grade of the seaweed.

[0021] In order to solve the above problem, the information processing device of the present invention is characterized in that it includes a control unit that obtains a first principal component score and a second principal component score of an image to be inferred based on the analysis results of a principal component analysis in a predetermined color space on a plurality of images of seaweed, inputs the first principal component score and the second principal component score of the image to be inferred to a learning model that has learned the relationship between the first principal component score and the second principal component score of each image and the grade of seaweed in each image using a plurality of teacher data, and obtains a score output from the learning model for determining the grade of seaweed in the image to be inferred.

[0022] In order to solve the above-mentioned problems, the present invention provides a method for producing a learning model, comprising the steps of: a) performing a principal component analysis in a predetermined color space on a plurality of images of seaweed, and obtaining a first principal component and a second principal component through the principal component analysis; and b) using a plurality of training data to machine-train a learning model in which the first principal component score and the second principal component score of an input image are input, and a score for judging the grade of the seaweed in the input image is output.

[0023] In order to solve the above problem, the program according to the present invention is characterized in that it is a program for causing a computer to execute the method for producing a learning model.

[0024] In order to solve the above problem, the information processing method of the present invention comprises the steps of: a) acquiring a first principal component score and a second principal component score of an image to be inferred based on the results of a principal component analysis in a predetermined color space on a plurality of images of seaweed; and b) inputting the first principal component score and the second principal component score of the image to be inferred into a learning model that has learned the relationship between the first principal component score and the second principal component score of each image and the grade of seaweed in each image using a plurality of teacher data, and acquiring a score output from the learning model for determining the grade of seaweed in the image to be inferred.

[0025] In order to solve the above problems, a program according to the present invention is a program for causing a computer to execute the information processing method. Effect of the Invention

[0026] According to the present invention, the grade of seaweed is determined taking into consideration the color of the seaweed, making it possible to appropriately determine the grade of seaweed while reducing the number of workers required. [Brief description of the drawings]

[0027] [Figure 1] 1 is a block diagram showing a grade determination system according to a first embodiment. [Diagram 2] 1 is a schematic diagram of a conveying device and an imaging device viewed from above. FIG. [Diagram 3] FIG. 2 is a schematic side view of the transport device and the imaging device. [Figure 4] FIG. 1 is a conceptual diagram showing an overview of a process. [Diagram 5] 1 is a flowchart showing a process in a grade determination device (information processing device). [Figure 6] 13 is a flowchart showing a generation process of teacher data. [Figure 7] 13 is a flowchart showing an inference process (grade determination process). [Figure 8] FIG. 1 is a conceptual diagram illustrating an overview of principal component analysis. [Figure 9] FIG. 1 is a diagram showing the significance of the first principal component. [Figure 10] FIG. 1 is a diagram showing the significance of the second principal component. [Figure 11] 13 is a graph in which combinations of the first principal component score and the second principal component score are plotted for a plurality of images. [Figure 12] FIG. 13 shows the correct answer scores and point shapes for each grade. [Figure 13] FIG. 12 is a diagram showing the learning results based on the learning model superimposed on FIG. [Figure 14] FIG. 13 is a diagram showing the estimated scores for each sample image. [Figure 15]FIG. 11 is a diagram comparing two images. [Figure 16] FIG. 11 is a diagram showing a learning model according to the second embodiment. [Figure 17] 13 is a graph in which combinations of index values ​​relating to the degree of focus and scores in the first embodiment are plotted for a plurality of images. [Figure 18] 18 is a diagram showing the learning results of the latter-stage learning model superimposed on FIG. 17. [Figure 19] FIG. 13 shows the refinement scores for each sample image. [Figure 20] This is a conceptual diagram that shows the RGB color space as a cube. [Figure 21] FIG. 21 is a conceptual diagram that diagrammatically represents the Lab color space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0029] <1. First embodiment> <1-1. System Overview> FIG. 1 is a block diagram showing a grading system 1 for grading seaweed.

[0030] As shown in Fig. 1, the grading system 1 includes a photographing device 20 that photographs an image of the nori seaweed, and a grading device 30 that judges the grade of the nori seaweed based on the image (the photographed image of the nori seaweed). The grading system 1 also includes a transport device 10 that transports the nori seaweed. The photographing device 20 is provided on the transport path of the transport device 10, and is capable of photographing the surface of the nori seaweed transported by the transport device 10.

[0031] Since the grade determination device 30 is a device that processes images, it is also referred to as an image processing device. Similarly, the grade determination system 1 is also referred to as an image processing system. Since the grade determination device 30 is a device that processes information, it is also referred to as an information processing device, and similarly, the grade determination system 1 is also referred to as an information processing system.

[0032] <1-2. Grade determination device 30> As shown in FIG. 1, the class determination device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, a communication unit 34, and an operation unit 35.

[0033] The controller 31 is a control device that is built into the grade determination device 30 and controls the operation of the grade determination device 30. The controller 31 also controls the operations of the transport device 10 and the photography device 20.

[0034] The controller 31 is configured as a computer system including one or more hardware processors (e.g., a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 realizes various processes by executing, in the CPU or the like, a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 32. The program (more specifically, a group of program modules) (also referred to as a "program product") may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the class determination device 30. Alternatively, the program may be downloaded via a communication network or the like and installed in the class determination device 30.

[0035] Specifically, the controller 31 executes processing at the generation stage of teacher data (phase PH1 (see Figure 4, etc.)), processing at the generation stage of the learning model (learning stage processing) (phase PH2), and processing at the inference stage using the learning model 400 (phase PH3), etc.

[0036] The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) and / or a solid state drive (SSD), etc. The storage unit 32 stores the learning model 400 (including learning parameters and programs related to the learning model) (and thus the trained model 420) and the like.

[0037] The communication unit 34 is capable of performing network communication via a network. In this network communication, various protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol) are used. By using this network communication, the class determination device 30 can transmit and receive various data (photographed image data, etc.) to and from a desired destination (for example, the photographing device 20, etc.) via wire or wirelessly.

[0038] The operation unit 35 includes an operation input unit 35a that receives operation input to the grade determination device 30, and a display unit 35b that displays and outputs various information. A mouse and a keyboard are used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both a part of the operation input unit 35a and a part of the display unit 35b may also be provided.

[0039] The grade determination device 30 is also referred to as a teacher data generation device because it has a function of generating teacher data for machine learning. In addition, the grade determination device 30 is also referred to as a learning model generation device because it has a function of machine learning the learning model 400 using the teacher data (a function of generating a learning model (trained model)). In addition, the grade determination device 30 is also referred to as an inference device because it is a device that executes inference regarding the grade of seaweed using the trained learning model 400.

[0040] In this embodiment, various processes (functions) are realized by one class determination device 30, but the present invention is not limited to this, and various processes may be shared and realized by multiple devices. For example, the teacher data generation process in the first phase PH1 (see the upper part of FIG. 4), the learning process (learning model generation process) in the second phase PH2 (see the middle part of FIG. 4), and the inference process in the third phase PH3 (see the lower part of FIG. 4) may each be executed by separate devices.

[0041] In addition, in this application, generating a learning model (trained model) means manufacturing (producing) a learning model (trained model), and a "method of generating a learning model (trained model)" means a "method of producing a learning model (trained model)."

[0042] <1-3. Transport device 10 and imaging device 20> FIG. 2 is a schematic diagram of the conveying device 10 and the photographing device 20 as viewed from above, and FIG. 3 is a schematic diagram of the conveying device 10 and the photographing device 20 as viewed from the side.

[0043] The conveying device 10 includes a belt conveyor 11 and is capable of conveying an object (here, seaweed 91) in a longitudinal direction (left-right direction in Figs. 2 and 3). Fig. 2 and other figures show how seaweed 91 placed on the surface of the belt conveyor 11 is conveyed from the right side (upstream side) to the left side (downstream side).

[0044] An image capturing device 20 is provided above the conveying path of the belt conveyor 11. The image capturing device 20 includes a linear light source 21, a linear line sensor camera (imaging device) 22, and a seaweed detection sensor 23. A cover 24 is provided to cover the light source 21, the line sensor camera 22, and the seaweed detection sensor 23. The light source 21, the line sensor camera 22, the seaweed detection sensor 23, and the cover 24 are each fixed to the base of the belt conveyor 11 by fixing members (not shown). Note that, for convenience of illustration, in FIG. 2 and other figures, the cover 24 is shown removed, but in reality, the system operates with the cover 24 attached.

[0045] The light source 21 is a white light source. For example, a lighting device including a linear fluorescent lamp, or a lighting device including a plurality of linearly arranged LEDs and a diffusion cover is used as the light source 21.

[0046] The line sensor camera 22 is configured with a group of imaging elements (CCD linear sensors, etc.) arranged in a line. The line sensor camera 22 receives light (white light) emitted from the light source 21 at a predetermined irradiation angle (emission angle) that is reflected by the seaweed 91. Color pixel values ​​(RGB pixel values) at each of a plurality of linear positions are generated by the photoelectric conversion action of the line sensor camera 22 and signal processing by an A / D converter, etc., and image data of the linear region (elongated rectangular region with a very small width) is generated. By repeatedly capturing images of such a linear region (elongated rectangular region with a very small width), an image 200 (color image) of a two-dimensional region (planar region) is acquired.

[0047] The seaweed detection sensor 23 is disposed above the transport path of the belt conveyor 11, and detects whether or not seaweed 91 is present directly below it.

[0048] <1-4. Creating seaweed images> When the belt conveyor 11 is driven with a plurality of pieces of seaweed 91 (seaweed cut into approximately rectangular shapes) placed on the belt conveyor 11 at appropriate intervals in the direction of travel, the plurality of pieces of seaweed 91 advance sequentially toward the photographing device 20.

[0049] Then, when the laver detection sensor 23 detects that a certain piece of laver 91 has arrived directly below the laver detection sensor 23 by the belt conveyor 11 (or after a predetermined time T1 has elapsed), it starts taking an image of the laver 91. After that, the line sensor camera 22 repeatedly takes images of the linear area at intervals of a minute time ΔT, and ends taking the image after a predetermined time T2 (>T1) has elapsed. A single image (image of the laver 91) is generated with the laver 91 as the subject by sequentially pasting together a plurality of linear areas (elongated rectangular areas with a minute width) that have been continuously photographed. Specifically, the grade determination device (image processing device) 30 acquires image data (linear images) of a plurality of linear areas of a certain piece of laver 91 from the photographing device 20, and generates a single image 200 (an image (planar image) having a two-dimensional spread) based on the plurality of linear images. For example, an image of about 8000 pixels x 8000 pixels in size is acquired as the image 200. The small time ΔT may be appropriately determined based on the conveying speed, etc. The times T1 and T2 may be appropriately determined based on the desired size of the blank area in the image (particularly the area other than the seaweed in the conveying direction) and the desired size of the image (size in the conveying direction), etc.

[0050] Furthermore, the grade determination device 30 trims the image (planar image) 200, and extracts an area of ​​a predetermined size at a predetermined position in the image 200 (for example, an area including the center of the image and having a certain size (approximately a few tens of pixels out of all the pixels in the image)). This trimming results in an image (seaweed image) 210 of a predetermined size (for example, about 3000 pixels x 3000 pixels) that includes only the seaweed (not including the blank area surrounding the seaweed). At this point, the image 210 is an RGB image (a color image in which the color of each pixel is expressed in the RGB color space). In detail, the image 210 is a color image in which each pixel is expressed by three components (three elements) of an R pixel value, a G pixel value, and a B pixel value.

[0051] In this example, the image of the seaweed 91 is generated with trimming, but the present invention is not limited to this, and the image of the seaweed 91 may be generated without trimming.

[0052] By repeatedly executing the above-described process (automatic process), images 210 are generated for a plurality of (different) pieces of nori seaweed 91. In other words, a plurality of images (seaweed images) 210 relating to a plurality of pieces of nori seaweed 91 are generated.

[0053] For example, the above-described process generates each image 210 for learning, and each image 210 (212) to be inferred is generated. According to the above-described process, it is possible to automatically and continuously generate a plurality of seaweed images.

[0054] <1-5. Generation and processing of training data> FIG. 4 is a conceptual diagram showing an overview of the process, and FIG. 5 is a flow chart showing the process in the grade determination device 30. As shown in FIG.

[0055] As shown in FIG. 5, the grade determination device (information processing device) 30 executes a process of generating teacher data (step S1), a process of generating a learning model (learning process) (step S2), and an inference process using the learning model (step S3).

[0056] In the following, first, the process of generating teacher data (step S1: phase PH1) will be described.

[0057] 6 is a flow chart showing the detailed process in step S1. The process in step S1 will be described below with reference to FIG.

[0058] <Pretreatment> In step S11 (FIG. 6), first, the grade determination device (image processing device) 30 acquires a photographed image (image of the seaweed 91) 210. As described above, the image 210 is acquired as an RGB image (a color image in which the color of each pixel is expressed in the RGB color space). In detail, the image 210 is a color image (image data) in which each pixel is expressed by three elements (three components) of an R pixel value, a G pixel value, and a B pixel value.

[0059] Next, the rank determination device 30 converts the color space of the image 210 from the RGB color space to the YCbCr color space. In other words, the rank determination device 30 converts each pixel value of the image 210 from the RGB color space to the YCbCr color space.

[0060] Here, the YCbCr color space is a color space that expresses colors with three elements (three components) Y, Cb, and Cr, which are different from the three elements R, G, and B in the RGB color space. The three elements Y, Cb, and Cr are calculated by converting the pixel values ​​R, G, and B of the original RGB image using a predetermined conversion formula. The element Y is luminance, and the elements Cb and Cr are chrominance. In other words, the YCbCr color space is a color space that is composed of three orthogonal axes, one axis (Y axis) related to luminance and two axes related to chrominance. The two axes related to chrominance are the Cb axis and the Cr axis, which are orthogonal axes. The CbCr plane is composed of the remaining two elements (two components) Cb and Cr, excluding the Y element (Y component). Hue (and saturation) is expressed by a combination of the two elements Cb and Cr.

[0061] Then, the grade determination device 30 calculates the average value of a plurality of pixels in the image 210 (for example, all pixels in the image 210) for each of the three elements (three components) Y, Cb, and Cr. More specifically, three values ​​are calculated: the average value Yave of the element Y for a plurality of pixels in the image 210, the average value Cbave of the element Cb for the plurality of pixels, and the average value Crave of the element Cr for the plurality of pixels. In other words, the average values ​​Yave, Cbave, and Crave are calculated by averaging the values ​​of a plurality of pixels in the image 210 for each channel (each component) in the (converted) YCbCr color space.

[0062] These three values ​​Yave, Cbave, and Crave are used as data representative of image 210 (and, by extension, the seaweed that is the subject of image 210). For convenience of explanation, the average values ​​Yave, Cbave, and Crave are also referred to simply as the average values ​​Y, Cb, and Cr (without the subscript "ave") in the following description. Also, in FIG. 4 and other figures, the lines (overlines) placed above the letters Y, Cb, and Cr indicate that they are average values.

[0063] By such processing, the average values ​​Y, Cb, and Cr for one image 210 are obtained.

[0064] By repeating such processing, the grade determination device 30 obtains the average values ​​Y, Cb, and Cr for each of the (learning) multiple images 210. Note that each of the average values ​​Y, Cb, and Cr is also expressed as a pixel value (representative pixel value) that represents each of the multiple images 210.

[0065] <Principal component analysis> In the next step S12 (FIG. 6), the class determination device 30 performs principal component analysis in a predetermined color space for the multiple images. Specifically, the class determination device 30 performs principal component analysis on each average value (multiple sets of average values ​​(Y, Cb, Cr)) obtained for the multiple images 210. In other words, a principal component analysis (principal component analysis in the YCbCr color space) is performed on three elements (average values ​​Y, Cb, Cr) for the multiple images 210 (see FIG. 8). The principal component analysis is also expressed as a principal component analysis on three components (axis elements) Y, Cb, Cr that constitute the YCbCr color space. The principal component analysis is performed on three elements (three variables) Y, Cb, Cr that express the representative pixels (representative pixel values) of each of the multiple images in a predetermined color space.

[0066] Through this principal component analysis, the first principal component and the second principal component are calculated (obtained).

[0067] When an experiment (principal component analysis on a plurality of seaweed images) was actually performed, one experimental result was that the first principal component (first principal component vector V1) was calculated as V1=(Y,Cb,Cr)=(1,0,0), and the second principal component (second principal component vector V2) was calculated as V2=V21=(Y,Cb,Cr)=(0,-0.6,+0,8) (see Figs. 8 and 10). Note that the value "0" here is, strictly speaking, a value that is not "completely zero" (a value that includes an error). Other values ​​(such as "1" and "0.8") also include an error. Fig. 8 is a conceptual diagram showing an overview of the principal component analysis in step S12.

[0068] In the principal component analysis, the directions of the principal component vectors V1 and V2 may be inverted when calculated. For example, the first principal component vector V1 may be calculated as (-1, 0, 0). The second principal component vector V2 may be calculated as V2=V22=(Y, Cb, Cr)=(0, +0.6, -0.8). In the principal component analysis, principal component vectors (especially the second principal component vector V2) having different directions may be calculated depending on the data to be analyzed, etc. For example, the second principal component vector V2 may be calculated as V2=V23=(Y, Cb, Cr)=(0, -0.8, +0, 6), or V2=V24=(Y, Cb, Cr)=(0, +0.8, -0, 6), etc.

[0069] Fig. 9 is a diagram showing the significance of the first principal component vector V1=(1,0,0). As shown in Fig. 9, it can be seen that the luminance information E1 (information representing the luminance Y in the YCbCr color space) is the most important element from this first principal component. In addition, as the first principal component score P1 for this first principal component vector V1=(1,0,0) increases, the luminance Y gradually increases. Note that the luminance Y is normalized, so that Y=0.0 when the minimum luminance is 0 and Y=1.0 when the maximum luminance is 255.

[0070] FIG. 10 is a diagram showing the significance of the second principal component vector V2 (specifically, the second principal component vector V21)=(0,-0.6,+0,8). In FIG. 10, for convenience of illustration, the color characteristics of each region in the CbCr plane are expressed by characters (such as "red", "blue", and "green"). Red (including its neighboring colors) is distributed in the upper left region (second quadrant) (mainly the region within the circle (dashed line) in the second quadrant, etc.) of the CbCr plane (Y=0.5) in FIG. 10. Furthermore, blue (including its neighboring colors) is distributed in the lower right region (fourth quadrant) (mainly the region within the circle (dashed line) in the fourth quadrant, etc.). Furthermore, purple (including its neighboring colors) is distributed in the upper right region (first quadrant) of the CbCr plane, and green (including its neighboring colors) is distributed in the lower left region (third quadrant). The Cb and Cr values ​​are also normalized, and the original values ​​from "0" to "255" are converted to values ​​from "-1.0" to "1.0."

[0071] From this second principal component, as shown in Fig. 10, it can be seen that color information E2 (color information related to the direction from red to blue or the opposite direction in the YCbCr color space) related to the direction D1 connecting (connecting) red and blue in the YCbCr color space is the second most important element after the luminance information E1. The color information E2 is information indicating color characteristics in the direction D1 (color characteristics appearing in the direction D1), and more specifically, information indicating the intensity (and positive or negative) of the color characteristics in the direction D1. The color information E2 can also be expressed as the intensity of the D1 direction component of the data vector V (not shown) (such as the magnitude of the inner product of the data vector V and the second principal component vector V2). Here, the data vector V is a three-dimensional vector composed of average values ​​(Y, Cb, Cr) in the YCbCr color space of the image 210.

[0072] Here, the direction D1 to be considered in the color information E2 does not mean only one direction (the direction of the vector V21) shown in FIG. 10, but may be the direction of the vector V23 (, V24) as described above. The direction D1 may be a direction between the direction of the vector V21 and the direction of the vector V23. The direction D1 may be a direction closer to the Cr axis than the direction of the vector V21, or may be a direction closer to the Cb axis than the direction of the vector V23. The direction D1 may be a direction shifted by several tens of degrees (for example, 30 degrees to 40 degrees) from the vector V25=(Y, Cb, Cr)=(0,-1,1) (in the CbCr plane). The direction of the second principal component vector V2 has a certain range (angle range) depending on the data to be analyzed, and the direction D1 also has a certain range. The direction D1 may be a direction connecting red and blue in the YCbCr color space. In other words, the color information E2 is information about such a direction D1 (the direction connecting red and blue in the YCbCr color space).

[0073] The color information E2 is also expressed as color information relating to the direction from the second quadrant to the fourth quadrant or the reverse direction of the CbCr plane (a plane consisting of the two remaining components Cb and Cr excluding the Y component) in the YCbCr color space.

[0074] Color information E2 can also be expressed as information indicating whether a color leans toward the red side or the blue side and the degree of leaning (the degree of leaning in the "red-blue" direction D1). Color information E2 can also be expressed as color information (color index value) indicating whether a color has a reddish or blueish tinge and the degree of leaning (the degree of "reddish or blueish" tinge). Simply put, color information E2 is information indicating color characteristics in the "red-blue" direction D1 in color space.

[0075] As described above, in principal component analysis, each principal component (especially the second principal component) may be calculated with the sign reversed (i.e., the direction of the principal component vector is reversed). However, in principal component analysis, the "direction" of the principal component vector (a concept that includes a certain direction and its opposite direction) is particularly important. In principal component analysis, the "direction" of the principal component vector is more important than the importance of "whether the principal component vector has a certain direction or its opposite direction (the positive and negative of the principal component score)." For example, the principal component vector V21=(0,-0.6,+0,8) and the principal component vector V22=(0,+0.6,-0,8) have the same "direction" and have the same meaning, although the positive and negative principal component scores for each vector are opposite to each other.

[0076] The principal component score P2 (P22) for the principal component vector V22 = (0, +0.6, -0, 8) becomes relatively more positive as the blue component becomes stronger and the red component becomes weaker (as the color approaches blue). Conversely, the principal component score P22 becomes relatively more negative as the red component becomes stronger and the blue component becomes weaker (as the color approaches red). In other words, this means that as the second principal component score P22 based on the second principal component vector V22 becomes larger (to the positive side), the color approaches blue.

[0077] On the other hand, the principal component score P2 (P21) for the principal component vector V21 = (0, -0.6, +0, 8) becomes a relatively positive value as the red component becomes stronger and the blue component becomes weaker (as the color approaches red). Conversely, the principal component score P21 becomes a relatively negative value as the blue component becomes stronger and the red component becomes weaker (as the color approaches blue). In other words, this means that as the second principal component score P21 based on the second principal component vector V21 becomes larger (to the positive side), the color approaches red.

[0078] However, these two second principal component vectors V21 and V22 have in common the fact that they have the same "direction" (though their orientations are opposite to each other) and indicate that the color information E2 related to the "red-blue" direction D1 is important. This point is very important.

[0079] Through such principal component analysis, it was found that the information of the first principal component is the luminance information E1, and the information of the second principal component is the above-mentioned color information E2. In other words, it was found that the luminance information E1 and the color information E2 relating to the red-blue direction are the two main elements of the image of the seaweed.

[0080] Furthermore, the principal component analysis aggregates the three elements in the YCbCr color space of the seaweed image 210 into two elements (specifically, two principal component scores). In other words, it is possible to obtain the effect of compressing the number of dimensions from "3" to "2".

[0081] In particular, in the above experimental results (principal component analysis results for the YCbCr color space of the nori image), the cumulative contribution rate up to the second principal component (the sum of the contribution rate of the first principal component and the contribution rate of the second principal component) was 99.99%. Such a cumulative contribution rate indicates that the information (color characteristics, etc.) specific to the nori in the nori image can be compressed in dimension very efficiently. In particular, the direction of the second principal component vector V2 (direction D1 connecting red and blue) (see FIG. 10) at this time is the direction in which data is widely distributed (scattered) (next to the direction of the first principal component vector V1), and is a very useful direction for evaluating the color of the nori. In other words, data is not very distributed (scattered) in the direction perpendicular to both the first principal component vector V1 and the second principal component vector V2 (direction connecting green and purple in the YCbCr color space (see FIG. 10)), and there is little need to consider information in that direction. By utilizing such characteristics, it is possible to appropriately compress the dimension of the information specific to the nori in the nori image.

[0082] FIG. 11 is a diagram in which a combination of the first principal component score P1 and the second principal component score P2 of the average value (Y, Cb, Cr) of each image for a plurality of (here, 120) images 210 is plotted on a graph (principal component space). In the graph, the horizontal axis indicates the first principal component score P1, and the vertical axis indicates the second principal component score P2. Note that the first principal component score P1 of a data vector V(Y, Cb, Cr) is the inner product of the data vector V and the first principal component vector V1 (the magnitude and positivity of the projected vector obtained by projecting the vector V onto the vector V1 axis). Similarly, the second principal component score P2 of a data vector V(Y, Cb, Cr) is the inner product of the data vector V and the second principal component vector.

[0083] However, in FIG. 11 and the like, the principal component scores Pi (i=1, 2) are shown as values ​​obtained by shifting the position of zero using the average value (average value for each component) Fi of the principal component scores of multiple data (values ​​obtained by subtracting the average value Fi from the original calculated value (original value Pi) for each component) (in simple terms, centered values). Therefore, the position of zero (for each principal component) on the graph is shifted to the position of the average value Fi, and the value of each principal component score Pi is converted to a value different from the original calculated value. The principal component scores Pi may be calculated by converting to such values ​​(centered values). Note that this is not limited to this, and the value of each principal component score Pi may be calculated as a value standardized using a standard deviation or the like (a value obtained by further dividing the value obtained by subtracting the average value Fi from the original calculated value for each component by the standard deviation, etc.).

[0084] 11, points on the relatively right side (positive side) of the horizontal axis indicate that the first principal component score P1 (and therefore the luminance Y) is relatively large. Conversely, points on the relatively left side (negative side) of the horizontal axis indicate that the first principal component score P1 (and therefore the luminance Y) is relatively small.

[0085] 11, the magnitude of the second principal component score P2 (P12) for the second principal component vector V21=(0,-0.6,+0,8) is shown along the vertical axis. Specifically, points relatively higher (+ side) along the vertical axis indicate that the second principal component score P2 is relatively large (and thus the redness (red component) is relatively strong). Conversely, points relatively lower (- side) along the vertical axis indicate that the second principal component score P2 is relatively small (and thus the blueness (blue component) is relatively strong).

[0086] In addition, in Fig. 11, each image 210 is plotted with a dot of a different shape for each grade (correct answer data) of the subject seaweed. In Fig. 11, the result of the inspector's judgment as to which grade each seaweed belongs (more specifically, the correct answer data (also called the correct answer label)) is shown as the grade of each seaweed.

[0087] As shown in FIG. 12, there are multiple classes (grades) of seaweed according to its "color," such as "Excellent," "Superior," "Special," "First Class," and "Second Class." "Excellent" is the highest grade, and "Excellent" seaweed has the best color (as seaweed color). As the grade decreases from "Superior," "Special," "First Class," "Second Class," and so on, the quality of the color of the seaweed decreases. Here, "Excellent" is represented by star-shaped dots, "Superior" by circular or diamond-shaped dots, "Special" by a specific shape (described below) or square-shaped dots, "First Class" by cross-shaped dots, and "Second Class" by cross-shaped or triangular dots (see also FIG. 12). The specific shape is a shape that combines a cross and an x ​​(a shape composed of line segments extending in eight directions from the center). FIG. 12 shows the dot shapes (dot types) on the graph for each grade and the correct score (correct label).

[0088] Here, seaweed is graded by "color" and also classified from other standpoints (such as lack of shine, weight outside the regulations, etc.). Specifically, seaweed is divided into several classes, such as "genuine grade", "B grade", ... In short, "genuine grade" seaweed is seaweed with good quality in all elements except for "color". On the other hand, "B grade" seaweed is seaweed whose surface appears to be "lose shine" due to cell rupture, etc.

[0089] Generally, seaweed is graded by combining a grade based on color with a class based on another aspect, for example, "B Special." Here, "B Special" indicates that the grade based on color is "Special" and the class based on another aspect (also called the sub-grade) is "B." When the grade based on color is "Special" and the class is "Main Grade," it may (originally) be expressed as "Main Special" (a combination of "Special" and the "Main" from "Main Grade"), but (in practice) it is also expressed simply as "Special" with the "Main" (Main Grade) omitted.

[0090] In Figures 11, 13, 14, etc., except Figure 12, the simple "Special" indicates "genuine grade" and "Special" seaweed. Similarly, "Special Grade", "Special", "1st Grade", and "2nd Grade" are shown with the "Genuine" indicating "genuine grade" omitted. On the other hand, "B Grade" is not omitted (with "B" added) and is shown as "B Special Grade", "B Special Grade", and "B2". Note that "Special Grade" is shown with circular dots, and "B Special Grade" is shown with diamond-shaped dots. Also, "Special" is shown with dots of the specific shape mentioned above, "B Special Grade" with square dots, "2nd Grade" with X dots, and "B2 Grade" with triangular dots (see also Figure 12).

[0091] In addition, in this application, a rating (grade) that combines a grade based on color and a class based on another viewpoint, such as "B Special," is also referred to as an "overall grade."

[0092] In the present invention, the main focus is not on distinguishing between "Grade B" and "Full Grade," but on grading by "color." As will be described in detail later, basically, only the grades related to color (such as "Excellent," "Superior," "Special," "First Grade," and "Second Grade") (see FIG. 12) are determined as the "grade of the seaweed." However, this is not limited to this, and for example, a grade (overall grade) based on information related to classification from another perspective (such as weight information) may also be determined as the "grade of the seaweed."

[0093] <Learning model> In this embodiment, a model is used as the learning model 400, which uses the first principal component score P1 and the second principal component score P2 related to the mean value vector V(Y,Cb,Cr) of each image (input image) as input and the score Q related to the grade of seaweed as output. In detail, a multiple regression analysis is performed using the first principal component score P1 indicating the luminance information E1 and the second principal component score P2 indicating the color information E2 as two explanatory variables x1, x2, and the grade of seaweed as the objective variable y. More specifically, a generalized linear regression model (interaction linear regression model) that takes into account the interaction between the two explanatory variables x1, x2 is used as the learning model 400. In other words, an interaction linear analysis (regression analysis) is performed that also takes into account the interaction term (the product term of both variables) of the explanatory variables x1, x2 (here, P1, P2). For example, a regression analysis process is performed to predict the objective variable y (grade of seaweed) expressed by the following formula (1). The objective variable y is y=b1*x1+b2*x2+c*(x1*x2)+d ···(1) The values ​​b1, b2, c, and d are parameters to be learned by the learning model 400. These parameters b1, b2, c, and d are optimized by a learning process to be described later.

[0094] <Teacher data generation> In the next step S13 (FIG. 6), the grade determination device 30 generates training data for training the learning model 400 as described above.

[0095] Specifically, the grade determination device 30 obtains a first principal component score P1 and a second principal component score P2 for each image 210 with respect to the mean value vector V(Y,Cb,Cr) of the image 210. Specifically, the first principal component score P1 and the second principal component score P2 for each image 210 are calculated based on the first and second principal components obtained in step S12 and the mean value vector V(Y,Cb,Cr) obtained in step S11 for each image 210 (each of the multiple learning images). In detail, the first principal component score P1 is calculated as the inner product of the mean value vector V and the first principal component vector V1, and the second principal component score P2 is calculated as the inner product of the mean value vector V and the second principal component vector V2. Note that each principal component score P1, P2 may be a value that has been centered or standardized as described above.

[0096] The grade determination device 30 also obtains a correct label for the grade (main grade) of the seaweed that is the subject of each image 210. As the correct label, a value (a result of the inspection by an inspector) previously assigned to each image may be used. As the correct label for the seaweed grade (score Q), for example, a numerical value as shown in FIG. 12 may be used. Specifically, "-2" may be assigned to "Excellent", "-1" to "Superior", "0" to "Special", "1" to "First Class", and "2" to "Second Class".

[0097] Then, this set of data (the first principal component score P1, the second principal component score P2, and the correct label of the seaweed grade (score Q)) is generated as a (set of) training data.

[0098] Furthermore, such teacher data is generated for a plurality of (for example, one hundred to several thousand) images (seaweed images) 210, thereby generating a plurality of teacher data (teacher data groups).

[0099] An example of the teacher data group generated in this manner is the data group plotted in FIG.

[0100] <1-6. Learning process> Next, a learning process using the teacher data (step S2: phase PH2) (FIG. 5) is executed.

[0101] Specifically, the grade determination device 30 uses the multiple training data generated in step S1 to machine-train the learning model 400 (interaction linear regression model). This learning process optimizes the values ​​b1, b2, c, and d in the above formula (1), and generates the trained model 400 (420). In other words, the trained model 420 is a model that has learned the relationship between the first principal component score P1 and the second principal component score P2 of each image 210 and the grade of the seaweed in each image 210 using multiple training data.

[0102] <1-7. Inference processing> Next, the inference process using the teacher data (step S3: phase PH3) (FIG. 5) will be described. FIG. 7 is a flowchart showing the detailed process in step S3. The process in step S3 will be described below with reference to FIG.

[0103] In step S31, the same process as in step S11 is performed on the inference target image 210 (also referred to as 212). Specifically, the grade determination device 30 converts the inference target image 212 (specifically, its color space) from the RGB color space to the YCbCr color space, and calculates the average values ​​Yave, Cbave, and Crave (within the inference target image 212) of the three elements Y, Cb, and Cr.

[0104] In step S32, the grade determination device 30 obtains (calculates) the first principal component score P1 and the second principal component score P2 in the YCbCr color space of the inference target image 212. In this step S32 (inference stage), the principal component analysis result (first principal component (first principal component vector V1) and second principal component (second principal component vector V2)) obtained in the learning stage (step S12 (FIG. 6)) is used. Specifically, the first principal component score P1 is obtained as the inner product of the average value vector V (Yave, Cbave, Crave) of the inference target image 212 and the first principal component vector V1. Similarly, the second principal component score P2 is obtained as the inner product of the average value vector V (Yave, Cbave, Crave) of the inference target image 212 and the second principal component vector V2. Furthermore, the principal component scores Pi (i=1,2) may be values ​​that have been subjected to the same transformation (centering or standardization, etc.) as in the processing up to the learning stage (specifically, when generating the teacher data). Here, the principal component scores Pi are calculated as values ​​obtained by shifting the position of zero using the average value Fi (for each component) regarding a plurality of pieces of learning data (centering values ​​obtained by subtracting the average value Fi from the original calculated value for each component).

[0105] In step S33, the grade determination device 30 obtains a score Q (evaluation value related to the grade of the seaweed). The score Q is a score (evaluation value) for determining (estimating) the grade of the seaweed. Specifically, the grade determination device 30 inputs the luminance information E1 (first principal component score P1) and color information E2 (second principal component score P2) of the inference target image 212 into the learning model 400 (trained model 420) and obtains the score Q output (calculated) from the learning model 400 (420). The score Q is an output value from the learning model 420 trained based on multiple teacher data (composed of the luminance information E1 and color information E2 of each image and the correct data of the grade (correct grade information)). In other words, the score Q is an evaluation value obtained based on the analysis result of the interaction linear analysis using the luminance information E1 and color information E2.

[0106] Then, the grade determination device 30 estimates (determines) the grade of the seaweed in the inference target image based on the score Q. For example, the grade determination device 30 determines the grade of the seaweed to be the value obtained by rounding off the decimal point of the score Q (see FIG. 12). Seaweed with a score Q="1.2" is determined to be "first grade", seaweed with a score Q="-0.3" is determined to be "special" (a grade corresponding to "0"), and seaweed with a score Q="-0.8" is determined to be "excellent" (a grade corresponding to "-1").

[0107] 13 and 14 are diagrams showing the estimation results regarding the score Q.

[0108] Fig. 13 is a diagram showing the learning result (regression surface) of the learning model 400 (trained regression model) 400 superimposed on the graph of Fig. 11. For convenience of illustration, however, the curves U1 to U6 are drawn as the degree lines of the regression surface. The curves U1 to U6 respectively show the relationship between the explanatory variables x1 and x2 (here, P1 and P2) obtained by substituting each value from -2.5 to 2.5 in increments of 1 (six values ​​(-2.5, -1.5, -0.5, +0.5, +1.5, +2.5)) into y (=Q) in the above formula (1).

[0109] In Fig. 13, for example, a curve U2 is drawn which shows the relationship between explanatory variables x1 and x2 when y = -1.5. Also, a curve U3 is drawn which shows the relationship between explanatory variables x1 and x2 when y = -0.5. Furthermore, a curve U4 is drawn which shows the relationship between explanatory variables x1 and x2 when y = +0.5, and a curve U5 is drawn which shows the relationship between explanatory variables x1 and x2 when y = +1.5.

[0110] 13, when a point at coordinates (P1, P2) corresponding to the inference target image 212 exists between the curve U2 and the curve U3, the score Q of the point is estimated as a value between -1.5 and -0.5 (for example, -1.2 or -0.9). In this case, the grade determination device 30 estimates (determines) that the grade of the seaweed in the inference target image is "extra high quality" (corresponding to a score of "-1") (see FIG. 12) based on the score Q of the point (for example, -1.2).

[0111] In FIG. 13, as in FIG. 11, the “correct grade label” (correct grade information) of each point (corresponding to each image) is indicated by the shape of the point.

[0112] For example, in Fig. 13, many of the points with shapes (circles or diamonds) indicating the "excellent" grade (i.e., points with the correct label of grade = "excellent") are between the curves U2 and U3 (the grade is estimated to be "excellent"). This shows that the inference result (estimated result) by the grade determination device 30 is correct to a certain extent.

[0113] 13, a point with a shape (cross shape) indicating the "First Class" grade (i.e., a point with a correct label of grade = "First Class") is between curve U4 (Q = 0.5) and curve U5 (Q = 1.5) (the grade is estimated to be "First Class"). This indicates that the inference result (estimation result) by the grade determination device 30 is correct to a certain extent.

[0114] Fig. 14 is a diagram showing the estimated score Q of each sample. The graph in Fig. 14 shows the score Q (inference result) output when the principal component scores P1 and P2 of each image 210 (sample image) are input to the learning model 400 (trained model 420).

[0115] In Fig. 14, the score Q of each point of the shape (circle or diamond) indicating the "excellent" grade (i.e., the point having the correct label of grade = "excellent") generally has a value in the range of -1.5 to -0.5 (with some exceptions). This indicates that the inference result by the grade determination device 30 is correct to a certain extent.

[0116] 14, the score Q of each point of the shape (cross shape) indicating the "First Class" grade (i.e., the point having the correct label of grade = "First Class") generally has a value in the range of +0.5 to +1.5. This indicates that the inference result by the grade determination device 30 is correct to a certain extent.

[0117] <1-8. Effects of the embodiment> According to the above process, the grade determination device 30 calculates a score Q (a score for determining the grade of the seaweed) based on the luminance information E1 (more specifically, the first principal component score P1) of the image 210 of the seaweed to be graded and the color information E2 (more specifically, the second principal component score P2) indicating the color characteristics in the "red-blue" direction of the image 210. By using the score Q taking into account the color information E2, it is possible to appropriately determine the grade of the seaweed while reducing the number of workers. As a result, it is possible to automatically determine the grade of the seaweed more appropriately (in other words, to realize a more appropriate grading (more specifically, a grading closer to human senses)).

[0118] Moreover, the grade determination device 30 trains the learning model 400 using a plurality of teacher data. The learning model 400 is a model that receives as input the luminance information E1 (specifically, the first principal component score P1) of the image (learning input image) 210 of the nori seaweed and the color information E2 (specifically, the second principal component score P2) relating to the "red-blue" direction of the image 210, and outputs a score relating to the grade of the nori seaweed. Each teacher data is configured to include ground truth data relating to the grade of the nori seaweed. By generating such a model by machine learning, it is possible to realize appropriate grading based on the score Q output from the model.

[0119] In particular, two elements (specifically, luminance information E1 and color information E2) that are an aggregate of the original three elements (specifically, R value, G value, and B value in the RGB color space) of the pixel value information of the image 210 are used as input. This makes it possible to efficiently learn a model that estimates the score Q by dimensionality compression. Furthermore, by using luminance Y (luminance information) as one of the two elements and color information E2 relating to the "red-blue" direction D1 as the other element, it is possible to generate a model that accurately reflects the unique characteristics (particularly color characteristics) of the seaweed in the seaweed image. Ultimately, it is possible to more appropriately determine the grade of the seaweed based on the score Q.

[0120] In addition, in the above embodiment, the interaction between the two principal component scores P1, P2 (in other words, the luminance information E1 and the color information E2) is taken into consideration. Therefore, a more appropriate regression process than a simple linear regression process (which does not take the interaction into consideration) is realized, and an appropriate learning model 400 is generated. Therefore, it is possible to perform a more appropriate estimation process. For example, the magnitude line shown in FIG. 13 is a curve, not a straight line. More specifically, it is a curve in which the distance from the adjacent curve increases as the luminance increases. Therefore, an estimation process is realized according to the characteristic that the higher the luminance, the more likely it is that color differences will appear.

[0121] In addition, by using the above-mentioned conveying device 10 and photographing device 20 (see FIG. 2 and FIG. 3), it is possible to automatically and continuously generate the image 212 of the inference object (and the image 210 for learning). In addition, it is possible to automatically and continuously perform the grade judgment of a plurality of seaweeds 91.

[0122] <2. Second embodiment> The second embodiment is a modification of the first embodiment. The following description will focus on the differences from the first embodiment.

[0123] In the second embodiment, a score (specifically, an improvement score) is calculated based on an index value related to the degree of focus of the image, and the grade of the seaweed is determined based on the score.

[0124] Here, the seaweed has a “wave.” Therefore, the distance from the line sensor camera 22 to the subject (seaweed) varies, and the multiple seaweed images 210 have “variations” in the degree of focus.

[0125] The inventors of the present application have found through consideration of the above experimental results that the score Q is affected by such variations in the degree of focus (more specifically, an image with a good degree of focus has a high score Q).

[0126] For example, as shown in FIG. 14, many of the scores Q of the image 210 (see image 210d in FIG. 15, etc.) of the "second grade" (specifically, "main grade" and "second grade" (see cross marks)) of the seaweed are concentrated in the vicinity of "2" to "2.5" (see cross marks in FIG. 14 near Q=2). On the other hand, the image 210c in FIG. 15 is also an image of the "second grade" of the seaweed. Nevertheless, the score Q of the image 210c is calculated to be better (a value closer to "1") than the scores of the other images 210d (see cross marks in FIG. 14 indicating the image 210c having a score Q close to "1"). Therefore, there is a risk that the wrong grade ("first grade") will be estimated for the image 210c, which is actually "second grade". Note that the image 210c can also be expressed as an image having a score Q (a value closer to "1") that is far from the actual value (too good).

[0127] Fig. 15 is a diagram comparing these two images 210c and 210d. In Fig. 15, information about image 210d is shown on the right side of each row, and information about image 210c (an image having a better score Q (a value closer to "1") than image 210d) is shown on the left side of each row. In the top row of Fig. 15, both images 210c and 210d are shown, the middle row shows enlarged images of parts of both images 210c and 210d, and the bottom row shows power spectrum images of both images 210c and 210d.

[0128] As can be seen by comparing the enlarged views of the images 210c and 210d in the middle of Figure 15, the image 210c on the left has a higher degree of focus than the image 210d on the right. It is presumed that image 210c, which has a relatively high degree of focus, is more likely to have color differences (as a result), resulting in a relatively high score Q. Note that other images 210 with scores Q in the vicinity of "2" to "2.5" also have a similar degree of focus (relatively low degree of focus) to image 210d.

[0129] Therefore, in the second embodiment, the grade determination device 30 determines the grade of the seaweed based on the index value related to the focus degree of each image. Specifically, the grade determination device 30 uses another learning model 450 (see FIG. 16) in addition to the learning model 400 described in the first embodiment to obtain an improved score (also called an improved score) Q2, and determines the grade of the seaweed based on the improved score Q2. That is, the grade of the seaweed is determined using a two-stage learning model. The learning model 400 is also called a first learning model, and the learning model 450 is also called a second learning model.

[0130] 16 is a diagram showing a learning model 500 (two-stage learning model) according to the second embodiment. The learning model 500 has a configuration in which another learning model 450 is further connected to the rear stage of the learning model 400 similar to that of the first embodiment.

[0131] Learning model 450 is a model that takes as input the score Q (also referred to as Q1) output from learning model 400 for input image 210 and an index value P5 related to the degree of focus of the input image 210, and outputs an improved score Q2 related to the grade of the seaweed.

[0132] Here, as the index value P5 relating to the degree of focus of the image 210, the ratio between the low-frequency component and the high-frequency component in the power spectrum image (described next) is used.

[0133] A power spectrum image (see the bottom row of FIG. 15) is an image in which spatial frequency components obtained by a two-dimensional Fourier transform of image 210 are expressed as a grayscale image (brightness image). Parts closer to the center correspond to low frequency components, and parts further away from the center correspond to higher frequency components. The pixel value at each position is expressed in a brighter color the greater the component value (power spectrum) of the frequency component at that position. Note that, in generating a power spectrum image, an image having a pixel number that is a power of 2 for performing two-dimensional Fourier analysis, specifically, an image of a portion of image 210 (for example, 2048 pixels x 2048 pixels near the center) may be used.

[0134] Natural images (including seaweed images) with a relatively high degree of focus contain many high-frequency components, and the frequency components are distributed over a wide range (frequency region) up to a relatively high frequency. Specifically, as shown in the bottom left of FIG. 15, the power spectrum image of image 210c with a high degree of focus contains relatively many frequency components higher than a certain frequency (shown by a circle in the bottom of FIG. 15). In detail, bright pixels in the power spectrum image are distributed even outside the circle (i.e., high-frequency components are distributed over a wide range). Conversely, in the power spectrum image of image 210d with a low degree of focus (the image in the bottom right of FIG. 15), bright pixels are relatively concentrated in a frequency region lower than the certain frequency. In detail, bright pixels are relatively concentrated in the inner part of the circle.

[0135] Therefore, in the second embodiment, a value indicating the magnitude relationship between high-frequency components and low-frequency components is adopted as the index value P5 indicating the degree of focus. The index value P5 is, for example, a value obtained by dividing the amount of high-frequency components (amount of high-frequency components) by the amount of low-frequency components (amount of low-frequency components) (P5=amount of high-frequency components / amount of low-frequency components). In detail, a value obtained by dividing the integrated value (sum) of the power spectrum in the high-frequency region (region of frequencies higher than a certain frequency) by the integrated value of the power spectrum in the low-frequency region (region of frequencies lower than the certain frequency) (the ratio of both integrated values) is used as the index value P5. Alternatively, a value obtained by dividing the integrated value of the amplitude spectrum (square root of the power spectrum) in the high-frequency region by the integrated value of the amplitude spectrum in the low-frequency region may be used as the index value P5. Here, the latter value (a value obtained by dividing the integrated value of the amplitude spectrum in the high-frequency region by the integrated value of the amplitude spectrum in the low-frequency region) is used as the index value P5.

[0136] In the second embodiment, in the learning stage of the learning model, after the learning model 400 is learned in the same manner as in the first embodiment, the learning model 450 is further trained. Specifically, training data (training data for the learning model 450) consisting of a score Q (Q1) output from the learning model 400 (420) for each image 210, an index value P5 related to the focus degree of each image 210, and a correct answer label related to the grade of seaweed is used. Machine learning is performed on the learning model 450 using multiple training data.

[0137] Specifically, the grade determination device 30 performs multiple regression analysis using the score Q1 and the index value P5 as two explanatory variables and the score Q2 (also referred to as an improved score) relating to the grade of the seaweed as a response variable. More specifically, the grade determination device 30 uses a generalized linear regression model (interaction linear regression model) that takes into account the interaction between the two explanatory variables as the learning model 450. This machine learning (regression analysis) optimizes the learning target parameters of the learning model 450, and a trained learning model 450 (also referred to as a trained model 470) is generated.

[0138] Then, in the inference stage, the grade determination device 30 inputs the score Q1 output from the learning model 400 (420) for the image 212 to be inferred and the index value P5 related to the degree of focus of the image 212 to the learning model 450 (470). The grade determination device 30 obtains the refined score Q2 output (calculated) from the learning model 450 (trained model 470). Then, the grade determination device 30 determines the grade of the seaweed based on the refined score Q2.

[0139] 17 is a graph plotting combinations of the score Q1 and the index value P5 related to the degree of focus for each image 210 for multiple (here, 120) images. In the graph, the horizontal axis represents the score Q1 output from the learning model 400 for each image 210, and the vertical axis represents the index value P5 for each image 210.

[0140] In FIG. 17, it can be seen that, among a number of "second place" images (see the x marks), the index value P5 of image 210c stands out (having a value close to "5.5").

[0141] 18 and 19 are diagrams showing the estimation results regarding the score Q2.

[0142] Fig. 18 is a diagram showing the learning results (regression surface) of the learning model 450 (trained regression model) superimposed on the graph of Fig. 17. For convenience of illustration, however, curves U21 to U26 are drawn as degree lines of the regression surface. Curves U21 to U26 respectively show the relationship between two explanatory variables Q1 and P5 obtained by substituting each value in 1 increments from -2.5 to 2.5 into the objective variable.

[0143] As shown in Fig. 18, many of the points with shapes (circles or diamonds) indicating the "excellent" grade (i.e., points with the correct label of grade = "excellent") are between the curve U22 (Q2 = -1.5) and the curve U23 (Q2 = -0.5) (the grade is estimated to be "excellent"). This shows that the inference result by the grade determination device 30 is correct to a certain extent.

[0144] 18, among the points of the shape (marked with an x) indicating the "Second Class" grade (specifically, "Second Class" and "Regular Grade"), the point corresponding to the image 210c is present between the curve U25 and the curve U26 (the grade is estimated to be "Second Class"). This indicates that the inference result by the grade determination device 30 is correct to a certain extent.

[0145] Fig. 19 is a diagram showing the refined score Q2 for each sample. The graph in Fig. 19 shows the refined score Q2 (inference result) output when the score Q1 and the index value P5 for each image (sample image) are input to the learning model 450 (trained model 470).

[0146] In Fig. 19, the score Q of each point of the shape (X mark or triangle) indicating the "Second Grade" grade (i.e., the point having the correct label of grade = "Second Grade") generally has a value in the range of +1.5 to +2.5 (with some exceptions). This indicates that the inference result by the grade determination device 30 is correct to a certain extent.

[0147] In particular, as can be seen by comparing Figure 14 and Figure 19, image 210c is plotted at a position close to "1" (Q=Q1=1) in Figure 14, whereas it is plotted at a position close to "2" (Q2=2) in Figure 19. In other words, it can be seen that the estimation result (output score) by the learning model has been improved.

[0148] In this way, by judging the grade of the seaweed based on the index value related to the degree of focus of the image, it is possible to improve the accuracy of the score Q (specifically, Q2), and thus it is possible to more appropriately judge the grade of the seaweed.

[0149] In the second embodiment, the ratio of low-frequency components to high-frequency components in the power spectrum image is used as the index value P5 indicating the degree of focus, but the present invention is not limited to this. For example, other index values ​​such as an edge amount may be used as the index value P5 indicating the degree of focus.

[0150] <3. Modifications, etc.> Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

[0151] <Color space and color information> For example, in each of the above-mentioned embodiments, the principal component analysis is performed in the YCbCr space to obtain two principal components (two elements), the first principal component and the second principal component. However, the present invention is not limited to this, and the principal component analysis may be performed in another color space (for example, the YCbCr color space, the YPbPr color space, the YUV color space, the Lab color space, the RGB color space, or the HSV color space) to obtain two principal components (two elements), the first principal component and the second principal component. Then, the score Q may be obtained based on information on these principal components (two elements). Note that the Lab color space is, strictly speaking, a color space indicated by adding an * to each of the letters L, a, and b, but here it is simply referred to as the Lab color space.

[0152] When a principal component analysis was actually performed in the RGB color space on image data similar to that in the first embodiment, the following analysis results were obtained: the first principal component vector V1 = (R, G, B) = (0.59, 0.55, 0.57) and the second principal component vector V2 = (R, G, B) = (0.76, -0.16, -0.62).

[0153] This first principal component vector V1 is a vector having a direction close to vector V15 (=(1,1,1)) in Fig. 20 (a vector pointing from the innermost vertex of the cube in Fig. 20 to the innermost vertex of the cube), and represents (the direction of) luminance Y. Note that Fig. 20 is a conceptual diagram that shows a schematic representation of the RGB color space using a cube, and each of the R, G, and B values ​​is normalized to be in the range of 0 to 1.

[0154] Moreover, this second principal component vector V2 is a vector having a direction close to vector V25 (=(1,0,-1)) in FIG. 20. Vector V25 is a vector indicating the direction of the diagonal in the RB plane of the cube in FIG. 20 (direction connecting red and blue). Therefore, the second principal component vector V2 having a direction close to vector V25 can be expressed as color information E2 relating to the direction connecting red and blue in the RGB color space. The second principal component vector V2 etc. can also be expressed as color information relating to the direction from red to blue in the RGB color space or the opposite direction, color information indicating the degree of "reddish or blueish", or color information relating to the "red-blue" direction.

[0155] Similarly, principal component analysis in the Lab color space was performed, and the analysis results were obtained as follows: the first principal component vector V1 = (L, a, b) = (0.99, 0.06, -0.02) and the second principal component vector V2 = (L, a, b) = (-0.01, 0.58, 0.80). This first principal component vector V1 is a vector having a direction close to the vector V16 (= (1, 0, 0)) and represents the direction of lightness L (luminance Y). This second principal component vector V2 is a vector close to the vector V26 (= k4 * (0, 0.6, 0.8)). The second principal component vector V2 having a direction close to the vector V26 can be expressed as color information E2 relating to the direction connecting red and blue in the Lab color space, as shown in FIG. 21. The second principal component vector V2 etc. can also be expressed as color information relating to the direction from red to blue or the opposite direction in the Lab color space, color information indicating the degree of "reddish or blueish", or color information relating to the "red-blue" direction. Note that k4 is a coefficient for normalizing the magnitude of the principal component vector, and FIG. 21 is a conceptual diagram that typically expresses the Lab color space. Each value L, a, and b is normalized to be in the range of 0 to 1.

[0156] In the above-described embodiments, the conversion from the RGB color space to the YCbCr color space is performed, and the principal component analysis is performed in the YCbCr color space, but the present invention is not limited to this. For example, the principal component analysis may be performed in the RGB color space without converting from the RGB color space to the YCbCr color space.

[0157] As described above, the color space can be the YCbCr color space, the RGB color space, the Lab color space, or the HSV color space. However, considering the cumulative contribution rate up to the second principal component and the accuracy of the inference results by the inference model after learning, it is most preferable to use the YCbCr color space (or the YUV color space or the YPbPr color space) among various color spaces for the image of seaweed. The next most preferable color space is the RGB color space, and the next most preferable color space is the Lab color space.

[0158] The cumulative contribution rate up to the second principal component is larger when the YCbCr color space is used than when other RGB, Lab, or HSV color spaces are used. By using the color information E2 and luminance information E1 in the YCbCr color space, it is possible to construct a more appropriate model and, by extension, to obtain a more appropriate inference result (a result of judging the grade of the seaweed).

[0159] Moreover, color information E2 may be color information related to a direction D1 connecting red and blue in each color space (YCbCr color space, RGB color space, Lab color space, etc.). Most preferably, this direction D1 coincides with the direction of the second principal component vector V2 in each color space, but is not limited thereto. The direction D1 may be a direction shifted to a certain degree (for example, 20 degrees to 40 degrees) from the direction of the second principal component vector V2 in each color space, and may be a direction connecting red and blue in the color space.

[0160] <Principal component analysis> In the above-described embodiments, the grade determination device 30 does not need to perform principal component analysis every time the teacher data group changes. The first and second principal components once determined using a certain teacher data group may be used in a learning process executed using another teacher data group.

[0161] Furthermore, in the above-described embodiments, the grade determination device 30 performs the principal component analysis and determines the score based on information regarding the first principal component and the second principal component obtained by the principal component analysis, but is not limited to this.

[0162] For example, the grade determination device 30 may obtain the score Q based on the luminance information E1 and the color information E2 without performing the principal component analysis. The grade determination device 30 may use the luminance Y itself as the luminance information E1 without performing the principal component analysis, and may use the color information E2 obtained without performing the principal component analysis. Specifically, the above-mentioned second principal component vector V2 obtained in each of the above experiments may be adopted, and the inner product of the second principal component vector V2 and the data vector V may be used as the color information E2 related to the "red-blue" direction. Alternatively, the inner product of a predetermined vector having a direction close to the above-mentioned second principal component vector V2 obtained in each of the above experiments and the data vector V may be used as the color information E2 related to the "red-blue" direction. The predetermined vector is, for example, a neighboring vector such as (Y, Cb, Cr)=(0, -0.7, +0, 7) in the YCbCr color space.

[0163] <Learning model> In the second embodiment, a two-stage learning model is used, but the present invention is not limited to this, and a one-stage learning model may be used. More specifically, a one-stage learning model may be used in which an index value P5 relating to the degree of focus is input in addition to the luminance information E1 and color information E2, and a score relating to the grade of seaweed is output.

[0164] In addition, in the above-mentioned embodiments, a multiple regression analysis model is used as the learning model, but the present invention is not limited to this. For example, a neural network model composed of multiple layers may be used as the learning model. It is sufficient that parameters (learning parameters) in multiple layers (particularly multiple intermediate layers) in the neural network model are adjusted by machine learning.

[0165] Furthermore, in the above-described embodiments, a regression model is used as the learning model, but the present invention is not limited to this, and a classification model may also be used.

[0166] <Other> In addition, in each of the above-mentioned embodiments, the grade determination device (information processing device) 30 determines the grade of the seaweed based on the score Q, but the present invention is not limited to this. For example, a device provided separately from the information processing device 30 that outputs the score Q may determine the grade of the seaweed (specifically, the "color grade" or the "overall grade" etc.) based on the score Q (and other elements). Alternatively, the grade determination device 30 may display the score Q on the display unit 35b (or display it on another display device using the communication unit 34) to present (report) the score Q to a person (an inspector etc.), and the person may (finally) determine the grade of the seaweed based on the score Q. Alternatively, the person may determine the grade of the seaweed (particularly the "overall grade" etc.) based on the presented score Q and other elements (weight of the seaweed etc.). [Explanation of symbols]

[0167] 1 Grading System 10. Conveyor 11 Conveyor belt 20 Imaging Equipment 21 Light source 22 Line sensor camera 23 Seaweed detection sensor 30 Grade determination device 91 Seaweed 200,210,212 images 400,420,450,470,500 Learning Model D1 direction E1 Luminance information E2 Color Information P1, P2 Principal component scores P5 Index Value Q, Q1, Q2 scores V1,V2 Principal component vectors

Claims

1. a control unit that calculates a score for judging the grade of the seaweed based on luminance information of an image of the seaweed and color information of the image, the color information indicating color characteristics in a direction connecting red and blue in a predetermined color space; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the predetermined color space is any one of a YCbCr color space, a YUV color space, a YPbPr color space, an RGB color space, and a Lab color space.

3. 2 . The information processing apparatus according to claim 1 , wherein the luminance information and the color information are each calculated based on an average value obtained by averaging a plurality of pixel values ​​in the image for each channel in the predetermined color space.

4. The luminance information is information on a first principal component of the image based on a result of a principal component analysis of the average values ​​obtained for a plurality of images of seaweed, The information processing apparatus according to claim 3 , wherein the color information is information of a second principal component related to the image based on the analysis result of the principal component analysis.

5. The information processing apparatus according to claim 1 , wherein the score for the image is obtained based on an analysis result of an interaction linear analysis that takes into account an interaction between two explanatory variables, the luminance information and the color information.

6. The information processing device according to claim 1 , wherein the control unit calculates the score based also on an index value relating to a degree of focus of the image.

7. 2. The information processing device according to claim 1, wherein the score for the image is an evaluation value that is output by inputting the luminance information and the color information for the image into a learning model trained based on the luminance information, the color information, and graded correct answer data for a plurality of learning images.

8. the learning model is a first learning model, The control unit calculates an improved score for determining the grade of the seaweed based on the second learning model, the second learning model is a learning model that receives as input the score output from the first learning model for an input image and an index value related to a focus degree of the input image, and outputs the improved score, and is a learning model that is trained using a plurality of teacher data; The information processing device according to claim 7, characterized in that the control unit inputs the score output from the first learning model for the image to be inferred and an index value related to the degree of focus of the image into the second learning model, and obtains the improved score output from the second learning model.

9. a) calculating a score for judging the grade of the seaweed based on luminance information of an image of the seaweed and color information of the image, the color information indicating color characteristics in a direction connecting red and blue in a predetermined color space; An information processing method comprising:

10. A program for causing a computer to execute the information processing method according to claim 9.

11. A method for producing a learning model for determining the grade of seaweed, comprising: A step of learning the learning model based on multiple teacher data, which receives input of luminance information of an image of the seaweed and color information of the image that indicates color characteristics in a direction connecting red and blue in a predetermined color space, and outputs a score related to the rank of the seaweed; A method for producing a learning model, comprising:

12. a control unit which acquires a first principal component score and a second principal component score of an image to be inferred based on a result of a principal component analysis in a predetermined color space for a plurality of images of seaweed, inputs the first principal component score and the second principal component score of the image to be inferred to a learning model which has learned the relationship between the first principal component score and the second principal component score of each image and the class of seaweed in each image using a plurality of teacher data, and acquires a score output from the learning model for determining the class of seaweed in the image to be inferred; An information processing device comprising:

13. a) performing a principal component analysis in a predetermined color space on a plurality of images of seaweed, and acquiring a first principal component and a second principal component by the principal component analysis; b) a step of machine learning a learning model using a plurality of training data, the learning model inputting the first principal component score and the second principal component score of an input image and outputting a score for judging the grade of the seaweed of the input image; A method for producing a learning model, comprising:

14. a) obtaining a first principal component score and a second principal component score of an inference target image based on a result of a principal component analysis in a predetermined color space for a plurality of images of seaweed; b) inputting the first principal component score and the second principal component score of the inference target image into a learning model that has learned the relationship between the first principal component score and the second principal component score of each image and the grade of the seaweed in each image using multiple teacher data, and acquiring a score output from the learning model for determining the grade of the seaweed in the inference target image; An information processing method comprising:

15. A program for causing a computer to execute the method according to claim 13 or 14.

Images