Information processing device, information processing method, food inspection device, and food manufacturing method

Abstract

To provide an information processing device, information processing method, food product inspection device and food product manufacturing method which can detect a foreign matter existing in a food product.SOLUTION: An information processing device according to the present disclosure comprises: an inspection image generation unit which generates an inspection image on the basis of a difference between imaging data of first light obtained by polarizing reflection light of light polarized in the third direction and irradiated to a food product in the first direction in parallel to the third direction and imaging data of second light obtained by polarizing the reflection light in the second direction orthogonal to the first direction; and a detection processing unit which detects a foreign matter existing in the food product on the basis of the inspection image and a machine learning model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, a food inspection apparatus, and a food manufacturing method.

Background Art

[0002] When manufacturing processed foods such as frozen foods, it is necessary to detect and remove foreign substances present in the food. For example, when manufacturing frozen vegetables, it is necessary to detect and remove foreign substances such as insects mixed on the surface and inside of the vegetables. Patent Documents 1 and 2 describe a technique for generating an inspection image from a plurality of image data captured by combining near-infrared inspection light and a polarizer, and detecting foreign substances mixed on the surface and inside of vegetables based on the generated inspection image.

[0003] In the techniques described in Patent Documents 1 and 2, the process of detecting foreign substances from the inspection image is performed visually by a human. However, there are also foreign substances that cannot be detected by human vision.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The present disclosure is for solving the above problems, and an object thereof is to provide an information processing apparatus, an information processing method, a food inspection apparatus, and a food manufacturing method capable of detecting foreign substances present in food.

Means for Solving the Problems

[0006] The information processing device of this disclosure includes an inspection image generation unit that generates an inspection image based on the difference between imaging data of a first light obtained by polarizing the reflected light of light that is polarized in a third direction and irradiated onto food in a first direction parallel to the third direction, and imaging data of a second light obtained by polarizing the reflected light in a second direction orthogonal to the first direction, and a detection processing unit that performs detection processing to detect foreign objects present in the food based on the inspection image and a machine learning model. [Brief explanation of the drawing]

[0007] [Figure 1] This diagram shows the configuration of a food inspection device according to an embodiment. [Figure 2A] This figure shows an example of a parallel image. [Figure 2B] This figure shows an example of an orthogonal image. [Figure 3] This is a block diagram showing the functional configuration of the processing device in the food inspection apparatus according to the embodiment. [Figure 4A] This is a flowchart illustrating the details of the processing performed by the processing unit in the food inspection apparatus according to the embodiment. [Figure 4B] This is a flowchart illustrating the details of the processing performed by the processing unit in the food inspection apparatus according to the embodiment. [Figure 5] This figure shows an example of an examination image. [Figure 6A] This figure shows an example of a histogram of brightness values ​​in inspection images of food containing foreign objects. [Figure 6B] This figure shows an example of a histogram of brightness values ​​in an inspection image of food in which no foreign objects are present. [Figure 7] This figure shows the relationship between the luminance value before and after linear density conversion. [Figure 8] This figure shows an example of an examination image after linear density conversion has been performed. [Figure 9] This figure shows the results of a performance evaluation experiment according to the embodiment. [Figure 10]This is a block diagram showing an example of a processing device in a food inspection apparatus according to a modified example 1 of the embodiment. [Figure 11] This is a block diagram showing another example of a processing device in a food inspection apparatus according to a modified example 1 of the embodiment. [Figure 12] This figure shows a food manufacturing apparatus equipped with a food inspection device and a food removal device. [Modes for carrying out the invention]

[0008] Embodiments of the present disclosure will be described below with reference to the drawings. In the following description, the same or corresponding elements in the drawings will be denoted by the same reference numerals, and detailed descriptions will be omitted as appropriate.

[0009] Figure 1 shows the configuration of a food inspection device 100 according to an embodiment of the present disclosure. The food inspection device 100 is a device that detects foreign matter present in food, more specifically, foreign matter present on at least one of the surface and interior of food. Specific examples of food include vegetables or frozen vegetables, more specifically, broccoli, spinach, or edamame. Specific examples of foreign matter include organic matter, more specifically, larvae and adults of armyworms, moths, aphids, or ladybugs.

[0010] The food inspection device 100 includes a flat plate 2 on which the food item 1 to be inspected is placed, a light source 3 that irradiates the food item 1 with inspection light 11, and an imaging device 4 that captures reflected light 12 from the food item 1 and generates image data. The food inspection device 100 also includes polarizers 5-7, guide rails 8, a processing device 9, a drive device 10, an input device 13, and an output device 14. The processing device 9 includes an information processing unit 9a that performs detection processing to detect foreign objects present in the food item 1, and a controller 9b. The processing device 9 is also connected to the input device 13 and the output device 14, and performs input and output of information or data between the input device 13 and the output device 14.

[0011] The light source 3 irradiates the inspection light 11 onto the food 1 which is the inspection object. In the present embodiment, the light source 3 is constituted by an LED panel on which a plurality of light-emitting diodes (LEDs) are arranged on a substrate. The light source 3 is arranged parallel to the flat plate 2, that is, parallel to the XY plane in the figure. Also, the surface on which the LEDs of the light source 3 are arranged is directed in the direction of irradiating the inspection light 11 onto the food 1, that is, the -Z direction in the figure. The type of the light source 3 is not limited to the LED panel, and it may be a single LED lamp, and the use of light sources other than LEDs (for example, incandescent lamps, fluorescent lamps, lasers, superluminescent diodes, etc.) is not excluded either.

[0012] The wavelength of the inspection light 11 is preferably a wavelength that easily transmits through the food 1 which is the inspection object. For example, when the food 1 is a vegetable such as broccoli, spinach or edamame, the inspection light is preferably near-infrared light with a wavelength of about 600 nm to 900 nm. Since such a wavelength range is within the range that can be imaged by Si-based imaging elements such as general CCD sensors and CMOS sensors, the food inspection device 100 can be easily configured. However, the case of using light other than near-infrared light for the inspection light, for example, visible light or ultraviolet light, is not excluded either. Also, if an appropriate detection element is selected, the mid- and far-infrared and terahertz regions are not excluded either. <000..​​​​​Between the light source 3 and the food 1, a polarizer 5 is positioned parallel to the light source 3 and the flat plate 2, i.e., parallel to the XY plane in the figure. The polarizer 5 linearly polarizes the inspection light irradiated from the light source 3 toward the food 1 in the X direction in the figure.

[0015] Furthermore, polarizers 6 and 7 are positioned between the food 1 and the imaging device 4 at different heights, parallel to the light source 3 and the flat plate 2, i.e., parallel to the XY plane in the figure. Polarizers 6 and 7 are located at the same height in the Z direction and are integrally formed in the Y direction. Polarizers 6 and 7 are configured to slide along the guide rail 8 in the +Y and -Y directions in the figure by a drive device 10 attached to the guide rail 8. The drive device 10 is a drive unit including a motor, such as a stepping motor. If a rotation mechanism, such as those described in Patent Documents 1 and 2, is used instead of this sliding mechanism, polarizers 6 and 7 can be combined into one. Also, the motor included in the drive device 10 does not necessarily have to be a stepping motor; a regular DC motor or AC motor may be used. Furthermore, the drive device 10 may be removed, and the sliding of polarizers 6 and 7 may be performed manually.

[0016] When the polarizer 6 is positioned in the optical path between the food 1 and the imaging device 4, that is, in the state shown in Figure 1, the polarizer 6 linearly polarizes the reflected light 12 from the food 1 in the X direction in the figure. The direction polarized by the polarizer 5 (X direction) and the direction polarized by the polarizer 6 (X direction) are parallel. In other words, if the polarization direction of the light polarized by the polarizer 5 (third light) is the third direction, and the polarization direction of the light polarized by the polarizer 6 (first light) is the first direction, then the third direction and the first direction are parallel. The imaging device 4 captures the reflected light that has passed through the polarizer 6 and generates image data. Hereafter, such image data will be referred to as "parallel image P". Figure 2A is an example of a parallel image P, with a size of 224 × 224 and 256 grayscale levels from 0 to 255.

[0017] On the other hand, when the polarizer 7 is positioned in the optical path between the food 1 and the imaging device 4, the polarizer 7 linearly polarizes the reflected light 12 from the food 1 in the Y direction in the figure. In this case, the image data generated has a polarization direction (X direction) that is orthogonal to the polarization direction (Y direction) that is polarized by the polarizer 7. In other words, if the polarization direction of the light polarized by the polarizer 5 (third light) is the third direction, and the polarization direction of the light polarized by the polarizer 7 (second light) that is polarized from the reflected light 12 (second light) is the second direction, then the second direction is orthogonal to the third direction. The imaging device 4 captures the reflected light that has passed through the polarizer 7 and generates image data. Hereafter, such image data will be referred to as "orthogonal image C". Figure 2B is an example of orthogonal image C, with a size of 224 × 224 and 256 grayscale levels from 0 to 255.

[0018] In the configuration shown in Figure 1, polarizers 6 and 7 were arranged together at the same height. However, they may also be arranged at different heights in the Z direction and configured to slide independently in the Y direction.

[0019] For the polarizers 5 to 7 described above, for example, polarizers can be made by attaching a polarizing film to a substrate such as a glass plate or a thermoplastic resin film. However, if the inspection light 11 is near-infrared light, a polarizing film for visible light will not provide sufficient polarization performance, so it is preferable to use a wire grid type polarizing film, such as WGF (registered trademark) manufactured by Asahi Kasei E-Materials.

[0020] The input device 13 is an input unit that allows the user, who is the operator of the food inspection device 100, to input various instructions or information to the processing device 9. The input device 13 may be, for example, a mouse, keyboard, or touch panel. The input device 13 may also be a device that allows input by voice or gesture.

[0021] The output device 14 is an output device that outputs data or information to the user. The output device 14 is, for example, a display that shows data or information, such as a liquid crystal display or an organic EL (electroluminescence) display. Alternatively, the output device 14 may be a communication device that transmits data or information to a terminal device held by the user. In this embodiment, we assume that the output device 14 is a display.

[0022] Figure 3 is a block diagram of the processing unit 9 of the food inspection device 100. The processing unit 9 can be configured by, for example, a microcomputer or a personal computer. Some or all of the functions of the processing unit 9 may be realized by having the computer execute a program.

[0023] The processing device 9 comprises an information processing unit 9a and a controller 9b, which correspond to the information processing device according to this embodiment. The information processing unit 9a comprises an inspection image generation unit 91, a filter processing unit 92 (conversion unit), a clustering unit 93 (setting unit), a first linear density conversion unit 94a, a second linear density conversion unit 94b, a detection processing unit 95, a first reduction unit 96a, a second reduction unit 96b, and a determination unit 97. The detection processing unit 95 comprises a first detection unit 95a and a second detection unit 95b. The controller 9b controls the information processing unit 9a, the drive unit 10, the light source 3, and the imaging device 4.

[0024] The inspection image generation unit 91 generates an inspection image based on a parallel image P obtained by imaging the reflected light that has passed through the polarizer 6 with the imaging device 4, and an orthogonal image C obtained by imaging the reflected light that has passed through the polarizer 7 with the imaging device 4. For example, the inspection image is generated by weighting the parallel image P and the orthogonal image C. Specifically, by setting the weight of the parallel image P to 1 and the weight of the orthogonal image C to -1, the inspection image is generated based on the difference between the parallel image P and the orthogonal image C. If the pixel value (luminance value) of each pixel in the inspection image exceeds a certain range, the pixel value exceeding the certain range may be rounded up to the minimum value of the certain range, or rounded down to the maximum value of the certain range. One example of a certain range is 0 or more, corresponding to the minimum luminance, and 255 or less, corresponding to the maximum luminance. However, the range of pixel values ​​is not limited to this range and may be other ranges.

[0025] The filter processing unit 92 performs a filter operation in the inspection image generated by the inspection image generation unit 91, converting luminance values ​​other than those between the first threshold and the second threshold to a predetermined luminance value (a predetermined pixel value). In other words, it does not convert the luminance values ​​between the first threshold and the second threshold in the inspection image, but converts the other luminance values ​​to the same luminance value outside that range. The target luminance value is a value smaller than the first threshold or larger than the second threshold, for example, the minimum luminance value or the maximum luminance value. The filter processing unit 92 corresponds to a conversion unit that converts luminance values ​​outside the range between the first and second thresholds to a predetermined luminance value that is smaller than the first threshold or larger than the second threshold. The first and second thresholds may be stored in advance in the processing unit 9 or in a storage unit accessible from the processing unit 9, or they may be input as user-adjustable parameters from the input device 13. In this case, the user can input the parameters from the screen of the application that performs the inspection according to this embodiment.

[0026] As an example, all luminance values ​​below the first threshold and above the second threshold are converted to 0 or 255. As another example, luminance values ​​below the first threshold may be converted to 0, which corresponds to the minimum luminance, and luminance values ​​above the second threshold may be converted to 255, which corresponds to the maximum luminance. The first threshold is a value greater than the minimum luminance, and the second threshold is a value less than the maximum luminance. Luminance values ​​below the first threshold mainly originate from the background of the subject being photographed, and luminance values ​​above the second threshold mainly originate from food 1 itself, and are considered to be luminance values ​​with low correlation to foreign objects. Therefore, when the inspection image is considered as a grayscale image, by converting luminance values ​​that do not fall between the first and second thresholds to a predetermined luminance value outside this range (e.g., the minimum luminance value), an image focusing on the luminance range between the first and second thresholds (a specific gray area within the gray area between white and black) can be obtained. By obtaining an image that focuses on a specific gray area in this way, the amount of information with low correlation to foreign objects can be reduced from the inspection image, thereby improving the accuracy of foreign object detection. Hereinafter, the converted image may be referred to as the filtered inspection image.

[0027] The first linear density conversion unit 94a performs a linear density conversion on the filtered inspection image, converting the luminance values ​​between the first threshold and the second threshold to a wider luminance range (pixel value range) than the range between the first threshold and the second threshold. The luminance range includes the range between the first threshold and the second threshold. This normalizes the luminance values ​​between the first threshold and the second threshold. For example, the luminance values ​​between the first threshold and the second threshold are converted to luminance values ​​in the range of 0 to 255. Alternatively, the luminance values ​​between the first threshold and the second threshold may be converted to luminance values ​​in the range of greater than 0 and less than 255.

[0028] The first detection unit 95a uses a first machine learning model Ma to detect foreign objects present in the food 1 based on the filtered and linearly density-transformed inspection image. As the first machine learning model Ma, for example, a pre-trained regression model can be used. An example of a regression model is a neural network. For example, a model in which the fully connected layers of a pre-trained 16-layer convolutional neural network (CNN) known as "VGG16" have been fine-tuned may be used. That is, a model can be used in which VGG16 has been trained to detect foreign objects present in the food 1, such as on the surface and inside, from the filtered and linearly density-transformed inspection image. In the case of a neural network, a configuration can be used that has an input node into which the pixel values ​​of the linearly density-transformed inspection image are input, and an output node that outputs a value regarding the presence or absence of foreign objects as a judgment result. A configuration can be used in which 1 is output from the output node when a foreign object is detected. For example, as a judgment result, 1 may be output from the output node if a foreign object is detected, and 0 may be output from the output node if no foreign object is detected. Alternatively, the output node may output a value indicating the probability of the presence of a foreign object as the result of the judgment. The regression model is not limited to neural networks; other models such as linear regression models, logistic regression models, or multiple regression models may also be used.

[0029] The clustering unit 93 sets multiple luminance value ranges (first luminance range or first pixel value range) for the inspection image generated by the inspection image generation unit 91, and assigns identification information (luminance range identification information) to pixels belonging to the set luminance range to identify the luminance range to which the pixel belongs. This generates clusters in the inspection image as groups of pixel values ​​belonging to each luminance range. The clustering unit 93 corresponds to a setting unit that sets multiple luminance ranges and generates groups of pixel values ​​(clusters) corresponding to each luminance range. The process of generating clusters corresponding to luminance ranges in an inspection image is called clustering the inspection image. The identification information can be any value as long as it can identify each cluster. As an example of identification information, information representing color such as R, G, B (color information) may be used, or symbols such as CL1, CL2, CL3 may be used. The luminance range identification information may be treated as part of the pixel value. For example, the pixel value may include both the luminance value and the luminance range identification information. In the following explanation, we will assume that color information is mainly used as the luminance range identification information. Also, an inspection image to which luminance range identification information has been assigned is called a clustered inspection image. Details on how to set multiple brightness ranges will be provided later.

[0030] The second linear density conversion unit 94b performs a linear density conversion on the clustered inspection image, normalizing the luminance values ​​included in each luminance range. That is, the second linear density conversion unit 94b performs a linear density conversion on the luminance values ​​of pixels belonging to each luminance range of the clustered inspection image. In detail, the second linear density conversion unit 94b sets a first threshold and a second threshold for each luminance range and performs a linear density conversion on the luminance values ​​included in the luminance range in the same manner as the first linear density conversion unit 94a. This converts (normalizes) the luminance values ​​belonging to each luminance range to a wider luminance range (second luminance range or second pixel value range). For example, the luminance values ​​of pixels belonging to each luminance range are converted to luminance values ​​in the range of 0 to 255. The second linear density conversion unit 94b converts the luminance values ​​of pixels that do not belong to any luminance range to a predetermined luminance value (e.g., zero).

[0031] The second detection unit 95b uses a second machine learning model Mb to detect foreign objects present in food 1 based on the clustered and linearly density-transformed inspection image. As an example of the second machine learning model Mb, a regression model such as a neural network can be used, similar to the first machine learning model Ma. For example, a model trained to detect foreign objects present in food 1 from the clustered and linearly density-transformed inspection image can be used by fine-tuning the fully connected layer of VGG16. If the second machine learning model Mb is a neural network, a configuration can be used that has an input node into which the pixel values ​​(luminance values ​​and color information) of the clustered and linearly density-transformed inspection image are input, and an output node that outputs a value regarding the presence or absence of a foreign object as a determination result. For example, as a determination result, if a foreign object is detected, 1 may be output from the output node, and if no foreign object is detected, 0 may be output from the output node. Alternatively, as a determination result, a value indicating the probability of the presence of a foreign object may be output from the output node.

[0032] The determination unit 97 determines whether or not a foreign object is present in the food 1 based on the detection results of the first detection unit 95a and the second detection unit 95b. For example, if the determination unit 97 detects the presence of a foreign object by at least one of the first detection unit 95a and the second detection unit 95b, it determines that a foreign object is present in the food 1. If the first detection unit 95a and the second detection unit 95b output the probability of the presence of a foreign object, the determination unit 97 may determine that a foreign object is present in the food 1 if at least one of the probabilities or the average is greater than or equal to a threshold. If neither the first detection unit 95a nor the second detection unit 95b detects the presence of a foreign object, the determination unit 97 determines that a foreign object is not present in the food 1. The determination unit 97 may output information indicating the foreign object detection result to a display device (output device 14) and display the foreign object detection determination result on the display device. In addition, along with the foreign object detection determination result, the determination unit 97 may display an image of the food 1 (at least one of the parallel image P and the orthogonal image C), an inspection image, or both. If the presence of a foreign object is determined, information indicating the location of the foreign object may be displayed in the examination image.

[0033] Next, the details of the process performed by the processing unit 9 of the food inspection device 100 according to this embodiment to determine whether or not foreign matter is present in the food 1 will be explained with reference to the flowchart in Figure 4.

[0034] In step S101 of Figure 4, the inspection image generation unit 91 generates an inspection image from the difference between a parallel image P (Figure 2A) obtained by capturing the reflected light that has passed through the polarizer 6 with the imaging device 4, and an orthogonal image C (Figure 2B) obtained by capturing the reflected light that has passed through the polarizer 7 with the imaging device 4.

[0035] In detail, for example, the inspection image generation unit 91 generates an inspection image by subtracting the brightness value of each corresponding pixel in the orthogonal image C from the brightness value of each pixel in the parallel image P. In this case, for brightness values ​​for which the subtraction result is negative, the brightness value is set to a predetermined brightness value (zero in this example). The size and grayscale (brightness) range of the inspection image can also be arbitrarily selected. In this embodiment, as an example, the size of the inspection image is 224 × 224, and the grayscale is 256 levels (8 bits) from 0 to 255. Figure 5 is an example of an inspection image according to this embodiment, which is an inspection image of broccoli contaminated with cutworms, which are larvae of the cutworm moth species. The noise contained in the inspection image is presumed to originate from, for example, the shadows of the broccoli florets.

[0036] In step S102, the filter processing unit 92 generates a filtered inspection image in which the luminance values ​​other than those between the first threshold T1 and the second threshold T2 in the inspection image generated in step S101 are converted to predetermined luminance values. In this embodiment, as an example, the first threshold T1 = 30, the second threshold T2 = 225, and the predetermined luminance values ​​are zero and 255. In this case, in the inspection image of Figure 5, the filter processing unit 92 leaves the luminance values ​​between 30 and 225 as they are, and sets all luminance values ​​between 0 and 29 and between 226 and 255 to zero.

[0037] The filtering process described above is shown in Figures 6A and 6B when considered on a histogram. In Figures 6A and 6B, the horizontal axis represents brightness value and the vertical axis represents frequency. Figure 6A is an example of a histogram of an inspection image when cutworms are present in broccoli. Figure 6B is an example of a histogram of an inspection image when cutworms are not present in broccoli.

[0038] Comparing Figure 6A and Figure 6B, in Figure 6A, where cutworms are present, pixels with brightness values ​​in the range of approximately 30 to 225 appear. In contrast, in Figure 6B, where cutworms are not present, pixels with such brightness values ​​are almost nonexistent. That is, it is presumed that the brightness values ​​in the range of 30 to 255 in Figure 6A originate from the growth components, patterns, and moisture of the cutworms present in the broccoli. Brightness values ​​outside this range are thought to be predominantly derived from the background of the broccoli being photographed and the broccoli itself. The brightness value ranges D1 to D3 in Figure 6 will be discussed later.

[0039] Therefore, by setting the first threshold T1=30 and the second threshold T2=225, the filter processing unit 92 can convert the inspection image generated in step S101 above into an inspection image that mainly contains brightness values ​​derived from cutworms mixed in with the broccoli.

[0040] The above example involved cutworms found in broccoli, but similarly, in the case of other insects found in other vegetables, the inspection image can be converted to primarily contain brightness values ​​originating from the insects by appropriately setting the first threshold T1 and the second threshold T2. Furthermore, the same filtering process can generally be applied when detecting foreign substances such as organic matter found in food.

[0041] In step S103, the first linear density conversion unit 94a applies a linear density conversion defined by the following equation (1) to each pixel of the filtered inspection image obtained in step 102.

[0042]

number

[0043] However, in the above equation, X in X is the brightness value of the pixel before conversion. outis the brightness value after pixel conversion, M1 is the minimum brightness value (0) after conversion of the inspection image, M2 is the maximum brightness value (255) after conversion of the inspection image, T1 is the first threshold mentioned above, and T2 is the second threshold mentioned above.

[0044] Figure 7 shows the luminance value X before conversion in linear density conversion. in and the converted brightness value X out This diagram illustrates the relationship between the two. As can be seen from Figure 7, by applying a linear density transformation, the luminance values ​​between the first threshold T1 and the second threshold T2 are normalized between the minimum luminance value M1=0 and the maximum luminance value M2=255, thereby emphasizing the contrast between the luminance values ​​between T1 and T2 in the inspection image. Note that luminance values ​​other than those between the first threshold T1 and the second threshold T2 are converted to a predetermined minimum luminance value (=0) beforehand by the filtering process described above. Figure 8 is an example of an inspection image after the linear density transformation has been applied. It can be seen that the contrast of the foreign object (cutworm) in the center of the image is emphasized, and noise originating from shadows of broccoli florets, etc., has been removed.

[0045] In step S104, the first detection unit 95a performs a detection process to detect foreign objects contained in the food 1 based on the inspection image that underwent linear density transformation in step S103, using the first machine learning model Ma. The first machine learning model Ma has an input node into which the pixel values ​​of the inspection image are input, and an output node that outputs the discrimination result. For example, if a foreign object is detected, 1 is output from the output node, and if no foreign object is detected, 0 is output from the output node.

[0046] If no foreign matter is detected in step S104 (step S105 = NO), in step S106, the first reduction unit 96a performs a reduction process on the inspection image in which no foreign matter was detected. In the reduction process, the size of the inspection image is reduced by a predetermined percentage, and the surrounding area is filled with zero values ​​to return it to the same size as the original image. In this embodiment, as an example, an inspection image of size 224 × 224 is reduced to 90 percent to a size of 201 × 201, and then the surrounding area is filled with pixels of zero values ​​to return it to the same size as the original 224 × 224.

[0047] By performing this reduction process, if a foreign object is visible at the edge of the inspection image but is not detected by the first machine learning model Ma, the foreign object may become more easily detected as it moves closer to the center of the inspection image.

[0048] In step S107, the first detection unit 95a performs the detection process to detect foreign objects again based on the inspection image that has been reduced in step S106. The first machine learning model Ma outputs 1 if a foreign object is detected, and 0 if no foreign object is detected.

[0049] In step S108, the clustering unit 93 clusters the inspection image generated in step 101. For example, if the histogram of the inspection image is as shown in Figure 6A, the clustering unit 93 sets three luminance ranges corresponding to the three peaks in the histogram in Figure 6A. In this example, the range D1 of luminance values ​​from 30 to 94, which includes the leftmost peak of the histogram in Figure 6A, is associated with color information (luminance range identification information) indicating R. Similarly, the range D2 of luminance values ​​from 95 to 159, which includes the central peak, is associated with color information (luminance range identification information) indicating G. Similarly, the range D3 of luminance values ​​from 160 to 225, which includes the rightmost peak, is associated with color information (luminance range identification information) indicating B. Pixel value groups assigned the same color information (luminance range identification information) correspond to the same cluster. In this example, a cluster corresponding to range D1, a cluster corresponding to range D2, and a cluster corresponding to range D3 are generated. For pixels that are not associated with any of the R, G, or B color information, predetermined color information (for example, color information indicating black) may be associated with them, or the color information may be set to a null value.

[0050] Even if the three peaks do not appear clearly as shown in Figure 6A, or if the number of peaks is not three, the inspection image generated in step 101 can be converted into a clustered inspection image in the same manner as described above by experimentally investigating an appropriate range of brightness values.

[0051] In step S109, the second linear density conversion unit 94b performs linear density conversion on each of the luminance ranges D1 to D3 of the inspection image clustered in step 108. Specifically, the second linear density conversion unit 94b performs linear density conversion on the pixel data of the luminance range R corresponding to the leftmost peak in Figure 6A, setting the first threshold T1=30 and the second threshold T2=94, using the above-mentioned equation (1). As a result, the luminance values ​​of pixels belonging to luminance range D1 are converted to luminance values ​​in the range of 0 to 255.

[0052] Similarly, the second linear density conversion unit 94b performs a linear density conversion on the pixel data in the luminance range D2 corresponding to the central peak, setting the first threshold T1 = 95 and the second threshold T2 = 159, using the above-described equation (1). As a result, the luminance values ​​of pixels belonging to the luminance range D2 are converted to luminance values ​​in the range of 0 to 255.

[0053] Similarly, the second linear density conversion unit 94b performs a linear density conversion on the pixel data in the luminance range D3 corresponding to the rightmost peak, setting the first threshold T1=160 and the second threshold T2=225, using the above-described equation (1). As a result, the luminance values ​​of pixels belonging to the luminance range D3 are converted to luminance values ​​in the range of 0 to 255.

[0054] The second linear density conversion unit 94b converts the brightness values ​​of pixels that do not belong to the brightness range D1 to D3 to a predetermined brightness value (for example, zero).

[0055] In step S110, the second detection unit 95b uses the second machine learning model Mb to perform a detection process to detect foreign objects contained in the food 1 based on the inspection image that underwent linear density transformation in step S109. The second machine learning model Mb has an input node into which the pixel values ​​(luminance values ​​and color information) of the inspection image that underwent linear density transformation are input, and an output node that outputs the discrimination result. For example, if a foreign object is detected, 1 is output from the output node, and if no foreign object is detected, 0 is output from the output node.

[0056] If no foreign matter is detected in step S110 (step S111=NO), then in step S111, the second reduction unit 96b performs a reduction process on the inspection image in which no foreign matter was detected. In the reduction process, the size of the inspection image is reduced by a predetermined percentage, and the surrounding area is filled with zero values ​​to return it to the same size as the original image. In this embodiment, as an example, an inspection image of size 224×224 is reduced to 90 percent to a size of 201×201, and then the surrounding area is filled with zero values ​​to return it to the same size as the original 224×224.

[0057] By performing this reduction process, if a foreign object is visible at the edge of the inspection image but is not detected by the second machine learning model Mb, the foreign object may become more easily detected as it moves closer to the center of the inspection image.

[0058] In step S113, the second detection unit 95b performs a detection process to detect foreign objects again based on the inspection image that has been reduced in step S113. The second machine learning model Mb outputs 1 if a foreign object is detected, and 0 if no foreign object is detected.

[0059] In step S114, the determination unit 97 determines whether or not a foreign object is present in the food 1 based on the detection result of at least one of the steps S104, S107, S110, and S113. Specifically, if a foreign object is detected in at least one of these steps, the determination unit 97 determines that a foreign object is present on or inside the food 1.

[0060] In the flowchart of Figure 4A, the reduction process is performed once in step S106, but step S106 may be repeated one or more times until a foreign object is detected in the detection process in step S107. If no foreign object is detected after repeating step S106 a predetermined number of times, the process may proceed to step S108. Similarly, in the flowchart of Figure 4B, the reduction process is performed once in step S112, but step S112 may be repeated one or more times until a foreign object is detected in the detection process in step S113. If no foreign object is detected after repeating step S112 a predetermined number of times, the process may proceed to step S114.

[0061] Furthermore, in the flowcharts of Figures 4A and 4B, the reduction process is performed in steps S107 and S113 if no foreign matter is detected. However, the reduction process may also be performed between steps S101 and S102. For example, if it is known in advance that the areas most likely to contain foreign matter are the edges of the inspection image, the accuracy of foreign matter detection can be improved by performing the reduction process beforehand.

[0062] [Performance evaluation results] To evaluate the performance of the food inspection device 100 according to this embodiment, 209 inspection images for performance evaluation were created, including images showing cutworms mixed in on the surface or inside broccoli. Of the 209 inspection images for performance evaluation, 108 were images showing actual cutworm contamination, and the remaining 101 were images showing no contamination.

[0063] Figure 9 is a table showing the results of the performance evaluation experiment described above. As can be seen from this figure, only two inspection images failed to detect insects despite actually containing them, while the remaining 106 images successfully detected insects. Furthermore, there were no cases of false detection despite the absence of insects. As a result, the success rate of the detection was 98 percent.

[0064] As described above, according to this embodiment, an inspection image is generated based on imaging data of light transmitted through a polarizer that polarizes reflected light in a first direction (first polarizer) and imaging data of light transmitted through a polarizer that polarizes reflected light in a second direction different from the first direction (second polarizer). By performing a foreign object detection process based on the generated inspection image, foreign objects present in the food 1 can be detected.

[0065] According to this embodiment, by making the first and second directions orthogonal, inspection images that enable more accurate detection of foreign objects can be generated. Specifically, first, inspection light to be irradiated onto the food 1 is transmitted through a polarizer (third polarizer) that polarizes light in a direction parallel to the first direction (third direction), and the light transmitted through the third polarizer is irradiated onto the food 1. Next, the reflected light from the food 1 is transmitted through a polarizer 6 (first polarizer) that polarizes light in a direction parallel to the third direction (first direction), thereby acquiring a parallel image P as the captured image. Similarly, the reflected light from the food 1 is transmitted through a polarizer 7 (second polarizer) that polarizes light in a direction orthogonal to the third direction (second direction), thereby acquiring an orthogonal image C as the captured image. This makes it possible to generate inspection images that enable more accurate detection of foreign objects.

[0066] According to this embodiment, pixel values ​​(luminance values) outside the range between the first threshold and the second threshold in the inspection image are converted to predetermined luminance values ​​that are smaller than the first threshold or larger than the second luminance value. This reduces or eliminates information based on luminance values ​​outside the range between the first and second thresholds, i.e., information that is unlikely to originate from a foreign object, from the inspection image, thereby enabling detection of foreign objects with high accuracy.

[0067] According to this embodiment, multiple brightness ranges are set within the range of pixel values ​​(brightness values) that an inspection image can take, and information identifying the range to which a pixel belongs (brightness range identification information) is set for pixels belonging to one of the multiple brightness ranges, thereby clustering the inspection image. By using the clustered inspection image, it becomes possible to detect foreign objects with higher accuracy in the machine learning model Mb. In other words, pixels belonging to the same brightness range are considered to be highly likely to originate from the same foreign object or the same part of the foreign object. Therefore, by setting the same brightness range identification information for pixels that are highly likely to originate from the same foreign object or the same part of the foreign object and placing them in the same cluster, additional information that the pixels are highly likely to originate from the same foreign object or the same part of the foreign object can be reflected in the machine learning model Mb, thereby enabling highly accurate foreign object detection.

[0068] According to this embodiment, a histogram of brightness values ​​(pixel values) is generated based on the inspection image, and the above-mentioned multiple brightness ranges (pixel value ranges) are determined corresponding to the multiple peaks included in the histogram. This makes it possible to detect foreign objects with high accuracy even when the brightness range may fluctuate depending on the foreign objects present in the food being inspected.

[0069] According to this embodiment, a linear density transformation is performed on the inspection image (filtered inspection image, clustered inspection image) to convert the luminance values ​​belonging to the luminance range thought to originate from a foreign object to a wider range. This increases the sensitivity of the machine learning models Ma and Mb to the luminance range, making it possible to detect foreign objects with higher accuracy.

[0070] According to this embodiment, by performing a reduction process on the inspection image, even if a foreign object is captured at the edge of the inspection image, its position can be moved closer to the center. This improves the accuracy of foreign object detection, even if the machine learning models Ma and Mb are configured to have difficulty detecting foreign objects at the edges.

[0071] (Variation 1) Some of the components of the processing unit 9 shown in Figure 3 may be omitted.

[0072] For example, the clustering unit 93, the second linear concentration conversion unit 94b, the second detection unit 95b, and the second reduction unit 96b may be omitted from the processing device 9. An example of this configuration is shown in Figure 10. In the example configuration of Figure 10, steps S108 to S113 in Figure 4B can be omitted.

[0073] Furthermore, the filter processing unit 92, the first linear concentration conversion unit 94a, the first detection unit 95a, and the first reduction unit 96a may be omitted from the processing unit 9. An example of this configuration is shown in Figure 11. In the example configuration shown in Figure 11, steps S102 to S107 in Figure 4A can be omitted.

[0074] Furthermore, in the configuration of Figure 3 or Figure 10, the filter processing unit 92 and the first linear concentration conversion unit 94a may be omitted. In this case, the processes in steps S102 to S104 of Figure 4A can be omitted. Also, the first reduction unit 96a may be omitted. In this case, the processes in steps S105 to S107 of Figure 4A can be omitted.

[0075] Furthermore, in the configuration of Figure 3 or Figure 11, the clustering unit 93 and the second linear concentration conversion unit 94b may be omitted. In this case, the processes in steps S108 to S110 of Figure 4B can also be omitted. Also, the second reduction unit 96b may be omitted. In this case, the processes in steps S111 to S113 of Figure 4B can be omitted.

[0076] (Modification 2) Figure 12 shows a food manufacturing apparatus 200 equipped with the food inspection apparatus 100 and food removal apparatus 220 shown in Figure 1. The food manufacturing apparatus 200 performs detection of foreign matter contained in food between processing steps of food that is processed through multiple processing steps, and removes the detected foreign matter. The food manufacturing apparatus 200 performs detection and removal of foreign matter on food 1 that is transported on a transport surface S by a transport device 210 (e.g., a belt conveyor) that transports food between any processing steps. The food inspection apparatus 100 irradiates light onto the food 1 being transported on the transport surface S from an upstream processing step to detect foreign matter. The method of detecting foreign matter is the same as in the embodiment described above. When the food inspection apparatus 100 detects a foreign matter 1a in the food 1, it transmits the position information of the foreign matter 1a in the food 1 to the foreign matter removal apparatus 220. The foreign object removal device 220 controls a robot hand 230, which is movable in the X, Y, and Z axes, based on the position information of the foreign object 1a notified by the food inspection device 100, to remove the foreign object 1a from the food 1 being transported on the transport surface S. The foreign object removal device 220 controls the robot hand 230 to collect the removed foreign object 1a into a collection box (not shown). The food 1 from which the foreign object 1a has been removed is transported to the next processing step by the transport device 210. The food inspection device 100 may transmit information to the foreign object removal device 220 indicating that a foreign object 1a has been detected, instead of the position information of the foreign object 1a. In this case, the foreign object removal device 220 may use a sensor such as a camera that images the food 1 to identify the position of the foreign object 1a and remove the foreign object 1a from the food 1. According to this modified example 2, since the food inspection device 100 can detect foreign objects with high accuracy, the yield of food production that is free of foreign objects can be increased, and food waste, which is caused by food contaminated with foreign objects being discarded without being consumed, can be reduced.

[0077] While several embodiments of this disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the disclosure. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations are possible without departing from the spirit of the disclosure. These embodiments and their variations are included in the scope and spirit of the disclosure, as well as in the disclosure described in the claims and its equivalents.

[0078] Furthermore, the effects described herein are merely illustrative, and other effects may also occur.

[0079] Furthermore, this disclosure can also take the following form. [Item 1] An inspection image generation unit generates an inspection image based on the difference between imaging data of a first light obtained by polarizing the reflected light, which is polarized in a third direction and irradiated onto food, in a first direction parallel to the third direction, and imaging data of a second light obtained by polarizing the reflected light in a second direction orthogonal to the first direction. A detection processing unit performs detection processing to detect foreign objects present in the food based on the inspection image and a machine learning model. Equipped with an information processing device. [Item 2] The system further includes a conversion unit that converts pixel values ​​of pixels included in the inspection image that fall outside the range between a first threshold and a second threshold into predetermined pixel values. The predetermined pixel value is less than the first threshold or greater than the second threshold. The detection processing unit performs the detection processing based on the inspection image converted by the conversion unit, as described in item 1. [Item 3] The predetermined pixel value is the minimum or maximum pixel value that the inspection image can take. The information processing device described in item 2. [Item 4] The system further includes a first linear density conversion unit that linearly converts the pixel values ​​included between the first threshold and the second threshold in the inspection image converted by the conversion unit to a pixel value range wider than the range between the first threshold and the second threshold. The detection processing unit performs the detection process based on the linearly density-converted inspection image, as described in item 2 or 3. [Item 5] The system further includes a setting unit that sets a plurality of first pixel value ranges between the minimum and maximum pixel values ​​that the inspection image can take, and sets identification information for pixels belonging to the first pixel value range to identify the first pixel value range to which the pixel belongs. The detection processing unit performs the detection processing based on the inspection image on which the identification information has been set, according to any one of items 1 to 4. [Item 6] The setting unit generates a histogram of pixel values ​​based on the inspection image and sets the multiple first pixel value ranges corresponding to the multiple peaks included in the histogram. The information processing device described in item 5. [Item 7] The system further comprises a second linear density conversion unit that linearly converts the pixel values ​​included in the first pixel value range to a second pixel value range that is wider than the first pixel value range. The detection processing unit performs the detection process based on the linearly density-converted inspection image, as described in item 5 or 6. [Item 8] The system further includes a reduction unit that reduces the size of the inspection image and arranges pixels with predetermined pixel values ​​around the reduced image to generate an image of the same size as the inspection image before reduction. The detection processing unit performs the detection processing based on the image generated by the reduction unit, as described in any one of items 1 to 7. [Item 9] The aforementioned machine learning model is a neural network that takes the pixel values ​​of the inspection image as input and outputs a value regarding the presence or absence of the foreign object. An information processing device as described in any one of items 1 to 8. [Item 10] The light irradiated onto the food is near-infrared light. An information processing device described in any one of items 1 to 9. [Item 11] The aforementioned food is a vegetable; an information processing device as described in any one of items 1 to 10. [Item 12] The information processing device according to item 11, wherein the vegetable is one of broccoli, spinach, or edamame. [Item 13] The foreign substance is an organic substance, as described in any one of items 1 to 12. [Item 14] The aforementioned organic material is an insect, as described in item 13. [Item 15] An inspection image is generated based on the difference between imaging data of a first beam of light, which is obtained by polarizing the reflected light that is irradiated onto the food in a third direction, in a first direction parallel to the third direction, and imaging data of a second beam of light, which is obtained by polarizing the reflected light in a second direction orthogonal to the first direction. Based on the aforementioned inspection images and a machine learning model, foreign objects present in the food are detected. Information processing methods. [Item 16] A light source that emits light, A third polarizer that polarizes the light emitted from the light source in a third direction and transmits the polarized light to the food, A first polarizer that polarizes the reflected light from the food in a first direction parallel to the third direction, A second polarizer that polarizes the reflected light in a second direction perpendicular to the third direction, An imaging unit generates first imaging data by imaging the first light transmitted through the first polarizer, and generates second imaging data by imaging the second light transmitted through the second polarizer, An inspection image generation unit generates an inspection image based on the difference between the first imaging data and the second imaging data, A detection processing unit performs detection processing to detect foreign objects present in the food based on the inspection image and a machine learning model. A food inspection device equipped with the following features. [Item 17] Light emitted from a light source is polarized in a third direction by a third polarizer to produce third light, and this third light is irradiated onto the food being transported on the transport surface. The reflected light of the third light in the food is polarized by a first polarizer in a first direction parallel to the third direction to obtain the first light, and the first light is imaged to generate first imaging data. The reflected light is polarized by a second polarizer in a second direction perpendicular to the third direction to obtain a second beam of light, and the second beam of light is imaged to generate second imaging data. Based on the difference between the first imaging data and the second imaging data, an examination image is generated. Based on the aforementioned inspection images and a machine learning model, foreign objects present in the food are detected. Food production method. [Explanation of symbols]

[0080] 1 food 1a Foreign object 2 flat plate 3 light source 4. Imaging device (imaging unit) 4a lens 5. Polarizer (Third polarizer) 6. Polarizer (First polarizer) 7. Polarizer (Second Polarizer) 8 Guide rails 9 Processing Unit 9a Information Processing Unit (Information Processing Device) 9b Controller 10. Drive unit (drive mechanism) 11. Inspection light 12 Reflected light 13. Input device (input section) 14. Output device (output section) 91 Inspection Image Generation Unit 92 Filter Processing Unit (Conversion Unit) 93. Clustering Unit (Configuration Unit) 94a First linear concentration conversion unit 94b Second linear concentration conversion unit 95 Detection Processing Unit 95a First detection unit 95b Second detection unit 96a First reduced section (reduced section) 96b Second reduced section (reduced section) 97 Judgment section C orthogonal image P parallel image 100 Food Inspection Devices 200 Food manufacturing equipment 210 Conveying device 220 Foreign matter removal device 230 Robot Hand

Claims

1. An inspection image generation unit generates an inspection image based on the difference between imaging data of a first light obtained by polarizing the reflected light of light polarized in a third direction and irradiated onto food in a first direction parallel to the third direction, and imaging data of a second light obtained by polarizing the reflected light in a second direction perpendicular to the first direction. A reduction unit reduces the size of the inspection image and arranges pixels with predetermined pixel values ​​around the reduced image to generate an image of the same size as the inspection image before reduction. A detection processing unit performs a detection process to detect foreign objects present in the food based on the image generated by the reduction unit and a machine learning model. An information processing device equipped with the following features.

2. The system further includes a conversion unit that converts pixel values ​​of pixels included in the inspection image that fall outside the range between a first threshold and a second threshold into predetermined pixel values. The predetermined pixel value is less than the first threshold or greater than the second threshold. The information processing apparatus according to claim 1, wherein the reduction unit reduces the size of the inspection image converted by the conversion unit to generate the image.

3. The information processing apparatus according to claim 2, wherein the predetermined pixel value is the minimum or maximum pixel value that the inspection image can take.

4. The system further includes a first linear density conversion unit that linearly converts the pixel values ​​included between the first threshold and the second threshold in the inspection image converted by the conversion unit to a pixel value range wider than the range between the first threshold and the second threshold. The reduction unit reduces the size of the linearly density-converted inspection image to generate the image, as described in claim 2.

5. The system further includes a setting unit that sets a plurality of first pixel value ranges between the minimum and maximum pixel values ​​that the inspection image can take, and sets identification information for pixels belonging to the first pixel value range to identify the first pixel value range to which the pixel belongs. The reduction unit reduces the size of the inspection image on which the identification information is set, thereby generating the image, as described in claim 1.

6. The information processing apparatus according to claim 5, wherein the setting unit generates a histogram of pixel values ​​based on the inspection image and sets the plurality of first pixel value ranges corresponding to the plurality of peaks included in the histogram.

7. The system further comprises a second linear density conversion unit that linearly converts the pixel values ​​included in the first pixel value range to a second pixel value range that is wider than the first pixel value range. The reduction unit reduces the size of the linearly density-converted inspection image to generate the image, as described in claim 5.

8. The information processing apparatus according to claim 1, wherein the machine learning model is a neural network that takes the pixel values ​​of the image generated by the reduction unit as input and outputs a value regarding the presence or absence of the foreign object.

9. The information processing apparatus according to claim 1, wherein the light irradiated onto the food is near-infrared light.

10. The information processing apparatus according to claim 1, wherein the food is a vegetable.

11. The information processing apparatus according to claim 10, wherein the vegetable is any one of broccoli, spinach, or edamame.

12. The information processing apparatus according to claim 1, wherein the foreign substance is an organic substance.

13. The information processing apparatus according to claim 12, wherein the organic substance is an insect.

14. An inspection image is generated based on the difference between imaging data of a first light, which is obtained by polarizing the reflected light of light polarized in a third direction and irradiated onto the food in a first direction parallel to the third direction, and imaging data of a second light, which is obtained by polarizing the reflected light in a second direction orthogonal to the first direction. The size of the inspection image is reduced, and pixels with predetermined pixel values ​​are placed around the reduced image to generate an image of the same size as the inspection image before reduction. Based on the generated image and a machine learning model, foreign objects present in the food are detected. Information processing methods.

15. A light source that emits light, A third polarizer that polarizes the light emitted from the light source in a third direction to produce a third beam of light, and transmits the third beam of light to the food, A first polarizer that polarizes the reflected light of the third light in the food in a first direction parallel to the third direction, A second polarizer that polarizes the reflected light in a second direction perpendicular to the third direction, An imaging unit generates first imaging data by imaging the first light transmitted through the first polarizer, and generates second imaging data by imaging the second light transmitted through the second polarizer, An inspection image generation unit generates an inspection image based on the difference between the first imaging data and the second imaging data, A reduction unit reduces the size of the inspection image and arranges pixels with predetermined pixel values ​​around the reduced image to generate an image of the same size as the inspection image before reduction. A detection processing unit performs a detection process to detect foreign objects present in the food based on the image generated by the reduction unit and a machine learning model. A food inspection device equipped with the following features.

16. Light emitted from a light source is polarized in a third direction by a third polarizer to produce third light, and this third light is irradiated onto the food being transported on the transport surface. The reflected light of the third light in the food is polarized by a first polarizer in a first direction parallel to the third direction to obtain the first light, and the first light is imaged to generate first imaging data. The reflected light is polarized by a second polarizer in a second direction perpendicular to the third direction to obtain a second beam of light, and the second beam of light is imaged to generate second imaging data. Based on the difference between the first imaging data and the second imaging data, an inspection image is generated. The size of the inspection image is reduced, and pixels with predetermined pixel values ​​are placed around the reduced image to generate an image of the same size as the inspection image before reduction. Based on the generated image and a machine learning model, foreign objects present in the food are detected. Food production method.

Images