Inspection apparatus and inspection method

Abstract

To more facilitate checking of arrangement of food items using image recognition.SOLUTION: A check device 1 comprises: an acquisition unit 10 which acquires an image including arranged food items; a first estimation unit 11 which uses a learned first model NN 1 to estimate regions and categories of the food items in the image, and outputs a first estimation result including an output image associating the estimated regions and categories of food items with the image; a first determination unit 12 which, with respect to each of the categories of food items, determines whether or not the corresponding region satisfies a first criterion preliminarily set about the presence / absence and excess / deficiency of food items on the basis of the first estimation result; and a check unit 13 which checks arrangement of the arranged food items on the basis of the determination result of the first determination unit 12.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an inspection apparatus and an inspection method, and particularly to a food plating inspection technique using image recognition.

Background Art

[0002] As quality control of foods manufactured on a production line or the like, appearance inspections such as food plating inspections and ingredient omission inspections are known. Conventionally, these inspections have been performed visually. In recent years, with the development of image recognition technology, technologies that utilize image recognition for quality control of manufactured foods have also been developed in the field of food manufacturing.

[0003] For example, Patent Document 1 discloses a technique for inspecting a food to be inspected by taking images of the food at two different wavelengths, performing spectral analysis on each pixel to generate an inspection image, and inspecting the state of the food to be inspected.

[0004] However, in actual food manufacturing sites, quality inspections such as food plating inspections are still often performed visually. The reasons include that the ingredients to be plated on foods often have irregular shapes, and even the same ingredient may have different shapes, and the color and taste of the ingredients may not be constant. Therefore, it is often difficult to always perform the food plating inspection, which includes confirmation that the determined ingredients are plated on the manufactured food, the number, amount, and plating position of each ingredient, etc., according to the same standard. For this reason, the creation of the detection logic itself for food plating inspection using image recognition can be difficult.

[0005] Furthermore, in recent years, in response to the diversification of consumer needs, multi-variety small-lot production and shortening of product life cycles have advanced. Even if the detection logic is once created, the range of use may be limited, and it may be difficult to use food plating inspection using image recognition technology as a practical method.

Prior Art Documents

Non-Patent Documents

[0006] [Non-Patent Document 1] Shorten, C., Khoshgoftaar, T.M. “A survey on Image Data Augmentation for Deep Learning.” J Big Data 6, 60 (2019). https: / / doi.org / 10.1186 / s40537-019-0197-0 [Non-Patent Document 2] Khalifa, Nour Eldeen & Loey, Mohamed & Mirjalili, Seyedali. (2021). “A comprehensive survey of recent trends in deep learning for digital images augmentation.” Artificial Intelligence Review. 10.1007 / s10462-021-10066-4. [Non-Patent Document 3] Naveed, Humza. (2021). “Survey: Image Mixing and Deleting for Data Augmentation.” [Patent Document]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2019-74324 [Patent Document 2] Japanese Patent No. 6955734 [Patent Document 3] Japanese Patent No. 6929322 [Summary of the Invention] [Problems to be Solved by the Invention]

[0008] According to the conventional technology, it has been difficult to easily perform a food plating inspection using image recognition.

[0009] The present invention has been made to solve the above-described problems, and an object thereof is to more easily perform a food plating inspection using image recognition.

Means for Solving the Problems

[0010] In order to solve the above-described problems, an inspection apparatus according to the present invention includes an acquisition unit that acquires an image including arranged foodstuffs, and uses a learned first model to estimate a region and a type of the foodstuffs in the image, and outputs a first estimation result including an output image in which the estimated region and type of the foodstuffs are associated with the image; a first determination unit that determines, for each type of the foodstuffs, whether or not a corresponding region satisfies a first criterion preset with respect to the presence, absence, excess, and deficiency of the foodstuffs based on the first estimation result; and an inspection unit that performs an inspection regarding the arrangement of the foodstuffs based on a determination result by the first determination unit.

[0011] Further, in the inspection apparatus according to the present invention, the first determination unit may specify at least one of the number and size of the regions for each type of the foodstuffs, and determine whether or not at least one of the number and size of the regions satisfies the first criterion.

[0012] Further, in the inspection apparatus according to the present invention, a learned second model that uses the output image included in the first estimation result as an input is used to estimate a relative arrangement relationship between a plurality of the foodstuffs in the output image, and a second estimation result in which the estimated arrangement relationship is associated with the output image is output; a second determination unit that determines whether or not the arrangement relationship satisfies a second criterion preset with respect to the validity of the arrangement relationship based on the second estimation result; and the inspection unit may perform an inspection regarding the arrangement of the foodstuffs based on a determination result by the second determination unit.

[0013] In addition, in the inspection apparatus according to the present invention, further provided is a presentation unit that presents at least the inspection result by the inspection unit, and the presentation unit may present at least any one of the information including the image, the first estimation result, the determination result by the first determination unit, the second estimation result, the determination result by the second determination unit, and the inspection result by the inspection unit.

[0014] In addition, in the inspection apparatus according to the present invention, further provided is a learning device that performs a learning process for constructing a preset model. The learning device uses an image including the arranged food material as a first learning image, learns a first model of a neural network, extracts a first feature amount for identifying the region and type of the food material, and obtains the learned first model based on the extracted first feature amount. The first learning unit includes a first storage unit that stores the learned first model obtained by the first learning unit. The first estimation unit may read the learned first model from the first storage unit and estimate the region and type of the food material in the image by the learned first model.

[0015] In addition, in the inspection apparatus according to the present invention, the first model may be an image recognition model by instance segmentation.

[0016] In addition, in the inspection apparatus according to the present invention, the learning device may further include a second learning unit that uses the output image included in the first estimation result output by the first estimation unit as a second learning image, learns a second model of a neural network, extracts a second feature amount for identifying a relative arrangement relationship between a plurality of the food materials in the second learning image, and obtains the learned second model based on the extracted second feature amount, and a second storage unit that stores the learned second model obtained by the second learning unit.

[0017] In addition, in the inspection apparatus according to the present invention, further comprising a conversion unit that converts the output image included in the first estimation result to generate a more simplified converted image, and the second learning unit may use the converted image as the second learning image to learn the second model and obtain the learned second model.

[0018] In addition, in the inspection apparatus according to the present invention, the output image included in the first estimation result is a color pattern image in which the corresponding region is painted with an arbitrary color for each type of food material, and the converted image may be a color pattern image in which each region of the color pattern image is replaced with an object of the same shape.

[0019] In addition, in the inspection apparatus according to the present invention, the output image included in the first estimation result is a color pattern image in which the corresponding region is painted with an arbitrary color for each type of food material, and the converted image may be an image in which each region of the color pattern image is replaced with an object having a different shape for each color of the color pattern, and the color of the replaced object is set to a different color for each shape.

[0020] In addition, in the inspection apparatus according to the present invention, the output image included in the first estimation result is a color pattern image in which the corresponding region is painted with an arbitrary color for each type of food material, and the converted image may be an image in which each region of the color pattern image is replaced with an object having a different shape for each color of the color pattern, and all the colors of the replaced object of the color pattern are set to the same color.

[0021] In addition, in the inspection apparatus according to the present invention, further comprising a processing unit that processes the output image on the assumption that the inside of the longest estimated outer contour line of the region of the output image included in the first estimation result is the region, and the second learning unit may use the processed image as the second learning image to learn the second model and obtain the learned second model.

[0022] In addition, in the inspection apparatus according to the present invention, the learning apparatus further includes a data augmentation unit that performs data augmentation on the target image in which the region and type of the food ingredients are associated with the image to generate the second learning image, and the data augmentation unit may generate the second learning image based on label information regarding the validity of the relative arrangement relationship among the plurality of food ingredients in the target image, which is attached to the target image.

[0023] In addition, in the inspection apparatus according to the present invention, the data augmentation unit may perform data augmentation on the target image to which a first label indicating that the relative arrangement relationship among the plurality of food ingredients in the target image is valid is attached, and generate the second learning image to which the first label is attached.

[0024] In addition, in the inspection apparatus according to the present invention, the data augmentation unit may perform data augmentation on the target image to which a first label indicating that the relative arrangement relationship among the plurality of food ingredients in the target image is valid is attached, and generate the second learning image to which a second label indicating that the relative arrangement relationship among the plurality of food ingredients is not valid is attached.

[0025] In addition, in the inspection apparatus according to the present invention, the data augmentation unit may perform data augmentation on the target image to which a second label indicating that the relative arrangement relationship among the plurality of food ingredients is not valid is attached, and generate the second learning image to which the second label is attached.

[0026] In addition, in the inspection apparatus according to the present invention, it further includes an extraction unit that extracts the region for each type of the food ingredients from the target image, and the data augmentation unit may generate the second learning image by processing and repositioning the region extracted by the extraction unit while maintaining the relative arrangement relationship among the plurality of food ingredients in the target image.

[0027] Further, in the inspection apparatus according to the present invention, an extraction unit for extracting the regions for each type of the food material from the target image is further provided, and the data augmentation unit may generate the second learning image by replacing the positions of regions corresponding to two or more different types of the food material among the regions extracted by the extraction unit with each other.

[0028] Further, in the inspection apparatus according to the present invention, an extraction unit for extracting the regions for each type of the food material from the target image is further provided, and the data augmentation unit may generate the second learning image by deleting one or more of the regions extracted by the extraction unit from the target image.

[0029] Further, in the inspection apparatus according to the present invention, an extraction unit for extracting the regions for each type of the food material from the target image is further provided, and the data augmentation unit may perform at least any one of inversion, rotation, dilation, and contraction of each of the regions extracted by the extraction unit to perform data augmentation of the target image.

[0030] Further, in order to solve the above-described problems, an inspection apparatus according to the present invention includes an acquisition unit that acquires an image including an arranged food material, and uses a learned first model to detect a region of the food material in the image with a bounding box, and estimates a relative arrangement relationship between a plurality of the food materials and a type of the food material, and outputs a first estimation result including an output image in which the estimated arrangement relationship is associated with the image; a first determination unit that determines whether or not the arrangement relationship satisfies a preset criterion based on the first estimation result; and an inspection unit that performs an inspection regarding the arrangement of the food material based on a determination result by the first determination unit.

[0031] Further, in order to solve the above-described problems, a food inspected using the inspection apparatus according to the present invention is manufactured on a food production line and includes processed noodles, frozen foods, box lunches, and prepared foods in which the food materials are arranged and served.

[0032] In addition, in order to solve the above-described problems, the inspection method according to the present invention includes: a first step of acquiring an image including an arranged food ingredient; a second step of using a learned first model to estimate the region and type of the food ingredient in the image, and outputting a first estimation result including an output image in which the estimated region and type of the food ingredient are associated with the image; a third step of, based on the first estimation result, determining, for each type of the food ingredient, whether a corresponding region satisfies a first criterion preset with respect to the presence, absence, and excess or deficiency of the food ingredient; and a fourth step of performing an inspection regarding the arrangement of the food ingredient based on the determination result in the third step.

[0033] Further, in the inspection method according to the present invention, in the third step, for each type of the food ingredient, at least one of the number and size of the regions is specified, and it is determined whether at least one of the number and size of the regions satisfies the first criterion. In the fourth step, an inspection regarding the arrangement of the food ingredient may be performed based on the determination result.

[0034] Further, in the inspection method according to the present invention, a fifth step of using a learned second model that takes the output image included in the first estimation result as an input to estimate a relative arrangement relationship between a plurality of the food ingredients in the output image, and outputting a second estimation result in which the estimated arrangement relationship is associated with the output image; and a sixth step of, based on the second estimation result, determining whether the arrangement relationship satisfies a second criterion preset with respect to the validity of the arrangement relationship are further provided. The fourth step may perform an inspection regarding the arrangement of the food ingredient based on the determination result in the sixth step.

[0035] Further, in the inspection method according to the present invention, a learning step of performing a learning process for constructing a preset model is further provided. The learning step uses an image including the arranged food material as a first learning image, learns a first model of a neural network, extracts a first feature amount for identifying the region and type of the food material, and obtains the learned first model based on the extracted first feature amount. A seventh step; and an eighth step of storing the learned first model obtained in the seventh step in a first storage unit. The third step may read the learned first model from the first storage unit and estimate the region and type of the food material in the image by the learned first model.

[0036] Further, in the inspection method according to the present invention, the learning step uses the output image included in the first estimation result output in the third step as a second learning image, learns a second model of a neural network, and extracts a second feature amount for identifying the relative arrangement relationship between a plurality of the food materials in the second learning image. A ninth step of obtaining the learned second model based on the extracted second feature amount; and a tenth step of storing the learned second model obtained in the ninth step in a second storage unit may be further provided.

[0037] Further, in the inspection method according to the present invention, an eleventh step of generating a processed image obtained by processing the output image is further provided, assuming that the inside of the longest estimated outer contour line of the region of the output image included in the first estimation result is the region. The ninth step may learn the second model using the processed image as the second learning image and obtain the learned second model.

[0038] Further, in the inspection method according to the present invention, the learning step further includes a 12th step of performing data augmentation on a target image in which the region and type of the foodstuff are associated with the image to generate the second learning image, and the 12th step may generate the second learning image based on label information regarding the validity of the relative arrangement relationship among a plurality of the foodstuffs in the target image, which is attached to the target image.

[0039] Further, in the inspection method according to the present invention, the 12th step may perform data augmentation on a target image to which a first label indicating that the relative arrangement relationship among a plurality of the foodstuffs in the target image is valid is attached, and generate the second learning image to which the first label is attached.

[0040] Further, in the inspection method according to the present invention, the 12th step may perform data augmentation on a target image to which a first label indicating that the relative arrangement relationship among a plurality of the foodstuffs in the target image is valid is attached, and generate the second learning image to which a second label indicating that the relative arrangement relationship among the plurality of the foodstuffs is not valid is attached.

[0041] Further, in the inspection method according to the present invention, the 12th step may perform data augmentation on a target image to which a second label indicating that the relative arrangement relationship among a plurality of the foodstuffs is not valid is attached, and generate the second learning image to which the second label is attached.

[0042] Further, in the inspection method according to the present invention, it further includes a 13th step of extracting the region for each type of the foodstuff from the target image, and the 12th step may generate the second learning image by processing and then rearranging the region extracted in the 13th step while maintaining the relative arrangement relationship among a plurality of the foodstuffs in the target image.

[0043] In addition, in the inspection method according to the present invention, further, a 13th step of extracting the regions for each type of food material from the target image is provided, and in the 12th step, the positions of the regions corresponding to two or more different types of food materials among the regions extracted in the 13th step may be replaced with each other to generate the second learning image.

[0044] In addition, in the inspection method according to the present invention, further, a 13th step of extracting the regions for each type of food material from the target image is provided, and in the 12th step, one or more of the regions extracted in the 13th step may be deleted from the target image to generate the second learning image.

[0045] In addition, in the inspection method according to the present invention, further, a 13th step of extracting the regions for each type of food material from the target image is provided, and in the 12th step, at least any one of inversion, rotation, dilation, and contraction of each of the regions extracted in the 13th step may be performed to perform data augmentation of the target image.

[0046] In addition, in the inspection method according to the present invention, further, a 14th step of presenting at least the inspection result in the 4th step is provided, and the 14th step may present at least any one of information including the image, the first estimation result, the determination result in the 3rd step, and the inspection result in the 4th step.

Advantages of the Invention

[0047] According to the present invention, using a learned first model, the regions and types of food materials in an image including the arranged food materials are estimated, and based on the estimated first estimation result, for each type of food material, it is determined whether the corresponding region satisfies a first criterion preset regarding the presence, absence, and excess or deficiency of the food material. Therefore, it is possible to more easily perform a food plating inspection using image recognition.

Brief Description of the Drawings

[0048]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

[0049] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to FIGS. 1 to 25. Further, in the following description, the inspection target of the inspection apparatus according to the present invention is "arranged food materials", and the food materials constitute, for example, "food" manufactured on a food production line, and the number of food materials includes both one and a plurality. Further, the "inspection regarding the arrangement of food materials" may include one or more inspection items such as the type, presence or absence, number, and quantity of each of a plurality of food materials manufactured on a food production line and arranged in a container, for example, and includes an inspection regarding the arrangement position which is the place and the arrangement order of the food materials, and these may be collectively referred to as "food plating inspection". In addition, the arrangement order of the food materials shall include the degree of dispersion of the food materials.

[0050] [First Embodiment] FIG. 1 is a block diagram showing the configuration of an inspection apparatus 1 according to the first embodiment of the present invention. The inspection apparatus 1 according to the present embodiment estimates the regions and types of a plurality of food ingredients in an image including the plurality of food ingredients arranged in a food container or the like using a learned first model NN1 that has been subjected to learning processing in advance by an external server or the like. Further, based on the estimated result (first estimation result), the inspection apparatus 1 determines whether or not the corresponding region satisfies a first criterion preset for the presence, absence, and excess or deficiency of each of the plurality of food ingredients for each type of the plurality of food ingredients. Further, the inspection apparatus 1 performs an inspection regarding the arrangement of the food ingredients arranged in the container or the like based on the determination result. In the present embodiment, as an example, a case where instance segmentation that discriminates regions in an image where food ingredients exist by distinguishing the food ingredients from each other in pixel units is used as the learned first model NN1 will be described.

[0051] [Functional Blocks of Inspection Apparatus] The inspection apparatus 1 according to the present embodiment includes an acquisition unit 10, a first estimation unit 11, a first determination unit 12, an inspection unit 13, a presentation unit 14, and a memory 15.

[0052] The acquisition unit 10 acquires an image including the arranged food ingredients. For example, the acquisition unit 10 can acquire an image including a plurality of food ingredients arranged in a food container or the like. The acquisition unit 10 acquires, for example, a still image or a moving image of food taken by a camera 105 installed outside, either wired or wirelessly. In the present embodiment, the camera 105 is provided in a photographing environment pre-constructed on a food production line and photographs an image including the food ingredients arranged in a container or the like (hereinafter sometimes referred to as a "food image"). In particular, the camera 105 is fixedly installed in the photographing environment in the production line and photographs an image with a fixed angle of view. The acquisition unit 10 can perform known preprocessing such as adjusting the brightness of the food image acquired from the camera 105 and removing noise.

[0053] The first estimation unit 11 estimates the regions and types of a plurality of food ingredients in the image acquired by the acquisition unit 10 using the learned first model NN1, and outputs a first estimation result including an output image by associating the estimated regions and types of the plurality of food ingredients with the image acquired by the acquisition unit 10.

[0054] More specifically, the first estimation unit 11 gives the image acquired by the acquisition unit 10 to the learned first model NN1, performs the operation of the learned first model NN1, and for each pixel in the input image, extracts the type of food ingredient which is the class of the food ingredient and the region of the food ingredient instance which is the entity or individual of each food ingredient actually existing in the image.

[0055] The first estimation unit 11 reads the learned first model NN1 stored in the first memory 15A included in the memory 15 described later and executes the estimation process.

[0056] Here, the estimation process by the first estimation unit 11 will be described in more detail with reference to FIGS. 4 and 5. For example, the image i shown in FIGS. 4 and 5 shows an image of a plurality of food ingredients arranged in a food container acquired by the acquisition unit 10. For example, as toppings for the food "cold Chinese noodles", four types of food ingredients "a", "b", "c", and "d" are arranged. More specifically, for the food "cold Chinese noodles", for example, predetermined topping ingredients such as "ham" as the food ingredient "a", "egg" as the food ingredient "b", "cucumber" as the food ingredient "c", and "tomato" as the food ingredient "d" are arranged on top of the cold Chinese noodles in the container and served. Also, the quantity and amount of each of these food ingredients are determined, and it is preferable that the quantity and amount of the food ingredients of each product manufactured on the production line are constant.

[0057] The output image p shown in FIGS. 4 and 5 is an image in which the regions of each food ingredient instance included in the image i are extracted, the types "a", "b", "c", "d" which are the classes of the food ingredients are estimated, and which is associated with the image i.

[0058] Further, as shown in FIG. 4, the first estimation unit 11 performs an estimation process using the image i as an input, and outputs an image p in which regions "a1", "b1", "c1", and "d1" for each food ingredient instance are shown in a color pattern. For example, the type (class) of food ingredient is distinguished by a color pattern of an arbitrary color. When there are a plurality of regions of food ingredient instances of the same type, these plurality of regions are shown in, for example, a color pattern of the same color. The first estimation unit 11 can detect the number of each type of food ingredient based on the number of regions of food ingredients shown in a color pattern of the same color or the same pattern. When the number of regions in a specific type of food ingredient is zero, it indicates that the specific food ingredient does not exist.

[0059] Returning to FIG. 1, the first determination unit 12 determines, for each type of a plurality of food ingredients, whether the corresponding region satisfies a first criterion preset regarding the presence / absence and excess / deficiency of the food ingredient based on the first estimation result by the first estimation unit 11.

[0060] For example, the first determination unit 12 can specify at least one of the number and size of the regions for each type of a plurality of food ingredients, and determine whether at least one of the number and size of the regions satisfies the first criterion.

[0061] The first criterion is set as a criterion for determining whether a determined food ingredient exists, whether the food ingredient is arranged in a determined number, and whether the food ingredient exists in a determined amount.

[0062] The first criterion is set based on characteristics such as the shape and size of each type of food ingredient so that it is possible to more appropriately determine the presence / absence and excess / deficiency of the food ingredient. Therefore, when it is preferable to determine the excess / deficiency only by the number of food ingredients, a reference value for the number can be used. Alternatively, when it is more appropriate to determine the excess / deficiency by the size of the region of the food ingredient, a reference value related to the size of the region can be used. Similarly, it can also be determined by a combination of these. Thus, the first criterion may include a plurality of criteria.

[0063] For example, as the size of the area for each type of food ingredient, the first determination unit 12 can use, as the first criterion, the area value where the absolute area of the area of the food ingredient extracted in the image corresponds to the amount of the food ingredient. Note that the area value can be expressed by the number of pixels of the area in the image. Alternatively, as will be described later, the first criterion can also be set by using the total area of the estimated areas for each type of food ingredient as the amount of that food ingredient.

[0064] Also, when one type of food ingredient is composed of a plurality of food ingredients, the first determination unit 12 can use, as the first criterion, whether the area of the whole of one type of food ingredient including each area exceeds a certain area, and can determine whether there is an excess or deficiency of the same type of food ingredient. For example, there is shredded cheese etc. arranged in a certain amount within a certain range as pizza topping. For example, when the area surrounding the area of each shredded cheese exceeds a certain area and the number of shredded cheeses exceeds a certain number, the first determination unit 12 can determine that the food ingredient cheese satisfies the first criterion and there is no excess or deficiency of the food ingredient cheese.

[0065] For example, as shown in region B of FIG. 5, as the first criterion set in advance, a threshold value for the number of food ingredients set in advance for each type of food ingredient and a threshold value for the area in the area of the food ingredient occupying the image can be used. For example, in the example of food ingredient a in FIG. 5, the "estimated number" estimated by the first estimation unit 11 is "2", and the "estimated area", which is the absolute area of region a1 in food ingredient a, is "100" pixels. Also, the first criterion "number threshold value" set for the number is "3", and the first criterion "area threshold value" set for the area is in the range of "110 - 200" pixels.

[0066] The first determination unit 12 determines whether the number of each of the estimated food ingredients satisfies a first standard, and whether the area of the region of each of the estimated food ingredients satisfies the first standard. In the example of FIG. 5, for the number of the estimated food ingredient a, the first determination unit 12 determines that it does not satisfy "3", which is the "number threshold" as shown in the item "number determination" ( "NG"). Further, for the area of the region a1 in the estimated food ingredient a, the first determination unit 12 determines that it does not satisfy the range of "110 - 200" pixels, which is the "area threshold" shown in the item "area determination" ( "NG").

[0067] Also, as shown in the "inspection item" shown in region B of FIG. 5, for example, the first determination unit 12 determines whether the number of the food ingredient a satisfies the first standard and whether the area of the region a1 satisfies the first standard. For example, for the food ingredients b and c, only the areas of the regions b1 and c1 are considered, and for the food ingredient d, only the number is considered in the determination of the presence, absence, excess, and deficiency of the food ingredient.

[0068] Based on the determination result of the first determination unit 12, the inspection unit 13 inspects the arrangement of a plurality of food ingredients arranged in a container or the like. For example, for all types of food ingredients "a", "b", "c", and "d" arranged in a food container or the like, when the determination by the first determination unit 12 regarding their presence, absence, excess, and deficiency is affirmative, the inspection unit 13 can output an inspection result indicating that the inspection regarding the arrangement of the food ingredients is qualified. On the other hand, when a negative determination result is obtained for even one of the plurality of types of food ingredients, the inspection unit 13 can output an inspection result indicating that the inspection regarding the arrangement of the food ingredients is unqualified.

[0069] For example, in the item "inspection result" in region B of FIG. 5, since the determination results regarding the presence, absence, excess, and deficiency of the food ingredients related to the food ingredients a and c among the food ingredients a to d do not satisfy the first standard, the final inspection result regarding the arrangement of the food ingredients, that is, the food plating inspection result, is unqualified "NG".

[0070] The prompting unit 14 presents at least the inspection results by the inspection unit 13. For example, the prompting unit 14 can include a display device 108 that displays the inspection results. As shown in FIG. 5, for example, in area A of the display screen of the display device 108, the prompting unit 14 can display an image i of a plurality of food ingredients captured by the camera 105, an output image p indicating the first estimation result by the first estimation unit 11, and an image s obtained by synthesizing the image i and the output image p in which the estimation result is distinguished by a color pattern.

[0071] In addition, the prompting unit 14 can display values corresponding to the respective items of "Variety: Cold Chinese Noodles", "Type of Ingredients", "Estimated Quantity", "Estimated Area", "Quantity Threshold", "Area Threshold", "Quantity Judgment", "Area Judgment", "Inspection Items", "Judgment Result", and "Inspection Result" of the food in area B provided on the display screen of the display device 108.

[0072] Note that as long as the prompting unit 14 can present the inspection results by the inspection unit 13 to the user of the inspection device 1, it can present information not only in the form of a screen display but also in other forms such as voice.

[0073] The memory 15 stores an image including food ingredients arranged on a food container or the like, which is acquired by the acquisition unit 10 from the camera 105. The memory 15 also stores the first estimation result by the first estimation unit 11, the preset first criterion used by the first determination unit 12, and the like.

[0074] The memory 15 includes a first memory 15A. The first memory 15A stores a learned first model NN1 which is an image recognition model by instance segmentation pre-trained and constructed on an external server or the like, and a first criterion such as threshold information used for judgment using the result thereof. For example, the first memory 15A can store in advance the learned first model NN1 for each food variety. In this way, by storing in advance the learned first model NN1 and the first criterion corresponding to each of a plurality of foods, even when different foods are manufactured on the same production line, by switching the learned first model NN1 used in the inspection device 1, it is possible to cope with the packaging inspection of different foods.

[0075] [Hardware Configuration of Inspection Device] Next, an example of the hardware configuration for realizing the inspection device 1 according to the present embodiment will be described with reference to the block diagram of FIG. 2.

[0076] As shown in FIG. 2, the inspection device 1 can be realized by, for example, a computer including a processor 102, a main storage device 103, a communication interface (I / F) 104, an auxiliary storage device 106, and an input / output I / O 107 connected via a bus 101, and a program for controlling these hardware resources. Further, the inspection device 1 is connected to an external camera 105 via the bus 101. Further, the inspection device 1 can include a display device 108 connected via the bus 101.

[0077] In the main storage device 103, programs for the processor 102 to perform various controls and operations are stored in advance. The functions of the inspection device 1 such as the acquisition unit 10, the first estimation unit 11, the first judgment unit 12, and the inspection unit 13 shown in FIG. 1 are realized by the processor 102 and the main storage device 103.

[0078] The communication I / F 104 is an interface circuit for network-connecting between the inspection device 1 and various external electronic devices. The acquisition unit 10 and the presentation unit 14 shown in FIG. 1 are realized by the communication I / F 104. Note that, from the communication I / F 104, the learned first model NN1 used by the first estimation unit 11 described in FIG. 1 and threshold information and the like used for the determination using the result thereof may be received from an external terminal via the network NW, or may be acquired using a storage medium such as a memory card. Further, from the communication I / F 104, the preset criteria used by the first determination unit 12 and the inspection unit 13 described in FIG. 1 may be received from an external terminal via the network NW, or may be acquired using a storage medium such as a memory card.

[0079] The camera 105 can convert an optical signal into an image signal to generate a moving image or a still image. A moving image or a still image including food ingredients arranged on a food container or the like, photographed by the camera 105, is used as an input image of the learned first model NN1, and the type of each food ingredient at each pixel level in the image by instance segmentation is detected for each instance by the processor 102.

[0080] The auxiliary storage device 106 is composed of a readable / writable storage medium and a driving device for reading and writing various information such as programs and data to and from the storage medium. As the storage medium, a semiconductor memory such as a hard disk or a flash memory can be used for the auxiliary storage device 106.

[0081] The auxiliary storage device 106 has a program storage area for storing the inspection program executed by the inspection device 1. The auxiliary storage device 106 also has an area for storing programs for performing estimation processing such as detection of food materials by instance segmentation, determination processing based on the estimation results, and inspection processing related to food plating. The memory 15 and the first memory 15A described in FIG. 1 are realized by the auxiliary storage device 106. Further, the auxiliary storage device 106 stores the learned first model NN1 and the first criteria such as threshold information used for determination using the results thereof. Furthermore, for example, it may have a backup area for backing up the above-described data, programs, and the like.

[0082] The input / output I / O 107 is composed of I / O terminals for inputting signals from external devices and outputting signals to external devices.

[0083] The display device 108 is composed of a liquid crystal display or the like. The presentation unit 14 described in FIG. 1 can also be realized by the display device 108.

[0084] Here, the program stored in the program storage area of the auxiliary storage device 106 may be a program in which processing is performed in time series in accordance with the order of the inspection method described in this specification, or may be a program in which processing is performed in parallel or at a necessary timing such as when a call is made. The program may be processed by one computer or may be distributedly processed by a plurality of computers.

[0085] [Operation of the Inspection Device] Next, the operation of the inspection device 1 having the above-described configuration will be described in detail with reference to the flowchart of FIG. 3 and the explanatory diagram of FIG. 4. The first memory 15A stores the learned first model NN1 obtained by learning processing performed in an external server (not shown) in advance and the first criteria such as threshold information used for determination using the results thereof, and the following processing is executed.

[0086] First, when the inspection starts, the first estimation unit 11 loads the learned first model NN1 from the first memory 15A (step S1). For example, the learned first model NN1 loaded in step S1 has an architecture of instance segmentation represented by Mask R-CNN, DeepMask, FCIS, etc. For example, as shown in FIG. 4, the learned first model NN1 identifies the food ingredients a, b, c, d arranged in a food container or the like for each pixel, estimates the ingredient instances included in the input image i, extracts the regions a1, b1, c1, d1 of the ingredient instances, and is a model in which features (first features) for estimating the food ingredients a, b, c, d for each region are learned.

[0087] Next, the acquisition unit 10 acquires an image including a plurality of food ingredients photographed by the camera 105 (step S2). As shown in FIG. 4, the image i acquired in step S2 is, for example, an image photographed from above so that the states of the food ingredients a, b, c, d arranged or served in a food container manufactured on the production line can be more easily grasped.

[0088] Next, the first estimation unit 11 performs the operation of the learned first model NN1 loaded in step S1 with the image acquired in the second step as an input (step S3). Next, the first estimation unit 11 outputs a first estimation result that is the operation result of the learned first model NN1 (step S4).

[0089] Specifically, the first estimation unit 11 estimates the regions a1, b1, c1, d1 of the plurality of food ingredients and the types a, b, c, d of the food ingredients in the image acquired in step S2 using the learned first model NN1, and outputs a first estimation result including an output image p in which the estimated regions and types of the plurality of food ingredients are associated with the input image i. The output image p is a color pattern image in which the corresponding regions are painted in arbitrary colors for each type of the plurality of food ingredients.

[0090] That is, the output image included in the first estimation result includes pixel information and position information within the image. In the present embodiment, since such instance segmentation is adopted as the first model NN1, even when a plurality of the same type of food ingredients are included in the image, it is possible to detect the food ingredients by distinguishing them for each instance.

[0091] Next, based on the first estimation result output in step S4, the first determination unit 12 determines whether or not the corresponding region satisfies a first criterion preset with respect to the presence / absence and excess / deficiency of food ingredients for each type of the plurality of food ingredients (step S5). For example, the first determination unit 12 can perform threshold processing on the estimated number of each food ingredient, that is, the number of regions or color patterns of the same type of food ingredient. Further, the first determination unit 12 can perform threshold processing on the area occupied by the region of the food ingredient estimated for each type of food ingredient within the image.

[0092] Next, based on the determination results regarding the presence / absence and excess / deficiency of each food ingredient obtained in step S5, the inspection unit 13 inspects the arrangement of the plurality of food ingredients arranged in a food container or the like (step S6). For example, in step S5, when all types of food ingredients satisfy the first criteria related to the number and area, the inspection unit 13 can output an inspection result indicating that the inspection regarding the arrangement of the plurality of food ingredients arranged in a food container or the like is passed. As described above, in the present embodiment, the inspection regarding the arrangement of the food ingredients arranged in a container or the like includes a food plating inspection regarding the type, presence / absence, number, quantity, etc. of each food ingredient.

[0093] Next, the presentation unit 14 presents the inspection result obtained in step S6 (step S7). For example, as shown in FIG. 5, the presentation unit 14 can display in a table format the estimated number and estimated area for each of the foodstuffs a, b, c, and d calculated in step S4 in area B of the display screen of the display device 108. Further, the presentation unit 14 can display, in the table provided in area B, values indicating the determination results in step S5 regarding the number and area. Also, as described above, the presentation unit 14 arranges and displays in area A the original image i, the output image p in which the areas of the foodstuff instances are distinguished by color patterns, and the image s obtained by synthesizing the original image i and the output image p. In this way, by arranging and displaying the input image and the output image together with the inspection result, and further the image obtained by synthesizing these, the user can more intuitively grasp the result of the food plating inspection.

[0094] As described above, according to the first embodiment, based on an image including foodstuffs arranged in a food container or the like, the type of foodstuff is estimated for each pixel in the image, and the area is extracted for each instance of the foodstuff. Therefore, it is possible to more easily perform a food plating inspection using image recognition.

[0095] Also, according to the first embodiment, among image recognition techniques, an object detection method related to instance segmentation is adopted. Therefore, even when the colors, shades, brightness, etc. of individual foodstuffs are different and the shapes are not constant, it is possible to easily estimate the type, number, and amount of foodstuffs for each instance in the image.

[0096] Also, according to the first embodiment, when photographing an image of food ingredients arranged in a food container or the like, it is less necessary to construct an individual photographing environment according to the type of food, and the inspection apparatus 1 and the camera 105 have a simpler configuration, so they can be easily incorporated into a food production line. Further, even when different foods are produced on the same production line, inspection can be performed by switching and using the learned first model NN1 prepared for each food. Furthermore, by the inspection apparatus 1 performing the food plating inspection that has conventionally been performed visually, labor savings can be contributed to.

[0097] In the described embodiment, the case where the camera 105 is installed outside has been described, but the acquisition unit 10 may be configured to include a camera.

[0098] Also, in the described embodiment, the case where all the functional units are provided in one inspection apparatus 1 has been described. However, each functional unit included in the inspection apparatus 1 can also be configured to be distributed on a network, in addition to the case where it is realized as one computer.

[0099] [Second Embodiment] Next, a second embodiment of the present invention will be described. In the following description, the same components as those in the above-described first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0100] In the first embodiment, the case where it is determined whether or not a first criterion preset regarding the presence, absence, excess, and deficiency for each type of a plurality of food ingredients arranged in a food container or the like is satisfied using the learned first model NN1 has been described. In addition to this, in the second embodiment, further, a learned second model NN2 is used to estimate the relative arrangement relationship between a plurality of types of food ingredients.

[0101] [Functional Blocks of Inspection Apparatus] FIG. 6 is a functional block diagram showing the configuration of the inspection apparatus 1A according to the present embodiment. The inspection apparatus 1A according to the present embodiment includes an acquisition unit 10, a first estimation unit 11, a first determination unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, and a second determination unit 17. It is different from the inspection apparatus 1 according to the first embodiment in that it further includes a second memory 15B, a second estimation unit 16, and a second determination unit 17. Hereinafter, the description will focus on the configuration different from the first embodiment.

[0102] The second memory 15B stores a learned second model NN2 constructed by prior learning processing in an external server or the like. More specifically, the second memory 15B stores the learned second model NN2 and a second criterion described later, which is a criterion for the result of the operation of the learned second model NN2.

[0103] Here, the learned second model NN2 is a model in which a feature amount (second feature amount) for identifying the relative arrangement relationship between a plurality of types of foodstuffs to be inspected is learned, and may be a neural network model such as machine learning, particularly deep learning. By using such a learned second model NN2, it is possible to classify the arrangement positions of the foodstuffs in an image of an unknown food. In the present embodiment, two learning models, the learned first model NN1 stored in the first memory 15A and the learned second model NN2 stored in the second memory 15B, are used in pairs for the arrangement inspection of the same item of food.

[0104] FIG. 7 is a diagram for explaining an outline of an estimation process regarding the arrangement position of foodstuffs using the learned second model NN2 performed by the second estimation unit 16 and the second determination unit 17 according to the present embodiment. The images i shown on the left side of FIGS. 7(a) and 7(b) are examples of images of foodstuffs acquired by the acquisition unit 10. For example, assume that the position between the foodstuffs in FIG. 7(a) (b is arranged next to a, and c is arranged next to b) is a correct arrangement relationship. On the other hand, in FIG. 7(b), the positions of foodstuffs a and b are reversed. In the present embodiment, an error in the arrangement relationship between foodstuffs as shown in FIG. 7(b) is detected.

[0105] As can be seen from the image p shown in the center of FIGS. 7(a) and 7(b), the output image included in the first estimation result obtained by the first estimation unit 11 is a color pattern image in which the types (classes) of food ingredients are labeled in pixel units and the regions are distinguished by color patterns for each instance of the food ingredient. The pre-constructed learned second model NN2 learns the features of the color pattern image for each food ingredient instance obtained by the first estimation unit 11, that is, the relative arrangement relationship between the food ingredients, extracts the features of the image when an unknown color pattern image is input, and is a classifier that classifies the correctness of the relative arrangement relationship between the food ingredients.

[0106] Examples of the second model NN2 include two-class classification that learns and classifies good products and defective products, and one-class classification that learns and classifies only good products like an autoencoder.

[0107] Returning to FIG. 6, the second estimation unit 16 uses the learned second model NN2 that takes the output image included in the first estimation result by the first estimation unit 11 as an input, estimates the relative arrangement relationship between a plurality of types of food ingredients in the output image, and outputs a second estimation result in which the estimated arrangement relationship is associated with the output image from the first estimation unit 11.

[0108] The second estimation unit 16 directly uses, as an input, the output image in which the color pattern is displayed for each region of the food ingredient instance included in the first estimation result. Further, the second estimation unit 16 extracts features indicating the arrangement relationship, which is the relative position between a plurality of types of food ingredients, from the input color pattern output image, and probabilistically classifies the correctness of the arrangement relationship between the plurality of types of food ingredients from the features. Note that the relative arrangement relationship between the food ingredients estimated by the second estimation unit 16 may be not only the arrangement relationship between a plurality of food ingredients of different types, but also the relative arrangement relationship between individual food ingredients when one type of food ingredient is composed of a plurality of food ingredients according to the setting.

[0109] The second determination unit 17 determines whether or not a second criterion preset regarding the validity of the relative arrangement relationship between a plurality of types of food materials arranged in a food container or the like is satisfied based on the second estimation result by the second estimation unit 16. For example, the second determination unit 17 can provide a preset threshold value for the classification result indicated by probability and determine the correctness of the estimated arrangement relationship.

[0110] In addition, the second criterion for determining the validity of the arrangement relationship when one type of food material is composed of a plurality of food materials can be set as a criterion for determining whether a plurality of food materials constituting the same type of food material are in a certain arrangement. By using such a second criterion, the second determination unit 17 can determine the presence or absence of the validity of the relative arrangement relationship between a plurality of food materials constituting the same type of food material.

[0111] For example, shredded cheese arranged as a pizza topping will be described as an example. For example, it is assumed that it is preferable that a plurality of shredded cheeses are scattered on the pizza dough without bias with a certain distance from each other. As the second criterion in this case, the presence or absence of the validity of the relative arrangement relationship of a plurality of shredded cheeses, in other words, whether the arrangement of the shredded cheese satisfies the criterion, or a criterion for determining whether the degree of scattering of the shredded cheese arranged as a topping is constant is set. For example, based on such a second criterion, the second determination unit 17 can detect whether each shredded cheese is arranged at a predetermined position, and further whether it is not topped biasedly at a specific position.

[0112] The inspection unit 13 performs an inspection regarding the arrangement of a plurality of food materials arranged in a food container or the like based on the determination result by the first determination unit 12 and the determination result by the second determination unit 17. In the present embodiment, the inspection unit 13 performs a final food plating inspection from the determination result regarding the presence, absence, and excess or deficiency of the food materials by the first determination unit 12 and the determination result regarding the arrangement position of the food materials by the second determination unit 17.

[0113] The prompting unit 14 presents the inspection results by the inspection unit 13. More specifically, as shown in FIG. 10, the prompting unit 14 causes the "estimated quantity", "estimated area" by the first estimation unit 11 for each food material, the "quantity threshold value", and the "area threshold value" used by the first determination unit 12 to be displayed in the area B displayed on the display screen of the display device 108. Further, the prompting unit 14 causes the values of the determination results "quantity determination", "area determination", and the item "determination result" of the inspection result based on the "inspection item" to be displayed in the area B.

[0114] Also, the prompting unit 14 causes the value of the second estimation result "probability (0.999)" by the second estimation unit 16 and the value of the determination result "NG" regarding the validity of the relative arrangement relationship between food materials by the second determination unit 17 to be displayed in the item "location inspection result" of the table provided in the area B.

[0115] Furthermore, the prompting unit 14 causes the final inspection result ("inspection result") regarding the arrangement of a plurality of food materials based on the determination result (item "determination result") regarding the presence, absence, excess, or deficiency of food materials by the first determination unit 12 and the determination result ("location inspection result") regarding the validity of the arrangement relationship between food materials by the second determination unit 17 to be displayed in the area B.

[0116] Also, the prompting unit 14 can display side by side in the area A the image i of the inspection target acquired by the acquisition unit 10, the output image p of the color pattern for each type of food material including the first estimation result by the first estimation unit 11, and the image s obtained by synthesizing the image i of the inspection target and the output image p.

[0117] [Operation of the inspection device] Next, the operation of the inspection apparatus 1A having the above-described configuration will be described with reference to the flowchart of FIG. 8 and the explanatory diagram of FIG. 9. It is assumed that the learning processes of the first learning model NN1 and the second learning model NN2 are performed in advance by an external server or the like. In addition, the first memory 15A stores a first standard including a pre-constructed learned first learning model NN1 and threshold value information and the like used for the determination process for the calculation result thereof. Similarly, it is assumed that the second memory 15B stores a second standard including a learned second learning model NN2 and threshold value information and the like used for the determination process for the calculation result thereof.

[0118] First, the first estimation unit 11 loads the learned first model NN1 from the first memory 15A (step S10). Next, the second estimation unit 16 loads the learned second model NN2 from the second memory 15B (step S11).

[0119] Next, the acquisition unit 10 acquires an image including a plurality of food ingredients arranged on a food container or the like, which is photographed by the camera 105 (step S12). Next, the first estimation unit 11 performs the calculation of the learned first model NN1 loaded in step S10 (step S13).

[0120] Specifically, as shown in FIG. 9, the first estimation unit 11 uses the learned first model NN1 to estimate the regions a1, b1, c1, d1 of the plurality of food ingredients and the types a, b, c, d of the food ingredients in the image i acquired in step S10, and outputs a first estimation result including an output image p in which the estimated regions and types of the plurality of food ingredients are associated with the input image i. The output image p is a color pattern image in which the corresponding regions are painted in an arbitrary color for each type of the plurality of food ingredients.

[0121] That is, the output image included in the first estimation result includes pixel information and position information in the image. In the present embodiment, since such instance segmentation is adopted as the first model NN1, even when a plurality of food ingredients of the same type are included in the image, it is possible to detect the food ingredients by distinguishing each instance.

[0122] Next, the first estimation unit 11 inputs the first estimation result, which is the calculation result of the learned first model NN1, to the second estimation unit 16 (step S14). More specifically, as shown in step S13 of FIG. 9, the first estimation unit 11 inputs the output image p in which a color pattern is assigned to each region of the food material instance to the second estimation unit 16. Note that the first estimation result by the first estimation unit 11 is also passed to the first determination unit 12.

[0123] Next, the second estimation unit 16 performs the calculation of the learned second model NN2 loaded in step S11 and passes the calculation result to the second determination unit 17 (step S15). More specifically, the second estimation unit 16 uses the learned second model NN2 that takes the output image included in the first estimation result as an input, estimates the arrangement relationship between a plurality of types of food materials in the output image, and outputs a second estimation result in which the estimated arrangement relationship is associated with the output image.

[0124] Next, the first determination unit 12 determines whether or not to satisfy a first criterion preset regarding the presence, absence, excess, and deficiency for each type of food material based on the first estimation result by the calculation of the learned first model NN1 in step S13 (step S16). More specifically, the first determination unit 12 reads out the first criterion such as determination items and thresholds for each type of food material stored in the first memory 15A in association with the learned first model NN1 from the first memory 15A and performs the process of step S16.

[0125] For example, the first determination unit 12 can perform threshold processing on the estimated number of each food material, that is, the number of regions or color patterns of the same type of food material. In addition, the first determination unit 12 can perform threshold processing on the area occupied by the estimated region of the food material in the image for each type of food material.

[0126] Next, the second determination unit 17 determines whether the arrangement relationship among the plurality of types of food materials satisfies a second criterion preset regarding the validity of the arrangement relationship based on the second estimation result obtained in step S15 (step S17). More specifically, the second determination unit 17 reads out from the second memory 15B the second criterion including thresholds and the like stored in the second memory 15B in association with the learned second model NN2 and performs the process of step S17.

[0127] For example, in region B of FIG. 10, based on the probability indicating whether the relative arrangement relationship between the food materials estimated by the second estimation unit 16 is correct as indicated by the value of the item "location inspection result", it is determined that the position where the food materials are arranged is an error "NG".

[0128] Next, the inspection unit 13 performs an inspection on the arrangement of the plurality of food materials arranged in the food container or the like based on the determination result regarding the presence or absence and excess or deficiency of each food material obtained in step S16 and the determination result regarding the validity of the arrangement relationship among the plurality of types of food materials obtained in step S17 (step S18).

[0129] For example, in step S16, if all types of food materials satisfy the first criterion regarding the number and area, and in step S17, if the arrangement relationship among the plurality of types of food materials satisfies the second criterion, the inspection unit 13 can output an inspection result indicating that the inspection on the arrangement of the plurality of food materials arranged in the food container or the like is passed. In the present embodiment, the inspection on the arrangement of the food materials arranged in the container or the like includes a food plating inspection regarding the type, presence or absence, number, quantity, and arrangement position of each food material.

[0130] Next, the presentation unit 14 presents the inspection result regarding the arrangement of the food materials obtained in step S18 (step S19). For example, as shown in FIG. 10, the presentation unit 14 can display the estimated number and estimated area for each of the food materials a, b, c, and d in a table format in area B of the display screen of the display device 108. Further, the presentation unit 14 can display the determination result based on the comparison with the threshold values for the number and area in the table provided in area B.

[0131] Furthermore, the presentation unit 14 can display the second estimation result by the second estimation unit 16 and the determination result regarding the validity in the arrangement relationship between multiple types of food materials by the second determination unit 17 in the table in area B. Also, the presentation unit 14 can display the result of the comprehensive food plating inspection performed by the inspection unit 13 in the table in area B.

[0132] Also, as described above, the presentation unit 14 arranges and displays the image i captured by the camera 105, the output image p of the color pattern, and the image s obtained by synthesizing the image i and the output image p of the color pattern in area A respectively. Further, the presentation unit 14 arranges and displays, in area A, an image of the food with the food materials arranged at the correct positions as a reference image, so that the user can more intuitively grasp the inspection result regarding the arrangement of the food materials.

[0133] Here, the processing time when performing a food plating inspection using the inspection device 1A according to the present embodiment will be described with reference to FIG. 11. The horizontal axis in FIG. 11 represents the data interval [msec], and the vertical axis represents the frequency of the number of foods to be inspected. The processing time is the time recorded as the processing time from when an image of the food produced by the camera 105 is captured until the estimation process by instance segmentation by the first estimation unit 11, the classification by the second estimation unit 16, further, the saving of the image, and the saving of each numerical value. Note that the accuracy of the instance segmentation and the classifier is designed to have a sufficient accuracy for practical use in advance.

[0134] The inspection time for the first meal shown at the far right of the horizontal axis in Fig. 11 is approximately 700 msec. Also, it can be seen that the processing time for meals after the first meal is 400 msec. This indicates that there is a margin in the processing time when the tact time in the food production line is, for example, 1.0 sec. Thus, it can be understood that the inspection apparatus 1A according to the present embodiment has a simpler configuration while maintaining sufficient inspection accuracy and can obtain sufficient processing time.

[0135] As described above, according to the second embodiment, by using the first estimation result by the first estimation unit 11 as the input to the classifier of the second estimation unit 16, it is possible to perform an inspection regarding the arrangement of food materials using image recognition with a simpler configuration.

[0136] Also, according to the second embodiment, in one inspection apparatus 1A, it is possible to perform an inspection not only on the presence, number, and quantity of food materials arranged in a food container or the like, but also on the relative positions among a plurality of types of arranged food materials, and a more efficient and effective inspection regarding the arrangement of food materials is realized.

[0137] Also, according to the second embodiment, it is possible to perform an inspection regarding the correctness of the arrangement of food materials regardless of the shape and material of the container or the like in which the food materials are arranged.

[0138] [Third Embodiment] Next, a third embodiment of the present invention will be described. In the following description, the same components as those in the above-described first and second embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

[0139] In the second embodiment, the case where the estimation process is performed using the pre-trained first model NN1 and the pre-trained second model NN2 that have been learned in an external server or the like in advance was described. In contrast, in the third embodiment, a learning apparatus having a first learning unit 18 that learns the first model NN1 and a second learning unit 19 that learns the second model NN2 is further provided, and each learning process is performed in the own apparatus.

[0140] [Functional Blocks of the Inspection Device] FIG. 12 is a block diagram showing the configuration of the inspection device 1B according to the present embodiment. The inspection device 1B includes an acquisition unit 10, a first estimation unit 11, a first determination unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, a second determination unit 17, a first learning unit 18, and a second learning unit 19. The inspection device 1B is different from the configuration of the second embodiment in that it further includes a first learning unit 18 and a second learning unit 19. Hereinafter, the description will focus on the configurations different from the first and second embodiments.

[0141] The first learning unit 18 uses an image including food ingredients arranged in a food container or the like as a learning image (first learning image) to train a first model of a neural network, extracts a first feature amount for identifying the region and type of the food ingredients, and obtains a trained first model NN1 based on the extracted first feature amount. The trained first model NN1 obtained by the first learning unit 18 is stored in the first memory 15A.

[0142] More specifically, the first learning unit 18 uses a neural network model having an instance segmentation architecture such as Mask R-CNN as the first model NN1. The first learning unit 18 identifies a plurality of food ingredients a, b, c, d included in the learning image pixel by pixel, and learns a feature amount such that the types a, b, c, d of the food ingredients in the region of the food ingredient instances included in the learning image are estimated.

[0143] For example, as shown in FIG. 9, learning images for each of the food ingredients a, b, c, d are given as inputs to the first model NN1. The first learning unit 18 performs a learning operation of the first model NN1 and learns the features of the food ingredients a, b, c, d. The number of learning images used in the learning process is such that sufficient accuracy is ensured in the subsequent inspection process. For example, 10 images of each of the food ingredients a, b, c, d captured by the camera 105 may be cut out and used as learning images. Note that images previously acquired by the acquisition unit 10 may be used as the learning images.

[0144] The second learning unit 19 uses the output image included in the estimation result by the first estimation unit 11 as a learning image (second learning image) to train a second model by a neural network, extracts a second feature amount for identifying the relative arrangement relationship between food materials in the learning image, and obtains a trained second model NN2 based on the extracted second feature amount. The trained second model NN2 obtained by the second learning unit 19 is stored in the second memory 15B.

[0145] More specifically, the second learning unit 19 trains an output image of a color pattern by instance segmentation to construct a trained second model NN2 that outputs the correctness of the arrangement relationship between food materials as a classification result. Examples of the second model NN2 include two-class classification for training and classifying good and defective products, and one-class classification for training and classifying only good products like an autoencoder.

[0146] Although it will be described in detail in the sixth embodiment described later, for example, the second learning unit 19 can secure a required number of learning images by duplicating or processing a certain number of output images output by the first estimation unit 11 performing the operation of the trained first model NN1. Also, as shown in FIG. 9, as learning images, an image with correct mutual positions of color patterns representing a plurality of types of food materials a, b, c, d arranged in a food container or the like and an incorrect image are used. Note that an image with an incorrect arrangement position of the food materials may be created by processing the output image included in the first estimation result.

[0147] Note that the first learning unit 18 and the second learning unit 19 can construct the trained first model NN1 and the trained second model NN2 for each of a plurality of different food varieties and store them in the memory 15 (the first memory 15A, the second memory 15B). Also, the first memory 15A stores a first criterion such as a threshold value used for the operation result of the trained first model NN1 associated with the trained first model NN1. Similarly, the second memory 15B stores a second criterion such as a threshold value used for the operation result of the trained second model NN2 associated with the trained second model NN2.

[0148] In this embodiment, the first learning unit 18, the second learning unit 19, the first memory 15A, and the second memory 15B constitute a learning device that performs learning processing for constructing a preset model.

[0149] [Operation of the inspection device] Next, the operation of the inspection device 1B having the above-described configuration will be described using the flowchart of FIG. 13.

[0150] First, the acquisition unit 10 acquires, as a learning image, an image including a plurality of food ingredients arranged in a food container or the like, captured by the camera 105 (step S101). Next, the first learning unit 18 uses the learning image acquired in step S101 to learn a first model of a neural network that realizes a preset instance segmentation, extracts a first feature amount for identifying the region and type of the food ingredient for each pixel in the learning image, and obtains a learned first model NN1 based on the extracted first feature amount (step S102). Next, the learned first model NN1 obtained in step S102 is stored in the first memory 15A (step S103). More specifically, in step S103, a first criterion used in a determination process (step S111 described later) for the calculation result of the learned first model NN1 is further stored in the first memory 15A.

[0151] Next, the second learning unit 19 acquires, as a learning image, an output image from the learned first model NN1 (step S104). For example, the second learning unit 19 can acquire an output image output by an estimation process using the learned first model NN1 constructed in advance.

[0152] The second learning unit 19 uses the acquired learning image to learn a second model NN2 by a neural network, extracts a second feature amount for identifying the relative arrangement relationship among a plurality of types of food ingredients from the learning image, and obtains a learned second model NN2 based on the extracted second feature amount (step S105).

[0153] After that, the learned second model NN2 obtained in step S105 is stored in the second memory 15B (step S106). More specifically, in step S106, a second criterion used in the determination process (step S112 described later) for the calculation result of the learned second model NN2 is stored in the second memory 15B.

[0154] The acquisition unit 10 acquires an inspection image captured by the camera 105 (step S107). Next, the first estimation unit 11 reads the learned first model NN1 stored in the first memory 15A in step S103, and performs the calculation of the learned first model NN1 using the image acquired in step S107 as an input (step S108).

[0155] Also, in step S108, the first estimation unit 11 estimates the regions and types of a plurality of food ingredients in the image acquired in step S107 using the learned first model NN1, and outputs a first estimation result including an output image in which the estimated regions and types of the plurality of food ingredients are associated with the input image. The output image is a color pattern image in which corresponding regions are painted in arbitrary colors for each type of the plurality of food ingredients.

[0156] The output image included in the first estimation result includes pixel information and position information. In the present embodiment, since instance segmentation is adopted as the first model NN1, even when a plurality of food ingredients of the same type are arranged in the image, it is possible to detect them separately for each instance of the food ingredient.

[0157] Next, the first estimation unit 11 inputs the first estimation result to the second estimation unit 16 (step S109). The first estimation unit 11 outputs a first estimation result including an output image in which a color pattern is assigned to each region of the food ingredient. This first estimation result is used as an input image for the second estimation unit 16. Note that the first estimation result is also passed to the first determination unit 12.

[0158] Next, the second estimation unit 16 reads the learned second model NN2 stored in the second memory 15B in step S106, gives the output image included in the first estimation result to the learned second model NN2, performs the operation of the learned second model NN2, and passes the operation result to the second determination unit 17 (step S110).

[0159] More specifically, the second estimation unit 16 uses the learned second model NN2 that takes the output image included in the first estimation result as input, estimates the arrangement relationship between multiple types of food ingredients in the output image, and outputs a second estimation result in which the estimated arrangement relationship is associated with the output image.

[0160] Next, the first determination unit 12 determines whether or not to satisfy a first criterion preset regarding the presence, absence, excess, and deficiency for each type of food ingredient based on the first estimation result obtained by the operation of the learned first model NN1 in step S13 (step S111). More specifically, the first determination unit 12 reads out the first criterion including threshold values and the like stored in the first memory 15A in association with the learned first model NN1 from the first memory 15A and performs the determination process of step S111.

[0161] For example, the first determination unit 12 can perform threshold processing on the estimated number of each food ingredient, that is, the number of regions or color patterns of the same type of food ingredient. In addition, the first determination unit 12 can perform threshold processing on the area occupied by the estimated food ingredient region in the image for each type of food ingredient.

[0162] Next, the second determination unit 17 determines whether or not the arrangement relationship between multiple types of food ingredients satisfies a second criterion preset regarding the validity of the arrangement relationship based on the second estimation result obtained in step S110 (step S112). More specifically, the second determination unit 17 reads out the second criterion including threshold values and the like stored in the second memory 15B in association with the learned second model NN2 from the second memory 15B and performs the determination process of step S112.

[0163] Next, the inspection unit 13 performs an inspection on the arrangement of a plurality of food ingredients arranged in a food container or the like based on the determination results regarding the presence or absence and excess or deficiency of each food ingredient obtained in step S111 and the determination result regarding the validity of the arrangement relationship between the food ingredients obtained in step S112 (step S113).

[0164] For example, in step S111, if all types of food ingredients satisfy the first criteria regarding the number and area, and in step S112, if the arrangement relationship between the food ingredients satisfies the second criteria, the inspection unit 13 can output an inspection result indicating that the inspection on the arrangement of the plurality of food ingredients arranged in a food container or the like is passed. In the present embodiment, the inspection on the arrangement of the food ingredients arranged in a container or the like includes a food plating inspection regarding the type, presence or absence, number, quantity, and arrangement position of each food ingredient.

[0165] Next, the presentation unit 14 presents the inspection result regarding the arrangement of the food ingredients obtained in step S113 (step S114).

[0166] As described above, according to the third embodiment, the inspection device 1B includes a learning device. The first learning unit 18 learns the first model NN1 that realizes instance segmentation, and the second learning unit 19 learns the second model NN2 that classifies the correctness of the arrangement relationship between the food ingredients using the output image from the learned first model NN1 as a learning image. Therefore, the processes from the learning process to the estimation process and further to the inspection can be performed by one inspection device 1B with a simpler configuration.

[0167] Also, according to the third embodiment, the second model NN2 uses the output image of the learned first model NN1 as an input image. Therefore, the learning image for the second model NN2 can be easily prepared, and the learning process of the second model NN2 can be further simplified.

[0168] Further, according to the third embodiment, since the learning images can be easily obtained, if the first model NN1 and the second model NN2 are constructed for each food product, even when the manufactured food products are small in quantity and diverse in variety, or when product modifications and discontinuations are frequently carried out, it is possible to easily cope with the plating inspection of the food products related to the small-quantity and diverse-variety products.

[0169] Also, in the third embodiment, when acquiring the learning images, a simply constructed image shooting environment is used, and the processing of the acquired images themselves may be simple processing. Therefore, the plating inspection of food using image processing becomes easy.

[0170] [Fourth Embodiment] Next, a fourth embodiment of the present invention will be described. In the following description, the same components as those in the first to third embodiments described above are denoted by the same reference numerals, and the description thereof will be omitted.

[0171] In the second and third embodiments, the case where the second learning unit 19 and the second estimation unit 16 use the output image obtained by the operation of the learned first model NN1 as the learning image and the input image for the estimation process as they are has been described. In contrast, in the fourth embodiment, the learning image and the input image used in the second learning unit 19 and the second estimation unit 16 are converted, and a simpler image is used as the learning image of the second learning unit 19.

[0172] [Functional Blocks of the Inspection Device] FIG. 14 is a block diagram showing the configuration of the inspection device 1C according to the present embodiment. The inspection device 1C includes an acquisition unit 10, a first estimation unit 11, a first determination unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, a second determination unit 17, a first learning unit 18, a second learning unit 19, and a conversion unit 20. The inspection device 1C is different from the configuration of the third embodiment in that it further includes a conversion unit 20. Hereinafter, the description will focus on the configuration different from the first to third embodiments.

[0173] The conversion unit 20 converts the output image included in the first estimation result by the first estimation unit 11 to generate a more simplified converted image. For example, as shown in the example of FIG. 15, in the output image from the first estimation unit 11, for each type of food material, the corresponding region has the actual shape of the food material. Also, it is a color pattern image in which each region is painted with an arbitrary color. The conversion unit 20 converts the color pattern image into a color pattern image in which each region of the color pattern image is replaced with an object having the same shape. For example, the conversion unit 20 can replace the shapes of regions a1, b1, c1, d1 with an object such as a circle. In this case, the image converted by the conversion unit 20 is an image in which the colors of the color patterns of each region are different from each other but the shapes are the same.

[0174] As shown in FIGS. 15 and 16, the color pattern image has a complex shape in which each region has the outer shape of an actual food material instance. Therefore, the conversion unit 20 replaces each region with a more simplified object from which minute changes that do not affect boundary extraction are removed, thereby reducing the computational load in the learning process and the estimation process using the second model NN2.

[0175] To give another example, as shown in FIG. 16, the conversion unit 20 can replace each region of the color pattern image of the output image from the first estimation unit 11 with an object having a different shape for each color of the color pattern, and convert it into an image in which the color patterns of the replaced objects are all the same color. That is, the conversion unit 20 replaces a region having a complex shape with a simple-shaped object having a different shape for each type of food material. For example, a converted image can be generated by replacing the shape of the region with a more simplified geometric shape, such as a triangle, a square, a circle, a star, or any other shape. In one example, the converted image can be a monochrome binary image, thereby reducing the data size of the converted image.

[0176] Of course, it goes without saying that the conversion unit 20 can replace each region of the color pattern image with an object having a different shape for each color of the color pattern, and make the replaced objects have different colors for each shape of the object.

[0177] The second learning unit 19 uses the image converted by the conversion unit 20 as a learning image to perform learning of the second model NN2 and obtains the learned second model NN2.

[0178] The second estimation unit 16 uses the learned second model NN2 that takes the converted image generated by the conversion unit 20 as an input to estimate the relative arrangement relationship between a plurality of food ingredients in the converted image.

[0179] As described above, according to the fourth embodiment, in the conversion unit 20, a converted image in which the output image from the first estimation unit 11 is further simplified is generated, and the second model NN2 is learned using the converted image as a learning image. Therefore, it is possible to reduce the arithmetic load and the memory amount in the learning process of the second model NN2 in the second learning unit 19. In addition, the arithmetic load when performing the operation of the learned second model NN2 in the second estimation unit 16 can be reduced.

[0180] [Fifth Embodiment] Next, a fifth embodiment of the present invention will be described. In the following description, the same components as those in the first to fourth embodiments described above are denoted by the same reference numerals, and the description thereof will be omitted.

[0181] In the fourth embodiment, the case of generating a converted image in which the color pattern for each food ingredient region included in the output image from the first estimation unit 11 is converted and replaced with an object having a simpler shape has been described. In contrast, in the fifth embodiment, an image in which discontinuous portions included in the color pattern are complemented to be continuous is generated from the output image included in the output image from the first estimation unit 11 and used for learning the second model NN2.

[0182] [Functional Blocks of the Inspection Device] FIG. 17 is a block diagram showing the configuration of the inspection apparatus 1D according to the present embodiment. The inspection apparatus 1C includes an acquisition unit 10, a first estimation unit 11, a first determination unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, a second determination unit 17, a first learning unit 18, a second learning unit 19, and a processing unit 21. The inspection apparatus 1D is different from the configuration of the fourth embodiment including the conversion unit 20 in that it includes the processing unit 21. Hereinafter, the description will focus on the configuration different from the first to fourth embodiments.

[0183] The processing unit 21 generates a processed image in which the discontinuity of the color pattern is complemented and made continuous, assuming that the inside of the longest estimated outer contour line of the food material region included in the output image of the estimation result by the first estimation unit 11 is the food material region.

[0184] Specifically, as shown in FIG. 18, in the output image from the first estimation unit 11, it is assumed that the food material region b1 includes a discontinuous portion. For example, there may be a case where the shape of the food material itself has a cavity, or a case where a gap is formed depending on the processing state of the food material. In such a case, from the viewpoint of reducing the learning load of the neural network in the second learning unit 19, it is effective to clarify in advance that all regions within the region indicate the estimated food material.

[0185] Therefore, the processing unit 21 generates a processed image in which the discontinuous portion within the region b1 is complemented with the color pattern assigned to the region b1 and made continuous, regarding the holes included inside the region b1 as the region indicating the food material b.

[0186] The second learning unit 19 uses the processed image in which the discontinuous portion of the color pattern complemented and made continuous by the processing unit 21 as a learning image to perform learning of the second model NN2 and obtain a learned second model NN2.

[0187] The second estimation unit 16 performs the operation of the learned second model NN2 using, as input, a processed image of the output image in which the discontinuity of the color pattern generated by the processing unit 21 is complemented to be continuous, estimates the relative arrangement relationship between the foodstuffs, and outputs a second estimation result in which the estimated arrangement relationship is associated with the output image.

[0188] As described above, according to the fifth embodiment, in the processing unit 21, a processed image is generated in which discontinuous portions of the color pattern in the region of the foodstuffs included in the output image from the first estimation unit 11 are complemented to be continuous portions, and the second model NN2 is learned. Therefore, it is possible to reduce the computational load in the learning of the second model NN2 in the second learning unit 19 and perform learning more efficiently.

[0189] [Sixth Embodiment] Next, a sixth embodiment of the present invention will be described. In the following description, the same components as those in the above-described embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

[0190] In the above-described third embodiment, a configuration was described in which the learning device further includes a first learning unit 18 that learns the first model NN1 and a second learning unit 19 that learns the second model NN2. Also, in the third embodiment, the case where only the output image obtained by the operation of the learned first model NN1 is used as the learning image when the second learning unit 19 performs the learning of the second model NN2 was described. In contrast, in the sixth embodiment, the learning device performs data augmentation to increase the learning images used in the learning of the second model NN2.

[0191] FIG. 20 is a diagram for explaining the outline of data augmentation processing performed by the inspection apparatus 1E according to the present embodiment. The image data shown in the broken-line frame DA in FIG. 20 is a learning dataset for learning the second model NN2 that classifies the correctness of the arrangement relationship between foodstuffs. Each learning image t is given label information ("good", "bad") indicating whether the arrangement relationship between foodstuffs is correct or incorrect. When improving the classification accuracy of the second model NN2, it is considered effective to increase the number of learning images related to both labels, where the arrangement relationship between foodstuffs is correct (hereinafter sometimes referred to as "good product" or "good"), and the arrangement relationship is incorrect (hereinafter sometimes referred to as "defective product" or "bad"), and perform learning.

[0192] However, when the inspection apparatus 1E is used for inspection of a food production line, it is relatively easy to acquire a large number of learning images related to good products, but almost no learning images related to defective products can be acquired. In particular, the second model NN2 according to the present embodiment is a classifier that classifies two classes of good products and defective products, and in order to ensure and improve the accuracy, it is desirable to prepare a sufficient amount of learning images related to defective products and perform learning.

[0193] For example, Non-Patent Documents 1 to 3 disclose techniques for generating learning images by data augmentation processing. Further, Patent Document 2 discloses a technique for performing data augmentation processing by specifying the position and form information of the object to be expanded in order to prevent the boundary of the characteristic portion of the image data from becoming unnatural and generating a learning image. Further, Patent Document 3 discloses a technique for processing the characteristic portion of an image and performing data augmentation based on the processed data to efficiently generate a learning image.

[0194] However, all of the prior arts disclosed in Non-Patent Documents 1 to 3 and Patent Documents 2 and 3 perform data augmentation of so-called good product images to generate a large number of good product images with the same label, and it has been difficult to perform data augmentation based on good product images to generate a large number of defective product images with different labels.

[0195] In contrast, in the present embodiment, by utilizing the properties specific to the learning images of the second model NN2, it is possible to generate learning images related to defective products that are relatively difficult to obtain on the production line. More specifically, in the present embodiment, it is noted that the learning images of the second model NN2 are images of color patterns obtained by instance segmentation that discriminates regions within an image where food ingredients exist by distinguishing the food ingredients from each other on a pixel-by-pixel basis, and that the label information of the learning images is the correctness of the relative positional relationship between the food ingredients.

[0196] In the inspection apparatus 1E according to the present embodiment, based on these points of attention, data augmentation of target images related to non-defective products that can be relatively easily obtained is performed to enable the generation of a large number of learning images related to defective products that are relatively difficult to obtain on an actual production line.

[0197] [Functional Blocks of Inspection Apparatus] FIG. 19 is a block diagram showing the configuration of the inspection apparatus 1E according to the present embodiment. The inspection apparatus 1E includes an acquisition unit 10, a first estimation unit 11, a first determination unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, a second determination unit 17, a first learning unit 18, a second learning unit 19, an extraction unit 22, and a data augmentation unit 23. The inspection apparatus 1E is particularly different from the configurations of the third to fifth embodiments in that it further includes an extraction unit 22 and a data augmentation unit 23 in the learning apparatus. Hereinafter, the description will focus on the configurations different from those of the third to fifth embodiments.

[0198] The extraction unit 22 extracts regions for each type of food ingredient from an image to be subjected to data augmentation (target image). The target image is an image to be subjected to data augmentation by the data augmentation unit 23 described later, and is an image to which label information regarding the correctness of the relative positional relationship between food ingredients has been given in advance. For example, an output image in which the estimated regions and types of food ingredients included in the first estimation result output by the first estimation unit 11 are associated with the image can be used as the target image. Alternatively, in the case where an image created in advance by a user, engineer, etc. is used as the target image, etc., an image to which label information has been given can be acquired from the outside in advance and used as the target image. In the present embodiment, in particular, an image to which label information has been given is used as the target image for an output image in which regions for each food ingredient are represented as color patterns. The extraction unit 22 can also extract only the regions of food ingredients related to a specified color pattern from the target image according to the setting.

[0199] For example, among the three images included in the folder of the target image data DA0 in FIG. 21, the extraction unit 22 extracts the regions a1, b1, c1, and d1 of the food ingredients a, b, c, and d included in the image p surrounded by the thick frame (hereinafter sometimes referred to as "target image p"). A first label "good", indicating that the relative arrangement relationship between the food ingredients is correct in advance, is given to the target image p. In addition, the information on the regions a1, b1, c1, and d1 of the food ingredients a, b, c, and d extracted by the extraction unit 22 includes color information and coordinate information of each region.

[0200] The target image is not limited to being an output image of a color pattern. For example, as described in the fourth embodiment, a converted image obtained by replacing the output image of the color pattern with an object having a simpler shape can also be used. Similarly, as the target image, a processed image in which discontinuous portions included in the color pattern are complemented to be continuous, as described in the fifth embodiment, can also be used.

[0201] The data augmentation unit 23 performs data augmentation on the target image in which the area and type of food ingredients are associated with the image to generate a learning image (second learning image). More specifically, the data augmentation unit 23 generates a learning image based on the label information regarding the validity of the relative arrangement relationship among a plurality of food ingredients given to the target image. Note that the label information is information corresponding to a second criterion for the second determination unit 17 to determine the validity of the relative arrangement relationship among a plurality of types of food ingredients arranged in a food container or the like. That is, a first label "good" is pre-assigned to an image that satisfies the second criterion, and a second label "bad" is pre-assigned to an image that does not satisfy the second criterion.

[0202] The data augmentation unit 23 can perform data augmentation on a target image to which a first label "good" indicating that the relative arrangement relationship among a plurality of food ingredients in the target image is appropriate, and generate a learning image to which a second label "bad" indicating that the relative arrangement relationship among the plurality of food ingredients is inappropriate is assigned. Further, the data augmentation unit 23 can perform data augmentation on a target image to which the first label "good" is assigned and generate a learning image to which the first label "good" with the same label information is assigned. Similarly, the data augmentation unit 23 can perform data augmentation on a target image to which the second label "bad" is assigned and further generate a learning image to which the second label "bad" is assigned.

[0203] For example, the data augmentation unit 23 can generate a learning image by processing the area extracted by the extraction unit 22 while maintaining the relative arrangement relationship among a plurality of food ingredients in the target image. Specifically, as shown in the data augmentation DA1 of FIG. 21, while maintaining the relative positional relationship among the areas a1, b1, c1, and d1 included in the target image p to which the first label "good" surrounded by a thick frame is assigned, each of the areas a1, b1, c1, and d1 is rotated to generate a learning image with the first label "good" in consecutive numbers. In the example of the data augmentation DA1 in FIG. 21, for the learning image shown on the left, the area a1 of the target image p is rotated by a random angle starting from the center coordinates, for the learning image shown in the middle, the area b1 is further rotated, and for the learning image shown on the right, the area c1 is further rotated.

[0204] In this way, the data augmentation unit 23 rotates each of the regions a1, b1, c1, and d1 from one target image p with the first label "good" assigned thereto, and relocates each region to the same position, so that a large number of learning images with the first label "good" assigned thereto and belonging to the same label can be generated in sequence. In the data augmentation DA1 shown in FIG. 21, the case where each region is rotated is illustrated. However, as long as it is a processing operation that can be performed while maintaining the positional relationship between the regions, other processes such as changing the size and shape of the regions and relocating them to the same position are also included.

[0205] Next, an example of performing data augmentation on a target image with the first label "good" assigned thereto to generate a learning image with the second label "bad" assigned thereto will be described. In this case, the data augmentation unit 23 can generate a learning image by replacing the positions of regions corresponding to two or more different types of food materials among the regions extracted by the extraction unit 22.

[0206] For example, in the data augmentation DA2 shown in FIG. 21, the positions of regions b1 and c1 of the target image p with the first label "good" assigned thereto in the thick frame of the target image data DA0 are replaced with each other and relocated to generate a learning image with the second label "bad" assigned thereto shown on the left. Similarly, the image shown in the middle is a learning image with the second label "bad" assigned thereto, which is generated by replacing and relocating regions a1 and d1 of the target image p. The image shown on the right is a learning image with the second label "bad" assigned thereto, which is generated by replacing and relocating regions b1 and d1 of the target image p.

[0207] The data augmentation unit 23 can calculate the number of combinations of replacing any two colors for a target image with the first label "good" assigned thereto, and generate images for all such combinations, thereby generating a large number of learning images with the second label "bad" assigned thereto.

[0208] Next, another example of performing data augmentation on a target image with the first label "good" to generate a training image with the second label "bad" will be described. In this case, for example, the data augmentation unit 23 can generate a training image by deleting one or more of the regions extracted by the extraction unit 22 from the target image.

[0209] Specifically, as shown in the data augmentation DA3 of FIG. 21, the data augmentation unit 23 extracts only one of the regions a1, b1, c1, and d1 included in the thick-frame target image p with the first label "good", and deletes and rearranges the other regions to generate a training image with the second label "bad".

[0210] Furthermore, another example of performing data augmentation on a target image with the first label "good" to generate a training image with the second label "bad" will be given. In this case, the data augmentation unit 23 can perform data augmentation on the target image by performing at least any one of inversion, rotation, dilation, and contraction of each of the regions extracted by the extraction unit 22.

[0211] For example, as in the data augmentation DA4 shown in FIG. 21, the data augmentation unit 23 can generate a training image with the second label "bad" by vertically inverting and rearranging the thick-frame target image p with the first label "good".

[0212] In the data augmentation DA5 shown in FIG. 21, the data augmentation unit 23 generates a training image with the second label "bad" by horizontally inverting and rearranging the thick-frame target image p with the first label "good".

[0213] The data augmentation unit 23 may further perform data augmentation on the training image to which the first label "good" or the second label "bad" obtained by performing data augmentation on the target image is assigned, and generate a training image to which the first label "good" or the second label "bad" is assigned. For example, as shown by the data augmentation DA4 in FIG. 21, the data augmentation unit 23 rotates the training image having the second label "bad" obtained by mirroring the target image having the first label "good" about the vertical axis by a random angle generated starting from the center coordinates of the region of each color pattern, thereby generating a training image assigned with the second label "bad".

[0214] In this way, the data augmentation unit 23 can generate a desired number of training images assigned with the second label "bad" based on the target image assigned with the first label "good". When the data augmentation unit 23 repeatedly performs data augmentation processing to generate training images related to defective products, training images related to non-defective products may be generated. However, training images related to non-defective products may be excluded from the generated training images according to a pre-setting.

[0215] [Operation of the inspection device] Next, the operation of the inspection device 1E having the above-described configuration will be described with reference to FIG. 22. FIG. 22 is a flowchart showing each step from the generation of the training image of the second model NN2 to the learning process of the second model NN2 in the inspection device 1E.

[0216] First, the learning device acquires a target image which is an image to be subjected to data augmentation (step S201). For example, the learning device can use, as the target image, the output image p of the color pattern in which a plurality of food material regions and types obtained as the calculation result of the learned first model NN1 by the first estimation unit 11 are associated with the input image i.

[0217] Next, the learning device acquires the label information of the target image acquired in step S201 (step S202). For example, the learning device can acquire the label information of the target image by an external input.

[0218] Next, the learning device performs preliminary settings when generating learning images (step S203). Specifically, a specific target image can be selected from the target images for which label information was obtained in step S202. Also, in step S203, the number of colors in the region for each food ingredient included in the target image, and the color of the region of the color pattern extracted by the extraction unit 22 can be specified. Furthermore, in step S203, the number of learning images to be generated, the rotation angle range of the region, etc. are set in advance.

[0219] Next, the extraction unit 22 extracts the color pattern related to the region of the food ingredient from the target image based on the setting information in step S203 (step S204). Subsequently, the data augmentation unit 23 performs processing such as processing of the region extracted by the extraction unit 22, performs data augmentation of the target image, and generates a learning image (step S205).

[0220] For example, in step S205, the data augmentation unit 23 can process the region extracted by the extraction unit 22 in step S202 while maintaining the relative arrangement relationship between a plurality of food ingredients in the target image to which the first label "good" is assigned, and generate a learning image. In this case, the data augmentation unit 23 assigns the second label "bad" to the generated learning image (step S206).

[0221] Next, the second learning unit 19 uses the learning image obtained in step S206 to perform learning of the second model NN2, and obtains the learned second model NN2 (step S207). The learning images used in step S207 can include a sufficient number of good product images and defective product images from the viewpoint of the classification accuracy of the second model NN2.

[0222] Next, the learning device stores the learned second model NN2 obtained in step S207 in the second memory 15B (step S208). Thereafter, the process returns, for example, to step S10 in FIG. 8, and performs operations on the learned first model NN1 and the learned second model NN2 learned using the training images generated by data augmentation, and can perform an inspection regarding the arrangement of food ingredients (steps S10 to S19 in FIG. 8).

[0223] As described above, according to the sixth embodiment, the inspection device 1E performs data augmentation on the target image data obtained by the operation of the learned first model NN1 using instance segmentation, and enlarges the training images used for the training of the second model NN2. Therefore, based on the good product images that are relatively easy to obtain on the food production line, a large number of defective product images that are relatively difficult to obtain can be generated. As a result, it is possible to prepare a sufficient number of training images related to defective products, and the classification accuracy of the learned second model NN2 can be further improved.

[0224] [Seventh Embodiment] Next, a seventh embodiment of the present invention will be described. In the following description, the same components as those in the first to sixth embodiments described above are denoted by the same reference numerals, and the description thereof is omitted.

[0225] In the first to sixth embodiments, the regions and types of a plurality of food ingredients are estimated using the learned first model NN1 that performs instance segmentation, and the first estimation result including the output image in which the estimated regions and types of food ingredients are associated with the image is output, and the case of performing an inspection regarding the arrangement of food ingredients has been described. Also, the arrangement relationship between food ingredients was classified using the learned second learning model NN2.

[0226] In contrast, in the seventh embodiment, as the first model NN1, any deep learning model such as R-CNN, YOLO, Mask R-CNN, SSD, etc., which is a model that performs object detection by surrounding the type (class) of food ingredients and the position within the image with a bounding box, can be used. Note that a bounding box is a boundary line or boundary region that surrounds an object within an image, and it is understood by those skilled in the art that it usually has a rectangular shape.

[0227] In this embodiment, the first model NN1 uses the image of food obtained by the acquisition unit 10 as an input or training image, and the output includes each food ingredient indicated by a label for each anchor box and grid and a region indicating the bounding box. More specifically, the output obtained by the operation of the learned first model NN1 includes, for example, the probability (score) of including the center of the food ingredient within the grid regardless of the label indicating the food ingredient, the class of the food ingredient which is the classification label of the center of the food ingredient included within the grid, and position information indicating the relative position (vertical, horizontal) and size (width, height) within the grid of the bounding box where the center of the food ingredient is located within the grid.

[0228] The first learning unit 18 uses the input image obtained by the acquisition unit 10 as a training image, and learns features capable of predicting the class of the food ingredient and the bounding box including the food ingredient from the training image.

[0229] The first estimation unit 11 can estimate the type of each food ingredient included in the input image and the bounding box surrounding the food ingredient by the operation of the learned first model NN1. The first estimation unit 11 estimates the relative arrangement relationship between a plurality of food ingredients and the type of the food ingredient, and outputs a first estimation result including an output image in which the estimated arrangement relationship is associated with the image.

[0230] The first determination unit 12 determines whether or not the arrangement relationship satisfies a preset criterion based on the first estimation result. Also, the first determination unit 12 can determine the presence or absence of each food ingredient from the estimation result by the first estimation unit 11.

[0231] Here, when the same type of food ingredients are composed of multiple ingredients, the first determination unit 12 can determine whether or not the relative arrangement relationship between the ingredients satisfies a standard set based on the four vertices or the center point of the bounding boxes of the estimated multiple ingredients. With such a configuration, the first determination unit 12 can determine the validity of the arrangement or the degree of scattering of the same type of food ingredients. For example, it becomes possible to determine whether shredded cheese arranged as a pizza topping is arranged at a certain position.

[0232] Based on the determination result by the first determination unit 12, the inspection unit 13 performs an inspection regarding the arrangement of the multiple ingredients arranged in the food container or the like.

[0233] As described above, according to the seventh embodiment, even when an object detection model that specifies the position of the food ingredients in the image and surrounds them with a bounding box is used as the first model NN1, it is possible to perform a food plating inspection including the presence or absence of the food ingredients arranged in the food container or the like and the validity of the arrangement relationship of the food ingredients.

[0234] [Usage example of the inspection device] Here, a usage example of the inspection device 1 according to the embodiment of the present invention will be described with reference to FIGS. 23 to 25.

[0235] FIG. 23 shows an example of an inspection screen displayed on the display device 108 provided in the inspection device 1. The image i on the left side of the screen shows the food photographed by the camera 105. The image s on the right side of the screen is an image obtained by synthesizing a color pattern image showing the estimated plating position of the food ingredients using instance segmentation and the original image i. In this example, the food variety is "Product B", and the food ingredients to be inspected are four types: "wasabi", "shredded nori", "cucumber", and "green onion".

[0236] On the inspection screen, the first criteria ("inspection items") and judgment results ("overall result", "recognized quantity", "recognized total area") set for each food ingredient are displayed. Also, on the inspection screen, the classification result of the arrangement positions among the food ingredients is shown ("location inspection (Class / Score)", "0.96076"). Further, the inspection result of the food to be inspected is displayed as "OK (qualified)" in the upper right corner of the screen. The second criteria regarding the arrangement positions among the food ingredients are displayed in the item "Score threshold".

[0237] As described above, in the inspection apparatus 1, by the user switching the learned first model NN1 and learned second model NN2, as well as the judgment criteria such as thresholds for their calculation results for each food variety, it is possible to perform plating inspections of a plurality of different foods with a single inspection apparatus 1. The user can, for example, click on the icon mc ("inspection setting change") displayed on the inspection screen, or perform a touch operation if the inspection screen is a touch panel, to perform setting changes such as switching the variety of the food to be inspected.

[0238] When the user touches or clicks on the icon mc displayed on the inspection screen shown in Fig. 23, before transitioning to the setting change screen, for example, it switches to a password request screen. When changing the food to be inspected to a food of another variety, the user performs an input operation using a keyboard or the like to input the user name and password. By performing such a password request, it is possible to limit the setting items that can be changed according to the authority given to the user.

[0239] Fig. 24 shows an example of the setting change screen displayed on the display device 108. When changing the variety of the food to be inspected, the user can input the desired food variety from the varieties in the pull-down list of the menu f ("file selection") displayed on the setting screen of the "inspection setting file". In the example of Fig. 20, it is selected from among the three food varieties of food "Product A", "Product B", and "Product C".

[0240] For example, when the user selects "Product B", the learned first model NN1 ("Segmentation model") and the learned second model NN2 ("Classification model") corresponding to the food "Product B" are called following from the memory 15. Further, the threshold value for each ingredient (the "inspection threshold value", the "inspection item") used as a judgment criterion for the calculation results of the learned first model NN1 and the learned second model NN2 is also called from the memory 15.

[0241] As shown in an example of the setting change screen in FIG. 25, the user selects "Product B" in the file selection menu f, and further determines the calling of the variety by the confirmation icon e. In the areas dl, ct, etc. of the setting change screen, the currently set learned first model NN1, learned second model NN2, the threshold value for each ingredient used as a judgment criterion, etc. are displayed. Also, in the example of FIG. 25, as a model used for inspecting the arrangement position between ingredients, in addition to the learned second model NN2, another position correction model prepared in advance can be selected. The user can select either the position correction model displayed in the directory in the area pc or the learned second model NN2 displayed in the area dl.

[0242] Also, the user can change the first criterion (the "inspection item") including the threshold value for each ingredient used as a judgment criterion for the calculation result of the learned first model NN1 by making a pull-down selection for each ingredient. For example, the user can select "number", "single area", "total area", combinations thereof, and "uninspected" from the pull-down menu for the "inspection item" for the ingredient "cucumber". Similarly, the user can change the threshold value set for the inspection item of each ingredient by making a pull-down selection.

[0243] When making setting changes such as changing the "inspection item" and changing the threshold value for each ingredient, the user can click the icon w to overwrite and save the settings, or click the icon n to create a new file. Also, the file that has become unnecessary can be deleted with the icon d.

[0244] The embodiments of the inspection apparatus and inspection method of the present invention have been described above. However, the present invention is not limited to the described embodiments, and various modifications that can be conceived by those skilled in the art can be made within the scope of the invention described in the claims.

[0245] For example, in the described embodiment, when the inspection process by the inspection apparatus is started, the case where the acquisition unit 10 acquires the image of the inspection target captured by the camera 105 according to the flow of food production in the production line has been described. However, the inspection apparatus can acquire the image of the manufactured food captured in the past and perform the inspection process not only when food is being manufactured on the food production line but also, for example, when the production line is not operating. Examples include the case of performing maintenance on the production line.

[0246] In addition, in the described embodiment, the inspected foods include processed noodles, frozen foods, boxed lunches, and prepared dishes that are manufactured on the production line and in which ingredients are arranged and served. Note that the inspected foods include not only those in which ingredients are arranged in a container but also those in which ingredients are arranged without using a container.

[0247] In addition, in the described embodiment, the case where the number of types of ingredients to be inspected is plural has been exemplified, but the number of types of ingredients may be singular.

[0248] Note that the learning apparatus that performs the learning processes of the first model NN1 and the second model NN2 can be configured separately and independently from the inspection apparatus.

[0249] In addition, the various functional blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented using a general-purpose processor, GPU, digital signal processor (DSP), application-specific integrated circuit (ASIC), FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described above.

[0250] It is possible to use a microprocessor as the general-purpose processor, but alternatively, a processor, controller, microcontroller, or state machine according to the prior art can also be used. The processor can also be implemented, for example, as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors connected to a DSP core, or a combination of computing devices in any such configuration.

Description of Reference Numerals

[0251] 1... inspection device, 10... acquisition unit, 11... first estimation unit, 12... first determination unit, 13... inspection unit, 14... presentation unit, 15... memory, 15A... first memory, 101... bus, 102... processor, 103... main storage device, 104... communication I / F, 105... camera, 106... auxiliary storage device, 107... input / output I / O, 108... display device, NW... network.

Claims

1. An acquisition unit that acquires an image including arranged food ingredients; A first estimation unit that uses a learned first model to estimate the region and type of the food ingredients in the image, and outputs a first estimation result including an output image in which the estimated region and type of the food ingredients are associated with the image; A first determination unit that determines, for each type of the food ingredients, whether the corresponding region satisfies a first criterion preset regarding the presence, absence, and excess or deficiency of the food ingredients based on the first estimation result; An inspection unit that performs an inspection regarding the arrangement of the food ingredients based on the determination result by the first determination unit; A second estimation unit that uses a learned second model with the output image included in the first estimation result as an input to estimate the relative arrangement relationship between a plurality of the food ingredients in the output image, and outputs a second estimation result in which the estimated arrangement relationship is associated with the output image; A second determination unit that determines whether the arrangement relationship satisfies a second criterion preset regarding the validity of the arrangement relationship based on the second estimation result and includes; The inspection unit performs an inspection regarding the arrangement of the food ingredients based on the determination result by the second determination unit An inspection device characterized by this.

2. In the inspection device according to Claim 1, The first determination unit specifies at least one of the number and size of the regions for each type of the food ingredients, and determines whether at least one of the number and size of the regions satisfies the first criterion An inspection device characterized by this.

3. In the inspection device according to Claim 1 or Claim 2, Furthermore, it includes a presentation unit that presents at least the inspection result by the inspection unit, The presentation unit presents at least any one of the information including the image, the first estimation result, the determination result by the first determination unit, the second estimation result, the determination result by the second determination unit, and the inspection result by the inspection unit An inspection device characterized by this.

4. In the inspection device according to any one of Claims 1 to 3, Furthermore, it includes a learning device that performs a learning process for constructing a preset model, The learning device uses an image including the arranged food ingredients as a first learning image, learns a first model of a neural network, extracts a first feature amount for identifying the region and type of the food ingredients, and obtains the learned first model with the extracted first feature amount. A first learning unit; A first storage unit that stores the learned first model acquired by the first learning unit is provided, The first estimation unit reads the learned first model from the first storage unit and estimates the region and type of the food ingredient in the image using the learned first model The inspection device is characterized by this

5. In the inspection device according to claim 4, The first model is an image recognition model by instance segmentation The inspection device is characterized by this

6. In the inspection device according to claim 4 or claim 5, The learning device uses the output image included in the first estimation result output by the first estimation unit as a second learning image to train a second model of a neural network, extracts a second feature amount for identifying the relative arrangement relationship between a plurality of the food ingredients in the second learning image, and obtains the learned second model using the extracted second feature amount. A second learning unit, A second storage unit that stores the learned second model acquired by the second learning unit The inspection device is further characterized by including this

7. In the inspection device according to claim 6, Furthermore, it includes a conversion unit that converts the output image included in the first estimation result to generate a more simplified converted image, The second learning unit uses the converted image as the second learning image to train the second model and obtains the learned second model The inspection device is characterized by this

8. In the inspection device according to claim 7, The output image included in the first estimation result is a color pattern image in which the corresponding region is painted with an arbitrary color for each type of the food ingredient, The converted image is a color pattern image in which each region of the color pattern image is replaced with an object of the same shape The inspection device is characterized by this

9. In the inspection device according to claim 7, The output image included in the first estimation result is a color pattern image in which the corresponding region is painted with an arbitrary color for each type of the food ingredient, The converted image is an image in which each region of the color pattern image is replaced with an object having a different shape for each color of the color pattern, and the color of the replaced object is set to a different color for each shape The inspection device is characterized by this

10. In the inspection device according to claim 7, The output image included in the first estimation result is a color pattern image in which the corresponding region is painted in an arbitrary color for each type of the food material. The converted image is an image obtained by replacing each region of the color pattern image with an object having a different shape for each color of the color pattern and making the colors of all the replaced objects the same color. An inspection apparatus characterized by the above.

11. In the inspection apparatus according to claim 6, further comprising a processing unit that generates a processed image obtained by processing the output image on the assumption that the inside of the longest estimated outer contour line of the region of the output image included in the first estimation result is the region. The second learning unit learns the second model using the processed image as the second learning image and obtains the learned second model. An inspection apparatus characterized by the above.

12. In the inspection apparatus according to any one of claims 6 to 11, the learning apparatus further comprises a data augmentation unit that performs data augmentation of a target image in which the region and type of the food material are associated with the image to generate the second learning image. The data augmentation unit generates the second learning image based on label information regarding the validity of the relative arrangement relationship between a plurality of the food materials in the target image, which is attached to the target image. An inspection apparatus characterized by the above.

13. In the inspection apparatus according to claim 12, the data augmentation unit performs data augmentation of a target image to which a first label indicating that the relative arrangement relationship between a plurality of the food materials in the target image is valid is attached, and generates the second learning image to which the first label is attached. An inspection apparatus characterized by the above.

14. In the inspection apparatus according to claim 12 or claim 13, the data augmentation unit performs data augmentation of a target image to which a first label indicating that the relative arrangement relationship between a plurality of the food materials in the target image is valid is attached, and generates the second learning image to which a second label indicating that the relative arrangement relationship between the plurality of the food materials is not valid is attached. An inspection apparatus characterized by the above.

15. In the inspection apparatus according to claim 12 or claim 14, the data augmentation unit performs data augmentation of a target image to which a second label indicating that the relative arrangement relationship between a plurality of the food materials is not valid is attached, and generates the second learning image to which the second label is attached. An inspection apparatus characterized by the above.

16. In the inspection apparatus according to any one of claims 12 to 15, further comprising an extraction unit that extracts the regions for each type of the food ingredients from the target image, while maintaining the relative arrangement relationship among the plurality of the food ingredients in the target image, the data expansion unit generates the second learning image by processing and then rearranging the regions extracted by the extraction unit. An inspection apparatus characterized by the above.

17. In the inspection apparatus according to any one of claims 12 to 15, further comprising an extraction unit that extracts the regions for each type of the food ingredients from the target image, the data expansion unit generates the second learning image by replacing the positions of the regions corresponding to two or more different types of the food ingredients among the regions extracted by the extraction unit. An inspection apparatus characterized by the above.

18. In the inspection apparatus according to any one of claims 12 to 15, further comprising an extraction unit that extracts the regions for each type of the food ingredients from the target image, the data expansion unit generates the second learning image by deleting one or more of the regions extracted by the extraction unit from the target image. An inspection apparatus characterized by the above.

19. In the inspection apparatus according to any one of claims 12 to 15, further comprising an extraction unit that extracts the regions for each type of the food ingredients from the target image, the data expansion unit performs at least any one of inversion, rotation, dilation, and contraction of each of the regions extracted by the extraction unit to perform data expansion of the target image. An inspection apparatus characterized by the above.

20. A first step of acquiring an image including arranged food ingredients, a second step of using a learned first model to estimate the regions and types of the food ingredients in the image, and outputting a first estimation result including an output image in which the estimated regions and types of the food ingredients are associated with the image, a third step of, based on the first estimation result, determining for each type of the food ingredients whether the corresponding region satisfies a first criterion preset regarding the presence / absence and excess / deficiency of the food ingredients, a fourth step of performing an inspection regarding the arrangement of the food ingredients based on the determination result in the third step. Using a trained second model that takes the output image included in the first estimation result as input, estimating the relative arrangement relationship between the plurality of food ingredients in the output image, and outputting a second estimation result that associates the estimated arrangement relationship with the output image; A sixth step of determining, based on the second estimation result, whether the arrangement relationship satisfies a second criterion preset with respect to the validity of the arrangement relationship; comprising; The fourth step performs an inspection regarding the arrangement of the food ingredients based on the determination result in the sixth step. An inspection method characterized by this.

21. In the inspection method according to claim 20, In the third step, for each type of food ingredient, at least one of the number and size of the regions is specified, and it is determined whether at least one of the number and size of the regions satisfies the first criterion. The fourth step performs an inspection regarding the arrangement of the food ingredients based on the determination result. An inspection method characterized by this.

22. In the inspection method according to claim 20 or claim 21, Further comprising a learning step of performing a learning process for constructing a preset model, The learning step is Using an image including the arranged food ingredients as a first learning image, training a first model of a neural network to extract a first feature amount for identifying the region and type of the food ingredients, and obtaining the trained first model based on the extracted first feature amount; a seventh step; An eighth step of storing the trained first model obtained in the seventh step in a first storage unit; comprising; The third step reads the trained first model from the first storage unit and estimates the region and type of the food ingredients in the image by the trained first model. An inspection method characterized by this.

23. In the inspection method according to claim 22, The learning step is Using the output image included in the first estimation result output in the third step as a second learning image, training a second model of a neural network to extract a second feature amount for identifying the relative arrangement relationship between the plurality of food ingredients in the second learning image, and obtaining the trained second model based on the extracted second feature amount; a ninth step; A tenth step of storing the trained second model obtained in the ninth step in a second storage unit; An inspection method characterized by further comprising

24. In the inspection method according to claim 23, further, a first 11th step of generating a processed image obtained by processing the output image is provided, assuming that the inside of the longest estimated outer contour line of the region of the output image included in the first estimation result is the region, The 9th step is to train the second model using the processed image as the second training image and obtain the trained second model An inspection method characterized by

25. In the inspection method according to claim 23 or claim 24, The training step further includes a 12th step of performing data augmentation on a target image in which the region and type of the foodstuff are associated with the image to generate the second training image, The 12th step generates the second training image based on label information regarding the validity of the relative arrangement relationship among the plurality of foodstuffs in the target image, which is attached to the target image An inspection method characterized by

26. In the inspection method according to claim 25, The 12th step performs data augmentation on a target image to which a first label indicating that the relative arrangement relationship among the plurality of foodstuffs in the target image is valid is attached, and generates the second training image to which the first label is attached An inspection method characterized by

27. In the inspection method according to claim 25 or claim 26, The 12th step performs data augmentation on a target image to which a first label indicating that the relative arrangement relationship among the plurality of foodstuffs in the target image is valid is attached, and generates the second training image to which a second label indicating that the relative arrangement relationship among the plurality of foodstuffs is not valid is attached An inspection method characterized by

28. In the inspection method according to claim 25 or claim 27, The 12th step performs data augmentation on a target image to which a second label indicating that the relative arrangement relationship among the plurality of foodstuffs is not valid is attached, and generates the second training image to which the second label is attached An inspection method characterized by

29. In the inspection method according to any one of claims 25 to 26, further includes a 13th step of extracting the region for each type of the foodstuff from the target image The 12th step generates the second learning image by repositioning the processed region extracted in the 13th step while maintaining the relative arrangement relationship among the plurality of the food materials in the target image. A inspection method characterized by the above.

30. In the inspection method according to any one of Claims 25 to 28, further comprising a 13th step of extracting the region for each type of the food material from the target image, the 12th step generates the second learning image by replacing the positions of the regions corresponding to two or more different types of the food materials among the regions extracted in the 13th step. A inspection method characterized by the above.

31. In the inspection method according to any one of Claims 25 to 28, further comprising a 13th step of extracting the region for each type of the food material from the target image, the 12th step generates the second learning image by deleting one or more of the regions extracted in the 13th step from the target image. A inspection method characterized by the above.

32. In the inspection method according to any one of Claims 25 to 28, further comprising a 13th step of extracting the region for each type of the food material from the target image, the 12th step performs at least any one of inversion, rotation, dilation, and contraction of each of the regions extracted in the 13th step to perform data augmentation of the target image. A inspection method characterized by the above.

33. In the inspection method according to any one of Claims 20 to 32, further comprising a 14th step of presenting at least the inspection result in the 4th step, the 14th step presents at least any one of the information including the image, the first estimation result, the judgment result in the 3rd step, and the inspection result in the 4th step. A inspection method characterized by the above.

Images