Processing method, program and processor

Abstract

To provide a processing method capable of efficiently discriminating the type of an object mounted on a tray.SOLUTION: A processing method includes an acquisition step of acquiring an image of an object mounted on a tray, and a discrimination step of discriminating the type of the object indicated by the image, on the basis of a learning model. The discrimination step discriminates the type of the object on the basis of a plurality of learning models corresponding to each of a plurality of types. The discrimination step calculates a degree of confidence for each type of the corresponding object, on the basis of the learning models. The discrimination step outputs the type of the object having the largest calculated degree of confidence as a discrimination result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a processing method, a program, and a processing device for determining the type of an object based on a learning model. [Background technology]

[0002] Patent Document 1 discloses a meat discrimination device that includes a camera that photographs a portion of meat (a chunk of meat) to obtain image information, a trained model that inputs the image information from the camera and outputs candidate parts of the meat, and a control unit that discriminates the part of the portion of meat photographed by the camera based on the candidate parts output from the trained model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-149142 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned meat discriminating device does not consider, for example, a technique for discriminating the part of meat that has already been cut and placed on a tray.

[0005] Therefore, in one aspect, an object of the present disclosure is to provide a processing method and the like that can efficiently distinguish the type of object placed on a tray. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, an acquisition step of acquiring an image of the object placed on the tray; a discrimination step of discriminating the type of the object shown in the image based on a learning model; A processing method is provided, comprising: [Effects of the Invention]

[0007] In one aspect, the present disclosure makes it possible to efficiently identify the type of object placed on a tray. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a device for taking images of meat on a tray. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of a processing system for executing a processing method according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram illustrating an example of an image acquired by the processing device. [Figure 4] FIG. 10 is a diagram illustrating examples of multiple types of parts. [Figure 5] FIG. 10 is a diagram illustrating a discrimination result in the processing device. [Figure 6] FIG. 4 is a diagram showing the boundary between the meat and the tray recognized in the image shown in FIG. 3. DETAILED DESCRIPTION OF THE INVENTION

[0009] In this embodiment, a processing method for identifying the part of meat placed on a tray as an object will be described. However, in the processing method of the present disclosure, the object is arbitrary, and includes other foods such as seafood, vegetables, and fruits, as well as non-food items. Furthermore, although this embodiment illustrates a case in which the part of meat is identified as the type of object, the attribute of the type of object is arbitrary. Furthermore, the shape of the tray is also arbitrary.

[0010] (Acquisition process) First, an acquisition process for acquiring an image of meat 80 placed on a tray 90 will be described. In this embodiment, the tray 90 is a container made of, for example, plastic and used when selling various foods. Usually, meat is packaged in a packaging material such as film, and a label indicating the part, price, etc. is attached to the packaging material before being sold.

[0011] FIG. 1 is a diagram illustrating an example of a device for taking an image of meat placed on a tray, and FIG. 2 is a diagram illustrating an example of the configuration of a processing system for carrying out the processing method of this embodiment.

[0012] As shown in Figures 1 and 2, tray 90 on which meat 80 is placed is transported to a predetermined position (the position shown in Figure 1) by transport device 200 such as a belt conveyor, and is photographed by camera 10A and camera 10B. As shown in Figure 1, when tray 90 is transported to the predetermined position, camera 10A and camera 10B sequentially take images of meat 80 placed on tray 90 from different angles. In addition, an illumination device 20A used when taking pictures with camera 10A is provided near camera 10A, and an illumination device 20B used when taking pictures with camera 10B is provided near camera 10B.

[0013] As shown in FIG. 2, camera 10A, camera 10B, lighting device 20A, lighting device 20B, and conveying device 200 are each connected to a processing device 50, which is a computer running a predetermined program, and their operations are controlled by the processing device 50.

[0014] The images captured by the camera 10A and the camera 10B are acquired by the processing device 50.

[0015] As shown in FIG. 2, the processing device 50 is connected to an output device 30 that outputs the discrimination results of the processing device 50, and an input unit 51 that accepts input operations by an operator.

[0016] FIG. 3 is a diagram illustrating an image acquired by the processing device.

[0017] Figure 3 shows an image captured by camera 10A. As shown in Figure 3, the image includes an area of meat 80 and an area of tray 90. The image captured by camera 10B also includes an area of meat 80 and an area of tray 90. Note that each of the images captured by camera 10A and camera 10B only needs to include a portion of tray 90, for example, at least a portion of the bottom surface 91 of tray 90.

[0018] (Discrimination process) Next, the discrimination step for discriminating the part of the meat 80 will be described.

[0019] Based on the learning model, the processing device 50 discriminates the part of the meat 80. Specifically, the processing device 50 performs inference based on the learning model and discriminates the part of the meat 80 from among a plurality of types of parts.

[0020] FIG. 4 is a diagram illustrating examples of parts of a plurality of types of meat.

[0021] In the example of FIG. 4, beef parts include cubes and diced thigh, chicken parts include wings and diced thigh, and pork parts include shredded meat, thigh offcuts, shoulder loin, and shoulder meat.

[0022] In this embodiment, a learning model corresponding to each of these eight parts is prepared, and the processing device 50 executes inference based on the corresponding learning model to determine (calculate) a confidence level indicating the likelihood that the meat 80 to be distinguished corresponds to each part. During the determination, eight learning models corresponding to the eight parts are used, and the confidence levels corresponding to each part are determined in parallel. Then, the processing device 50 determines, as a result of the determination, that the part with the highest confidence level is the part of the meat 80 to be distinguished.

[0023] FIG. 5 is a diagram illustrating a discrimination result in the processing device.

[0024] In the example of Fig. 5, eight parts are displayed on a display screen serving as output device 30 in descending order of the degree of confidence in their judgment, starting from the top. In Fig. 5, the part with the highest degree of confidence in its judgment is pork shoulder loin, and the confidence level for this is shown to be 99%. In other words, it is shown that meat 80 has been determined to be pork shoulder loin. Furthermore, the part with the second highest degree of confidence in its judgment is pork shoulder meat, and the confidence level for this is shown to be 15%. Furthermore, it is shown that the other parts have even lower degrees of confidence in their judgment.

[0025] The learning model used in the discrimination process can be specified based on an operator's operation on the input unit 51, etc. This can speed up the discrimination process. By inputting only the parts that require discrimination (classification), only the necessary learning model is selected, and it is also possible to avoid the execution of unnecessary processing during inference. For example, if it is known that the meat 80 is beef, it may be possible to specify only beef. It may also be possible to switch between an operating mode that uses all learning models and an operating mode that allows specification of some learning models by operating the input unit 51.

[0026] (Creating a learning model) Next, the creation of a learning model will be described.

[0027] As described above, in this embodiment, a learning model is prepared for each part of meat. Therefore, when creating a learning model, an independent learning model is created for each part of meat. For example, a learning model for pork shoulder loin is created using an image labeled as an image of pork shoulder loin placed on tray 90 as a training image. The same applies to other parts.

[0028] Multiple learning models can be combined in various ways. For example, multiple learning models corresponding to specific body parts can be combined and used. Also, a common learning model for multiple body parts may be prepared. Combining learning models can speed up inference, improve inference accuracy, and reduce the number of learning models.

[0029] The learning model can be created using any device, but for example, the processing device 50 can be used. In this case, the image captured by the camera 10A or the camera 10B can be acquired as a teacher image by the processing device 50. Furthermore, by using the image acquired in the acquisition step and used in the discrimination step as the teacher image, the discrimination accuracy can be improved.

[0030] Furthermore, in the processing device 50 or another device, for example, the boundary 81A (FIG. 6) between the meat 80 and the tray 90 and the outline of the tray 90 in the acquired image (FIG. 3) may be automatically recognized by image recognition processing.

[0031] FIG. 6 is a diagram showing the boundary between the meat and the tray recognized in the image shown in FIG.

[0032] In Figure 6, line 81A indicates the contour 81 (Figure 3) of the meat 80 recognized by image recognition processing, i.e., the boundary between the meat 80 and the tray 90. By recognizing line 81A and the contour of the tray 90, the area of the meat 80 and the area of the tray 90 in the image can be recognized, respectively. The recognition results of these boundaries, contours, or areas can be used as annotations for images in learning. Furthermore, the recognition results of the boundaries, contours, or areas recognized by image recognition processing may be corrected in accordance with input operations to the input unit 51 by an operator who views the image.

[0033] (Effects of this embodiment) In this embodiment, the part of the meat 80 can be identified based on an image of the meat 80 placed on the tray 90. This allows for greater flexibility in the process of selling the meat 80, for example. Also, it is no longer necessary to identify the part of the meat before it is placed on the tray 90, i.e., at the block meat stage or after it has been cut.

[0034] The process for selling meat 80 involves, for example, taking out a block (a chunk of meat), cutting it, arranging the meat 80 on a tray 90, wrapping it in film, and attaching a label. In this case, for example, the meat 80 cannot be packaged sequentially after it has been arranged; the temperature of the meat 80 rises between cutting and arranging it, so the meat 80 must be placed in a refrigerator before packaging. Therefore, after being placed in the refrigerator, the meat 80 is packaged and labeled all at once on the tray 90, but there is a possibility of worker error (e.g., wrong type of meat). For this reason, it is necessary to identify the part of the meat after it has been arranged on the tray.

[0035] Also, due to equipment limitations, it is possible that multiple pieces of meat 80 may be handled by one packaging machine. Meat cutting machines vary depending on the type of meat itself (chicken, pork, beef, etc.) and the cutting method, so adding a packaging process to all of them may not be practical in terms of space or cost. For example, the same packaging machine can be used for the packaging process regardless of the cut of meat 80, so there is no need for a large number of packaging machines. Therefore, even in such cases, it is necessary to identify the cut after the meat is served on the tray.

[0036] According to this embodiment, since the part of the meat 80 can be identified after it has been placed on the tray, it is possible to deal with such cases.

[0037] Furthermore, in this embodiment, the tray 90 carrying the meat 80 can be used as a guide during learning and inference (classification). This allows for reduced learning data and improves the accuracy of part identification. This eliminates the need for a wide variety of learning data (teacher images) that would be required for learning based on images of only the meat 80.

[0038] For example, meat 80 has an irregular shape, but tray 90 has a regular shape, so tray 90 can be easily recognized. By recognizing the area of tray 90 in the image and then learning and inferring, the area and orientation of meat 80 can be easily recognized. There is a certain pattern to how meat 80 is placed relative to the direction of tray 90 (for example, the longitudinal direction), so the direction of tray 90 serves as a guide.

[0039] Furthermore, because the tray 90 can be used as a guide, there is no need to acquire a large number of images of the meat 80 at different angles. For example, if learning is based on images of only the meat 80, a large number of images at different angles would be required. In contrast, in this embodiment, if the angle of the tray 90 is misaligned, it is sufficient to adjust the angle of the entire image based on the recognized angle of the tray 90 and perform inference. To take advantage of this advantage, a tray 90 with a directional shape, such as a rectangular or elliptical shape with a longitudinal direction, can be used.

[0040] When different types of trays 90 are used depending on the part of the meat 80, the type of tray 90 itself serves as a useful guide.

[0041] In this embodiment, the color of the tray 90 is set to a color that has optical contrast with the color of the meat 80, so that the line 81A can be easily recognized. This improves the accuracy of discrimination. By using a tray 90 of an appropriate color depending on the object, an appropriate contrast can always be obtained.

[0042] In this embodiment, a learning model is prepared for each part of the meat 80, and the determination process using multiple learning models is executed in parallel for each part of the meat 80. This speeds up inference and improves the determination accuracy for each part, resulting in a significant improvement in the accuracy of part identification. To identify the part of the meat 80 based on a single learning model, it is necessary to calculate the probability that the meat 80 is each part using only that single learning model. For example, it is necessary to calculate a 70% probability that the meat 80 is chuck loin and a 10% probability that the meat is shoulder meat. However, in practice, it is difficult to construct a learning model that consistently maintains an appropriate order of probability magnitude, and modifying the learning model is also not easy. In contrast, when multiple learning models are used, as in this embodiment, each learning model can be constructed by learning only the individual parts, which has the advantage of making it easier to optimize and modify the learning model.

[0043] (Variation) When acquiring images in the discrimination process or teacher images when creating a learning model, the intensity of the illumination light from illumination device 20A or illumination device 20B can be controlled by processing device 50. For example, by capturing images with camera 10A or camera 10B while changing the intensity of the illumination light and acquiring multiple images, it is possible to improve the discrimination accuracy in the discrimination process and the accuracy of the learning model. Furthermore, the intensity of the illumination light from illumination device 20A or illumination device 20B may be adjusted to prevent the influence of reflected light from tray 90 during learning or inference.

[0044] In this embodiment, the accuracy of discrimination can be improved by using multiple cameras 10A and 10B to capture images of meat 80 placed on tray 90 from multiple directions and outputting inference results (discrimination results) based on each captured image. Meat 80 placed on tray 90 is smaller than a whole piece of meat, and surface patterns can make it difficult to discern the type (part) of meat, but this method can improve discrimination accuracy. However, only images captured by a single camera, for example, camera 10A, may be used during learning and inference.

[0045] Furthermore, a sheet member that has an appropriate optical contrast with the meat 80 may be placed between the bottom surface 91 of the tray 90 and the meat 80. In this case, a sheet member of an appropriate color can be selected regardless of the color of the bottom surface 91, making it easy to recognize the boundary between the meat 80 and the sheet member. In this case, the entire configuration including the sheet member and tray 90 functions as a "tray" in the present disclosure.

[0046] Furthermore, images of the tray 90 and the meat 80 in the tray 90 may be learned during learning. For example, it is possible to learn that the white part of the area sandwiched between areas of meat 80 (the white part visible through the gap between adjacent pieces of meat 80) is the tray 90 (bottom surface 91). In this case, for example, if an image similar to this pattern appears during inference, that area can be prevented from being used for discrimination, which can improve the accuracy of inference.

[0047] As described above, according to this embodiment, it is possible to identify the part of the meat 80 based on an image of the meat 80 placed on the tray 90. This makes it possible to increase the degree of freedom in the process of selling the meat 80, for example. Furthermore, in this embodiment, the tray 90 on which the meat 80 is placed can be used as a guide during learning and inference (identification). This makes it possible to reduce the amount of learning data and improve the efficiency of identification during part identification.

[0048] Although each embodiment has been described in detail above, it is not limited to a specific embodiment, and various modifications and alterations are possible within the scope of the claims. It is also possible to combine all or a plurality of the components of the above-described embodiments. In the processing method of the present disclosure, the object is arbitrary, and includes other foods such as seafood, vegetables, and fruits, as well as non-food items. Furthermore, in this embodiment, a case where meat cuts are identified as the type of object is shown, but the attributes of the type of object are arbitrary. Furthermore, the shape of the tray is also arbitrary. [Explanation of symbols]

[0049] 10A, 10B camera 20A, 20B lighting equipment 30 Output Devices 50 Processing equipment 51 Input section 80 Meat 90 trays

Claims

1. an acquisition step of acquiring an image of the object placed on the tray; a discrimination step of discriminating the type of the object shown in the image based on a learning model; A processing method comprising:

2. The processing method according to claim 1 , wherein the type of the object is determined based on a plurality of learning models corresponding to a plurality of types, in the determining step.

3. The processing method according to claim 2 , wherein the determining step calculates a degree of confidence for each type of the corresponding object based on the learning model.

4. The processing method according to claim 3 , wherein in the discrimination step, the type of the object for which the calculated confidence level is greatest is output as a discrimination result.

5. The processing method according to claim 1 , wherein an image in which a boundary between the object and the tray is recognized is used during learning to create the learning model.

6. The processing method according to claim 1 , wherein an image in which the tray area is recognized is used during learning to create the learning model.

7. The processing method according to claim 1 , wherein images acquired while adjusting the intensity of illumination light are used during learning for creating the learning model or in the discrimination step.

8. The processing method according to claim 1 , wherein the determining step determines the type of the object using the images captured from a plurality of directions in the obtaining step.

9. an acquisition step of acquiring an image of the object placed on the tray; a discrimination step of discriminating the type of the object shown in the image based on a learning model; A program that causes a computer to execute the following.

10. an acquisition step of acquiring an image of the object placed on the tray; a discrimination step of discriminating the type of the object shown in the image based on a learning model; A processing unit that executes the above.

Images