Learning completed model generation method and device, commodity identification method and device

Abstract

The present invention relates to a method and device for generating a learned model, and a method and device for distinguishing a product. The measuring system (1) of the present invention comprises a server (200) for distinguishing the type of a product from a product image containing the product, and a measuring device (100) for distinguishing the type of a product based on an object image containing the product. The server (200) comprises: an acquisition unit for acquiring a product image and product information related to the type of the product; a segmentation unit for segmenting the product image into a plurality of regions and acquiring a plurality of segmented images; an extraction unit for extracting a plurality of segmented images that meet a specified condition related to the amount of product captured from the plurality of segmented images; and a generation unit for performing machine learning based on the plurality of segmented images extracted by the extraction unit to generate a discrimination model. The measuring device (100) comprises a discrimination unit (33) for acquiring a processing result processed using the discrimination model and distinguishing the type of the product based on the processing result.

Description

Technical Field

[0001] One aspect of the present invention relates to a learned model generating method, a learned model generating device, a commodity identifying method, a commodity identifying device, and a measuring device. Background Art

[0002] Currently, there are known methods for identifying commodities based on images containing commodities. For example, in the commodity identification method described in Patent Document 1 (Japanese Patent Publication No. 2015-64628), feature quantities of commodity images are extracted and compared with feature quantities in learning data to identify the type of commodity. Summary of the invention

[0003] As commodities sold in supermarkets, there are salads, side dishes, etc., for example. Salads, side dishes, etc. contain a mixture of multiple foods, and the shapes are not fixed. Thus, when the shape of the commodity is irregular, it is difficult to obtain the feature value of the commodity. Therefore, the discrimination accuracy of the commodity may be reduced.

[0004] An object of one aspect of the present invention is to provide a learned model generation method, a learned model generation device, a commodity identification method, a commodity identification device, and a measuring device that achieve improved commodity identification accuracy.

[0005] One aspect of the present invention involves a method for generating a learned model, which is a method for generating a discrimination model for discriminating the type of a product based on a product image containing the product, comprising: an acquisition step of acquiring a product image and product information related to the type of the product; a segmentation step of segmenting the product image into multiple areas to obtain multiple segmented images; an extraction step of extracting a specified segmented image from the multiple segmented images based on specified conditions related to the amount of product captured; and a generation step of associating the multiple segmented images extracted in the extraction step with the product information to perform machine learning, thereby generating a discrimination model.

[0006] In a learned model generation method according to one aspect of the present invention, a specified segmented image is extracted from a plurality of segmented images according to specified conditions related to the amount of goods captured. The segmented images include images that capture only goods, images that capture goods and objects other than goods (containers, backgrounds, etc.), and images that capture only objects other than goods. In the learned model generation method, segmented images are extracted from the above-mentioned segmented images according to specified conditions, and machine learning is performed based on the extracted segmented images. Thus, in the learned model generation method, machine learning based on appropriate teaching images can be performed. Therefore, in the learned model generation method, a discrimination model that improves the discrimination accuracy of goods can be generated.

[0007] In one embodiment, in the extraction step, multiple segmented images that meet the specified conditions can also be extracted from multiple segmented images. In this method, machine learning can be performed based on appropriate teaching images. Therefore, in the learning model generation method, a discrimination model that improves the discrimination accuracy of commodities can be generated.

[0008] In one embodiment, in the extraction step, segmented images in which the amount of objects other than the goods is below a threshold value may be extracted. In other words, in the extraction step, segmented images in which the amount of objects other than the goods is greater than the threshold value are extracted. In this method, segmented images with a high occupancy rate of goods are extracted. That is, in the learned model generation method, segmented images in which containers or backgrounds other than the goods are captured can be excluded. Therefore, in the learned model generation method, machine learning can be performed based on segmented images in which goods are captured. Therefore, in the learned model generation method, a discrimination model that improves the discrimination accuracy of goods can be generated.

[0009] In one embodiment, in the extraction step, an extraction model generated by machine learning based on images that do not contain products may be used to obtain non-product likelihoods representing the probability of being segmented images that do not contain products for each of the plurality of segmented images, and extract segmented images whose non-product likelihoods are below a threshold. In this method, only segmented images that contain products can be appropriately extracted.

[0010] In one embodiment, the generation of the extraction model may also include: a first step of obtaining a non-product image that does not contain a product; a second step of segmenting the non-product image into multiple regions to obtain multiple non-product segmented images; and a third step of performing machine learning based on the multiple non-product segmented images to generate an extraction model. Thus, in the method for generating a learned model, the extraction model can be appropriately generated.

[0011] In one embodiment, in the segmentation step, the product image may be segmented in such a way that the regions of the plurality of segmented images are rectangular and all of the same size; and in the second step, the non-product image may be segmented in such a way that the plurality of non-product segmented images are respectively of the same shape and size as the segmented image. In this method, since the segmented image and the non-product segmented image are of the same shape and size, there is no need to perform a conversion process for converting the shape and size of the image. Therefore, in the method for generating a learned model, a reduction in processing load is achieved.

[0012] In one embodiment, in the extraction step, one segmented image in which the amount of images other than the product is below a threshold and other segmented images in which the amount of images other than the product is greater than the threshold are extracted from multiple segmented images, so that the product information is associated with one segmented image, and non-product information other than the product is associated with other segmented images; in the generation step, machine learning is performed based on the segmented image extracted in the extraction step and other segmented images to generate a discriminant model. In this method, machine learning is performed based on segmented images marked with product information and other images marked with non-product information. In this way, since machine learning is performed by distinguishing (classifying) segmented images in which products are captured and segmented images in which the amount of products captured is small (not captured), a discriminant model that improves the accuracy of product discrimination can be generated.

[0013] In one embodiment, in the extraction step, based on the feature quantity of the pixels of the image not including the product, the segmented images whose pixel inclusion ratio is below a predetermined threshold value are extracted from the plurality of segmented images. In this method, only the segmented images that include the product can be appropriately extracted.

[0014] In one embodiment, in the segmentation step, the product image may be segmented in such a way that one segmented image overlaps at least a portion of the other segmented images. In this method, when segmenting the product image, even if one item included in the product is cut off in one segmented image, the other segmented image may include the item completely. Therefore, in the method for generating a learned model, since machine learning can be performed based on an appropriate teaching image, a discrimination model that improves the discrimination accuracy of the product can be generated.

[0015] In one embodiment, in the segmentation step, the segmentation region of the product image may be set in such a manner that the other regions are moved by a predetermined amount in a first direction or a second direction orthogonal to the first direction, which is a direction in which pixels of the product image are arranged, relative to one region. In this method, by moving the region by a predetermined amount, the product image can be segmented in such a manner that each pixel is included in each region arranged along the moving direction the same number of times.

[0016] In one embodiment, in the segmentation step, the product image may be segmented in such a way that the regions of the plurality of segmented images are rectangular and all have the same size. In this method, since the segmented images are all of the same shape and size, there is no need to perform a conversion process for converting the shape and size of the image. Therefore, in the method for generating a learned model, a reduction in processing load is achieved.

[0017] In one embodiment, the method may include a cutting step, comparing the product image with a base image that does not include the product, and cutting out a product area that at least includes the entire product and is smaller than the product image from the product image; and in a segmentation step, segmenting the product area into a plurality of areas to obtain a plurality of segmented images. In this method, since the product area that excludes the part other than the product is used to segment the product image, it is not necessary to segment unnecessary areas. Therefore, in the method for generating a model after learning, a reduction in processing load is achieved.

[0018] One aspect of the present invention involves a learned model generation device, which is a generation device of a discrimination model for discriminating the type of a product based on a product image containing the product, and comprises: an acquisition unit, which acquires the product image and product information related to the type of the product; a segmentation unit, which segments the product image into multiple areas and obtains multiple segmented images; an extraction unit, which extracts multiple segmented images from the multiple segmented images based on specified conditions related to the amount of product captured; and a generation unit, which performs machine learning based on the multiple segmented images extracted by the extraction unit to generate a discrimination model.

[0019] In a learned model generation device involved in one aspect of the present invention, a specified segmented image is extracted from a plurality of segmented images based on specified conditions related to the amount of goods captured. The segmented images include images that capture only goods, images that capture goods and objects other than goods (containers, backgrounds, etc.), and images that capture only objects other than goods. In the learned model generation device, segmented images are extracted from the above-mentioned segmented images based on specified conditions, and machine learning is performed based on the extracted segmented images. As a result, in the learned model generation device, machine learning can be performed based on appropriate teaching images. Therefore, in the learned model generation device, a discrimination model that improves the discrimination accuracy of goods can be generated.

[0020] One aspect of the present invention involves a commodity discrimination method that uses a discrimination model generated by the above-mentioned learned model generation method to discriminate the type of a commodity based on an object image containing the commodity, and includes: a first acquisition step, dividing the object image into multiple areas and acquiring multiple segmented object images; a second acquisition step, acquiring multiple segmented object images that satisfy specified conditions related to the amount of commodity captured from the multiple segmented object images; and a discrimination step, acquiring a processing result processed by the discrimination model for the multiple segmented object images acquired in the second acquisition step, and discriminating the type of the commodity based on the processing result.

[0021] In a commodity identification method according to one aspect of the present invention, the above-mentioned discrimination model is used to discriminate the type of commodity. Therefore, in the commodity identification method, the accuracy of commodity identification is improved. In addition, in the commodity identification method, multiple segmented object images that meet specified conditions related to the amount of commodity captured are obtained from multiple segmented object images. The segmented object images include images that only capture the commodity, images that capture the commodity and objects other than the commodity (containers, backgrounds, etc.), and images that only capture objects other than the commodity. In the commodity identification method, a segmented object image that meets the specified conditions is obtained from the above-mentioned segmented object images, and the obtained segmented object image is processed using the discrimination model. As a result, in the commodity identification method, processing can be performed according to an appropriate segmented object image. Therefore, in the commodity identification method, the accuracy of commodity identification is improved.

[0022] In one embodiment, in the second acquisition step, a segmentation target image in which the amount of objects other than the product is below a threshold value can be acquired. In this method, images with a high occupancy rate of the product are acquired. That is, images in which containers or backgrounds other than the product are captured can be excluded. Therefore, in the product identification method, the accuracy of product identification is improved.

[0023] In one embodiment, in the second acquisition step, the acquisition model generated by machine learning based on the image not containing the product can be used to acquire non-product likelihoods indicating the probability of being a segmentation target image not containing the product for each of the plurality of segmentation target images, and acquire segmentation target images whose non-product likelihoods are below a threshold. In this method, only segmentation target images containing the product can be appropriately acquired.

[0024] In one embodiment, in the first acquisition step, the target image may be segmented in such a manner that the regions of the plurality of segmented target images are rectangular and all have the same size. In this method, since the segmented target images are all of the same shape and size, there is no need to perform a conversion process for converting the shape and size of the image. Therefore, in the commodity identification method, a reduction in processing load is achieved.

[0025] An aspect of the present invention relates to a commodity discrimination method that uses a discrimination model generated by the above-mentioned learned model generation method to discriminate the type of a commodity based on an object image containing the commodity, and includes: a segmented image acquisition step of segmenting the object image into multiple regions to obtain multiple segmented object images; and a discrimination step of obtaining, for the multiple segmented object images obtained in the segmented image acquisition step, a commodity likelihood indicating the probability of being a commodity and a non-commodity likelihood indicating the probability of being something other than a commodity from the processing results of the processing using the discrimination model, and discriminating the type of the commodity based on the commodity likelihood.

[0026] In a commodity identification method according to one aspect of the present invention, the above-mentioned discrimination model is used to discriminate the type of commodity. Therefore, in the commodity identification method, the accuracy of commodity identification is improved. In addition, in the commodity identification method, the commodity likelihood indicating the probability of being a commodity and the non-commodity likelihood indicating the probability of being other than a commodity are obtained from the processing result processed by the discrimination model, and the type of commodity is discriminated based on the commodity likelihood. Therefore, in the commodity identification method, the accuracy of commodity identification is improved.

[0027] In one embodiment, in the discrimination step, the plurality of segmented object images may be processed using the discriminant model to obtain a plurality of processing results, and the types of commodities may be discriminated based on the plurality of processing results. In this method, since the discriminant model is used to process each segmented object image and a plurality of processing results are obtained, the discrimination accuracy of commodities is further improved.

[0028] In one embodiment, in the discrimination step, the processing results may be weighted according to the degree of the likelihood of the product indicating the probability that the product is of a certain type, and the type of the product may be discriminated by majority decision of the weights assigned to the processing results. In this method, the product can be discriminated with higher accuracy.

[0029] In one embodiment, in the discrimination step, when the target image includes a plurality of commodities, the categories of the plurality of commodities may be discriminated based on the processing results of each region including each of the plurality of commodities. In this method, the categories of the plurality of commodities included in the target image may be discriminated.

[0030] One aspect of the present invention involves a product discrimination device that uses a discrimination model generated by the above-mentioned learned model generation method to discriminate the type of a product based on an object image containing the product, and comprises: a first acquisition unit that divides the object image into multiple areas and obtains multiple segmented object images; a second acquisition unit that obtains multiple segmented object images that satisfy specified conditions related to the amount of product captured from the multiple segmented object images; and a discrimination unit that obtains processing results processed by the discrimination model for the multiple segmented object images obtained by the second acquisition unit, and discriminates the type of the product based on the processing results.

[0031] In a commodity identification device according to one aspect of the present invention, the above-mentioned discrimination model is used to discriminate the type of commodity. Therefore, in the commodity identification device, the accuracy of commodity identification is improved. In addition, in the commodity identification device, multiple segmented object images that meet specified conditions related to the amount of commodity captured are obtained from multiple segmented object images. The segmented object images include images that only capture the commodity, images that capture the commodity and objects other than the commodity (containers, backgrounds, etc.), and images that only capture objects other than the commodity. In the commodity identification device, the segmented object images that meet the specified conditions are obtained from the above-mentioned segmented object images, and the obtained segmented object images are processed using the discrimination model. As a result, in the commodity identification device, processing can be performed according to appropriate segmented object images. Therefore, in the commodity identification device, the accuracy of commodity identification is improved.

[0032] One aspect of the present invention relates to a product discrimination system comprising a generation device of a discrimination model for discriminating the type of a product based on a product image containing the product, and a discrimination device for discriminating the type of a product based on an object image containing the product, wherein the generation device comprises: an acquisition unit for acquiring a product image and product information related to the type of the product; a segmentation unit for segmenting the product image into multiple regions and acquiring multiple first segmented images; an extraction unit for extracting multiple first segmented images that satisfy a specified condition related to the amount of product captured from the multiple first segmented images; and a generation unit for associating the multiple first segmented images extracted by the extraction unit with the product information to perform machine learning and generate a discrimination model, wherein the discrimination device comprises: a first acquisition unit for segmenting the object image into multiple regions and acquiring multiple second segmented images; a second acquisition unit for acquiring multiple second segmented images that satisfy the specified condition related to the amount of product captured from the multiple second segmented images; and a discrimination unit for acquiring a processing result processed by the discrimination model for the multiple second segmented images acquired by the second acquisition unit, and discriminating the type of the product based on the processing result.

[0033] In a generating device of a commodity identification system involved in one aspect of the present invention, a plurality of first segmented images satisfying prescribed conditions related to the amount of commodities captured are obtained from a plurality of segmented images. The first segmented images include images that capture only commodities, images that capture commodities and objects other than commodities (containers, backgrounds, etc.), and images that capture only objects other than commodities. In the generating device, the first segmented images satisfying prescribed conditions are extracted from the above-mentioned first segmented images, and machine learning is performed based on the extracted first segmented images. Thus, in the generating device, machine learning can be performed based on appropriate teaching images. Therefore, in the generating device, a discrimination model that improves the accuracy of commodity discrimination can be generated.

[0034] In addition, in the discrimination device of the commodity discrimination system, a plurality of segmented images satisfying a prescribed condition related to the amount of commodity captured are obtained from the plurality of second segmented images. In the discrimination device, a second segmented image satisfying the prescribed condition is obtained from the second segmented images, and the obtained second segmented image is processed using the discrimination model. Thus, in the discrimination device, processing can be performed based on an appropriate second segmented image. Therefore, in the commodity discrimination system, the discrimination accuracy of the commodity is improved.

[0035] One aspect of the present invention relates to a measuring device that uses a discriminant model generated by the above-mentioned learned model generation method to discriminate the type of a commodity based on an object image containing the commodity and calculate the price of the commodity. The device comprises: a measuring unit that measures the weight of the commodity; a photographing unit that photographs the commodity; a discriminant unit that obtains a processing result processed by the discriminant model for the object image photographed by the photographing unit and discriminates the type of the commodity based on the processing result; and a calculation unit that calculates the price of the commodity based on the weight of the commodity measured by the measuring unit and the type of the commodity discriminated by the discriminant unit.

[0036] In a weighing device according to one aspect of the present invention, the type of a commodity is discriminated using the above-mentioned discrimination model. Therefore, in the weighing device, it is possible to improve the accuracy of discriminating commodities.

[0037] According to one aspect of the present invention, it is possible to improve the accuracy of commodity identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a diagram showing the structure of a measuring instrument according to one embodiment.

[0039] Figure 2 It is a three-dimensional diagram showing a measuring instrument.

[0040] Figure 3 It is a figure which shows the structure of a measuring device.

[0041] Figure 4 It is a diagram showing the configuration of a control device.

[0042] Figure 5 This is a diagram showing a commodity discrimination method in the discrimination unit.

[0043] Figure 6 A diagram showing an object image including a product.

[0044] Fig. 7A is a diagram showing an image to be segmented.

[0045] Figure 7B is a diagram showing an image to be segmented.

[0046] Figure 7C is a diagram showing an image to be segmented.

[0047] Fig.7D is a diagram showing an image to be segmented.

[0048] Figure 8 is a graph representing a neural network.

[0049] Fig. 9 is a graph representing a neural network.

[0050] Fig.10 This is a diagram showing a method for generating an extraction model by a learned model generating unit.

[0051] Fig.11 A diagram showing a non-product image that does not include a product.

[0052] Fig.12 This is a diagram showing a method of generating a discrimination model by a learned model generating unit.

[0053] Fig.13 A diagram showing product images including products.

[0054] Fig.14A is a graph representing a segmented image.

[0055] Fig. 14B is a graph representing a segmented image.

[0056] Fig. 14C is a graph representing a segmented image.

[0057] Fig.14D is a graph representing a segmented image.

[0058] Fig.15 A diagram showing an object image including a product. DETAILED DESCRIPTION

[0059] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that in the description of the drawings, the same or corresponding elements are denoted by the same reference numerals, and repeated descriptions are omitted.

[0060] like Figure 1 As shown, the weighing system (commodity identification system) 1 includes a weighing device (commodity identification apparatus) 100 and a server (learned model generation apparatus) 200. The weighing device 100 and the server 200 are connected to each other in a communicable manner.

[0061] The measuring device 100 includes a measuring device 2, a control device 3, a display device 4, and a camera (photographing unit) 5. The measuring device 2 and the control device 3 are connected to each other so as to be able to communicate with each other. The control device 3 and the camera 5 are connected to each other so as to be able to communicate with each other. The measuring device 2 and the display device 4 are connected to each other so as to be able to communicate with each other.

[0062] The weighing device 2 is an electronic scale. The weighing device 2 has a weighing device S (see Figure 6 ) and issues a label LC attached to the weighed product S. In this embodiment, the product S is food (salad, side dish) placed in a container P. Here, the container P has a transparent cover. The cover is colored or colorless and is made of a material that allows light to pass through. The camera 5 captures the light that passes through the cover of the container P, thereby capturing the product S in the container P.

[0063] like Figure 2 As shown, in this embodiment, the measuring device 2 is stored in a frame 10. The frame 10 has a storage portion 11, a holding portion 12, and a connecting portion 13. The storage portion 11 stores the measuring device 2. The storage portion 11 is box-shaped. The storage portion 11 is provided with a first opening 11a for exposing a measuring table 21a of a measuring portion 21 of the measuring device 2, and a second opening 11b for exposing an operating portion 22 of the measuring device 2.

[0064] The control device 3, the camera 5, the first lighting unit 6, and the second lighting unit 7 are arranged in the holding unit 12. The holding unit 12 is arranged on the storage unit 11. The control device 3 is arranged on the holding unit 12. The connecting unit 13 connects the storage unit 11 and the holding unit 12. The connecting unit 13 extends in the up-down direction. The display device 4 is arranged on the back of the connecting unit 13. The display device 4 is, for example, a liquid crystal display. The display device 4 performs a display for the customer.

[0065] The camera 5 is disposed above a weighing table 21a (described later) of the weighing device 2 at a position facing the weighing table 21a. The camera 5 outputs the captured image data to the control device 3. In addition, the camera 5 outputs the captured image data to the server 200.

[0066] The first lighting unit 6 and the second lighting unit 7 are, for example, LED lighting. The first lighting unit 6 and the second lighting unit 7 are arranged above the measuring table 21a of the measuring device 2 at a position opposite to the measuring table 21a. The first lighting unit 6 and the second lighting unit 7 are arranged at a predetermined interval in the width direction of the frame 10. Specifically, the first lighting unit 6 and the second lighting unit 7 are arranged at a position sandwiching the camera 5 in the width direction of the frame 10.

[0067] like Figure 3 As shown, the weighing device 2 includes a control unit 20 , a weighing unit 21 , an operation unit 22 , and a label issuing unit 23 .

[0068] The control unit 20 controls various operations in the weighing device 2 and includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The control unit 20 controls the display of a touch panel display 22 a described later.

[0069] The measuring unit 21 includes a measuring table 21a, a load cell (not shown), a signal processing circuit, and a transmission module. The measuring table 21a carries the product S. The load cell is disposed below the measuring table 21a. The load cell converts the mechanical strain generated by placing the object on the measuring table 21a into an electrical signal. The signal processing circuit amplifies the electrical signal output from the load cell and converts it into a digital signal. The transmission module outputs the digital signal to the control unit 20.

[0070] The operation unit 22 includes a touch panel display 22a and fixed keys 22b. The touch panel display 22a displays information related to the commodity S measured by the weighing device 2 and basic information required for the operation of the weighing device 2, etc., under the control of the control unit 20. The fixed keys 22b include a "unit price" key, a "quota" key, a "tare" key, a "print" key, and a "call" key required as a charging scale, and they are appropriately arranged together with the numeric keys.

[0071] The label issuing unit 23 issues a label LC. The label issuing unit 23 prints commodity information based on print information output from an output unit 29 described later, and issues a label LC. In the present embodiment, the label issuing unit 23 issues a so-called linerless label.

[0072] The control unit 20 includes an input unit 25 , a storage unit 26 , a calculation unit 27 , a control unit 28 , and an output unit 29 .

[0073] The input unit 25 inputs the number information output from the control device 3. The input unit 25 outputs the input number information to the calculation unit 27.

[0074] The storage unit 26 stores a commodity master file. The commodity master file stores commodity-related information related to each commodity S. The commodity master file is a table that associates a commodity number with a commodity name, unit price, quota, etc. The commodity master file can be updated (changed).

[0075] The calculation unit 27 calculates the price of the commodity S. After receiving the number information output from the input unit 25, the calculation unit 27 refers to the commodity master file according to the commodity number contained in the number information. The calculation unit 27 obtains the unit price of the commodity S corresponding to the commodity number from the commodity master file. The calculation unit 27 calculates the price (price) of the commodity S based on the measurement value output from the measuring unit 21 and the unit price of the commodity. After receiving the confirmation information from the control unit 28, the calculation unit 27 determines the price and outputs the print information to the label issuing unit 23. The print information contains at least information indicating the commodity name, weight, and price. In addition, the calculation unit 27 outputs display information to the touch panel display 22a so that the touch panel display 22a displays the calculated price.

[0076] When the "print" key is pressed among the fixed keys 22b of the operation unit 22, the control unit 28 causes the price calculated by the calculation unit 27 to be finalized. When the control unit 28 determines that the "print" key is pressed, it outputs finalization information to the output unit 29. The finalization information is information indicating that the price is finalized in the calculation unit 27. The output unit 29 outputs the finalization information to the calculation unit 27.

[0077] like Figure 4 As shown, the control device 3 includes a control unit 30 and a touch panel display 31. The control unit 30 controls the display of the touch panel display 31. The control device 3 may be a tablet terminal or the like in which the control unit 30 and the touch panel display 31 are integrated, or may be a computer.

[0078] The control unit 30 is a part that controls various actions in the control device 3, and its structure includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The control unit 30 has an input unit 32, a determination unit (first acquisition unit, second acquisition unit) 33, a storage unit 34, a control unit 35, and an output unit 36.

[0079] The input unit 32 inputs the image data output from the camera 5. The input unit 32 outputs the input image data to the discrimination unit 33. In addition, the input unit 32 inputs the learned model (discrimination model, acquisition model) transmitted from the server 200 and stores it in the storage unit 34.

[0080] The determination unit 33 determines the type of the commodity S placed on the weighing device 2 based on the object image (object image data) captured by the camera 5. The determination unit 33 detects that the commodity S is placed on the weighing table 21a of the weighing device 2 based on the object image data output from the input unit 32. Specifically, the determination unit 33 detects that the commodity S is placed based on the difference (background difference) between the object image data output from the input unit 32 and a pre-stored basic image (background image). In detail, the determination unit 33 determines that the commodity S is placed when the degree of change in the object image data compared to the basic image is greater than a threshold value.

[0081] When determining that the product S is placed, the determination unit 33 determines whether the placement of the product S is stable. That is, the determination unit 33 determines whether the position of the product S is determined. The determination unit 33 determines whether the placement of the product S is stable, for example, by using a frame difference method of continuous object image data. Specifically, when the difference between frames in the continuous object image data (for example, the number of pixels with a change in pixel value of a certain value or more) is less than a threshold value, the determination unit 33 determines that the placement of the product S is stable. When determining that the placement of the product S is stable, the determination unit 33 determines the product S based on the image in the object image data determined to be stable.

[0082] In this embodiment, the identification unit 33 uses the acquisition model and the identification model to identify the type of the commodity S. The following is a description of the commodity identification method by which the identification unit 33 identifies the type of the commodity S. Figure 5 As shown, as a commodity identification method, the identification unit 33 performs a first acquisition step S01, a second acquisition step S02, and a identification step S03.

[0083] The determination unit 33 divides the target image data into a plurality of regions, and acquires a plurality of segmented target images (second segmented images (first acquisition step S01: Figure 5 ).like Figure 6 As shown, the discriminating unit 33 divides the object image data G1 into a plurality of regions A. Specifically, the discriminating unit 33 divides the object image data G1 in such a manner that the regions A of the plurality of segmented object images are rectangular and all have the same size. The discriminating unit 33 divides the object image data G1 in such a manner that one segmented object image overlaps at least a portion of the other segmented object images. Specifically, the discriminating unit 33 sets the regions A of the segmented object image data G1 in such a manner that the other regions A are moved by a predetermined amount in the arrangement direction of the pixels of the object image data G1, i.e., the X direction (first direction) or the Y direction (second direction orthogonal to the first direction) relative to one region A. The predetermined amount is an amount corresponding to the width (1 / 2, 1 / 3, etc.) obtained by dividing the width of the region A in the moving direction by a natural number greater than 2. The natural number greater than 2 is a value that can divide the number of pixels in the moving direction of the region A in the object image data G1.

[0084] The determination unit 33 divides the object image data G1 into Fig. 7A , Figure 7B , Figure 7C as well as Fig.7D As shown, segmentation target images G11, G12, G13, G14, etc. are obtained. The segmentation target image G11 is an image in which the product S and the like are not captured, that is, only the background B is captured. The segmentation target image G12 is an image in which the container P and the background B are captured. The segmentation target image G13 is an image in which the product S, the container P, and the background B are captured. The segmentation target image G14 is an image in which only the product S is captured.

[0085] The determination unit 33 acquires a plurality of segmentation target images satisfying a predetermined condition related to the amount of product S captured from the plurality of segmentation target images (second acquisition step S02: Figure 5 ). Specifically, the discriminating unit 33 obtains a segmentation target image in which the amount of image other than the product S is below a threshold. In other words, the discriminating unit 33 obtains a segmentation target image in which the amount of image of the product S is greater than the threshold. The discriminating unit 33 obtains a segmentation target image in which the amount of image other than the product S is below the threshold by using an acquisition model. The acquisition model includes a neural network NW1. The acquisition model is the same model as the extraction model described later.

[0086] like Figure 8 As shown, the neural network NW1 of the acquisition model is composed of, for example, a first layer as an input layer, a second layer, a third layer and a fourth layer as intermediate layers (hidden layers), and a fifth layer as an output layer. The first layer directly outputs the input value x=(x0, x1, x2, ... xp) with p parameters as elements to the second layer. The second, third and fourth layers respectively convert the total input into output through an activation function and pass the output to the next layer. The fifth layer also converts the total input into output through an activation function, and the output is the output value y=y of the neural network with one parameter as an element. 0 .

[0087] In this embodiment, the neural network NW1 inputs the pixel value of each pixel of the segmentation target images G11 to G14 and outputs information indicating the processing result. The input layer of the neural network NW1 is provided with neurons corresponding to the number of pixels of the segmentation target images G11 to G14. The output layer of the neural network NW1 is provided with neurons for outputting information related to the processing result. According to the output value (non-product likelihood) of the neurons in the output layer, a segmentation target image in which the amount of images other than the product S is below the threshold can be obtained. The output value of the neuron is, for example, a value of 0 to 1. In this case, the larger the value of the neuron (the closer the value is to 1), the lower the possibility that the image is a segmentation target image in which the amount of images other than the product S is below the threshold, and the smaller the value of the neuron (the closer the value is to 0), the higher the possibility that the image is a segmentation target image in which the amount of images other than the product S is below the threshold. That is, when the neuron value is large, it means that the proportion of the background G and the like in the segmentation target image is large, and when the neuron value is low, it means that the proportion of the product S in the segmentation target image is large. The determination unit 33 acquires the neuron value output from the neural network NW1 and acquires a segmentation target image having a neuron value that is equal to or less than a threshold value.

[0088] The discriminating unit 33 discriminates the type of the commodity S using the discriminating model for the acquired plurality of segmentation target images (discriminating step S03: Figure 5 ). The discriminant model includes a neural network NW2.

[0089] like Fig. 9 As shown, the neural network NW2 is composed of, for example, a first layer as an input layer, a second layer, a third layer, and a fourth layer as intermediate layers (hidden layers), and a fifth layer as an output layer. The first layer directly outputs the input value x=(x0, x1, x2, ... xp) with p parameters as elements to the second layer. The second, third, and fourth layers convert the total input into output through an activation function and pass the output to the next layer. The fifth layer also converts the total input into output through an activation function, and the output is the output value y=(y 0 ,y 1 ,……,y q ).

[0090] In this embodiment, the neural network NW2 inputs the pixel value of each pixel of each segmentation target image, and outputs information representing the discrimination result of the product S for each segmentation target image. The input layer of the neural network NW2 is provided with neurons corresponding to the number of pixels of the image. The output layer of the neural network NW2 is provided with neurons for outputting information related to the discrimination result of the product S. The type of the product S can be discriminated based on the output value (product likelihood) of the neurons in the output layer. The output value y corresponds to the product likelihood of the product S. For example, the output value y1 corresponds to the product likelihood of the product S.1 The output value y corresponds to the likelihood of the product i With product S i The output value of the neuron is, for example, a value between 0 and 1. For example, when the output value y 1 When the value of is "0.8", product S 1 The likelihood of the product is "0.8", and the output value y 2 When the value of is "0.2", product S 2 The likelihood of the product is "0.2". In this case, the larger the value of the neuron (the closer the value is to 1), the higher the possibility that it is the product S of the target image data G1, and the smaller the value of the neuron (the closer the value is to 0), the lower the possibility that it is the product S of the target image data G1. That is, when the neuron value is large, the probability that it is product S is high, and when the neuron value is small, the probability that it is product S is low.

[0091] The discrimination unit 33 inputs the segmentation target image into the discrimination model. Based on the fact that the segmentation target image is input into the neural network NW2 of the discrimination model, the discrimination unit 33 obtains a discrimination result including an output value output from the neural network NW2 for each segmentation target image. The discrimination result includes all types of commodities registered in the commodity master file.

[0092] The discriminator 33 ranks the candidate commodities according to the discrimination result. Specifically, the discriminator 33 weights the values ​​of neurons of each segmented target image and ranks the commodities according to the majority decision of the weights assigned to the discrimination result. The discriminator 33 generates discrimination information that makes the commodity numbers correspond to the rankings for all types of commodities. The discriminator 33 outputs the image information and discrimination information of the image data used in the discrimination process to the control unit 35.

[0093] After the image information and the discrimination information are output from the discrimination unit 33, the control unit 35 causes the touch panel display 31 to display the image information and the discrimination information. The control unit 35 controls the display of the touch panel display 31 according to the input received in the touch panel display 31. Specifically, the control unit 35 causes one screen displayed on the touch panel display 31 to display an image of the commodity S based on the image information. The control unit 35 causes one screen to display the commodity name of the commodity with a high ranking among the commodity candidates in the discrimination information. The control unit 35 outputs number information indicating the commodity number to the output unit 36 ​​according to the discrimination information or the input received in the touch panel display 31. The output unit 36 ​​outputs the number information to the weighing device 2.

[0094] The storage unit 34 stores a commodity master file. The commodity master file is configured to include the same contents as the commodity master file stored in the storage unit 26 of the weighing device 2. The storage unit 34 stores a learned model.

[0095] like Figure 1 As shown, the server 200 includes a communication unit 201 and a learned model generation unit (acquisition unit, segmentation unit, extraction unit, generation unit) 202. The server 200 is a device that generates a learned model through machine learning. The server 200 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc.

[0096] The communication unit 201 communicates with the measuring instrument 100 . The communication unit 201 receives the image data transmitted from the measuring instrument 100 , and outputs the image data to the learned model generation unit 202 . The communication unit 201 transmits the learned model output from the learned model generation unit 202 to the measuring instrument 100 .

[0097] The learned model generation unit 202 acquires learning data used in machine learning, and performs machine learning using the acquired learning data to generate a learned model. In this embodiment, the learning data is a teaching image. The teaching image is, for example, image data acquired by the camera 5 of the measuring instrument 100.

[0098] The learned model generation unit 202 generates an extraction model and a discrimination model. First, a method for generating an extraction model is described. The extraction model is the same model as the acquisition model described above. The extraction model is generated by machine learning based on images that do not contain products. The following is a method for generating an extraction model by the learned model generation unit 202. Fig.10 As shown in FIG. 2 , as a method of generating an extraction model, the learned model generating unit 202 performs a first step S11 , a second step S12 , and a third step S13 .

[0099] The learned model generation unit 202 obtains a non-product image that does not include a product as a teaching image (first step S11: Fig.10 The non-product image may include a background B and a container P. The learning-completed model generation unit 202 divides the non-product image into a plurality of regions and obtains a plurality of non-product segmented images (second step S12: Fig.10 ).like Fig.11As shown, the learned model generation unit 202 divides the non-product image data G2 into a plurality of regions A1. Specifically, the learned model generation unit 202 divides the non-product image data G2 in such a manner that the regions A1 of the plurality of non-product segmented images are rectangular and all have the same size. The learned model generation unit 202 divides the non-product image data G2 in such a manner that one non-product segmented image overlaps at least a portion of the other non-product segmented images. Specifically, the learned model generation unit 202 sets the regions A1 for dividing the non-product image data G2 in such a manner that the other regions A1 are shifted by a predetermined amount in the arrangement direction of the pixels of the non-product image data G2, i.e., the X direction or the Y direction, relative to one region A1. The predetermined amount is an amount corresponding to the width obtained by dividing the width of the region A1 in the moving direction by a natural number greater than 2. The natural number greater than 2 is a value that rounds the number of pixels in the moving direction of the region A in the non-product image data G2.

[0100] The learned model generation unit 202 performs machine learning based on the acquired plurality of non-product segmented images to generate an extraction model (third step S13: Fig.10 ). The machine learning itself can be performed using known machine learning algorithms. The extraction model includes a neural network NW1 (refer to Figure 8 ).

[0101] The learned model generation unit 202 generates a discriminant model using the extracted model. The following describes a generation method (learned model generation method) for generating a discriminant model by the learned model generation unit 202. Fig.12 As shown, as a method of generating a discriminant model, the learned model generating unit 202 performs an acquisition step S21, a segmentation step S22, an extraction step S23, and a generation step S24.

[0102] The learned model generation unit 202 acquires the product image and product information related to the type of the product (acquisition step S21: Fig.12 The learning model generation unit 202 obtains, for example, product image data obtained by the camera 5 and product information indicating the type of product (e.g., salad) contained in the product image data. The product information corresponds to the information contained in the above-mentioned product master file, including the product number, product name, etc. In addition, the product image may be a sample image, etc., in addition to the image data obtained by the camera 5.

[0103] The learned model generation unit 202 divides the product image data into a plurality of regions and obtains a plurality of divided images (first divided images) (division step S22: Fig.12 ).like Fig.13As shown, the learned model generation unit 202 divides the product image data G3 into a plurality of regions A2. Specifically, the learned model generation unit 202 divides the product image data G3 in such a manner that the regions A2 of the plurality of segmented images are rectangular and all have the same size. The learned model generation unit 202 divides the product image data G3 in such a manner that the regions A2 and the regions A1 have the same shape and the same size. The learned model generation unit 202 divides the product image data G3 in such a manner that one segmented image overlaps at least a portion of the other segmented images. Specifically, the learned model generation unit 202 sets the regions A2 of the segmented product image data G3 in such a manner that the other regions A2 are moved by a predetermined amount in the arrangement direction of the pixels of the product image data G3, i.e., the X direction or the Y direction, relative to one region A2. The predetermined amount is an amount corresponding to the width obtained by dividing the width of the region A in the moving direction by a natural number greater than 2. The natural number greater than 2 is a value that can divide the number of pixels in the moving direction of the region A in the product image data G3.

[0104] The learned model generation unit 202 divides the product image data G3 into Fig.14A , Fig. 14B , Fig. 14C as well as Fig.14D As shown, segmented images G31, G32, G33, G34, etc. are obtained. The segmented image G31 is an image in which the product S and the like are not captured, that is, only the background B is captured. The segmented image G32 is an image in which the container P and the background B are captured. The segmented image G33 is an image in which the product S, the container P, and the background B are captured. The segmented image G34 is an image in which only the product S is captured.

[0105] The learned model generation unit 202 obtains a plurality of segmented images that satisfy a predetermined condition related to the amount of product S captured from the plurality of segmented images (extraction step: S23). Specifically, the learned model generation unit 202 obtains a segmented image in which the amount of capture other than product S is below a threshold. In other words, the learned model generation unit 202 obtains a segmented image in which the amount of capture of product S is greater than the threshold. The learned model generation unit 202 obtains a segmented image in which the amount of capture other than product S is below the threshold through the extraction model. The learned model generation unit 202 obtains a neuron value output from the neural network of the extraction model, and obtains a segmented image in which the neuron value is below the threshold. The learned model generation unit 202 makes the acquired segmented image correspond to product information (product type (product name, product number, etc.)). That is, the learned model generation unit 202 marks the product information with respect to the acquired segmented image.

[0106] The learned model generation unit 202 associates the extracted multiple segmented images with the product information to perform machine learning and generate a discrimination model (generation step S24: Fig.12). The machine learning itself can be performed using known machine learning algorithms. Fig. 9 As shown, the learned model generation unit 202 generates a discriminant model including a neural network NW2. The discriminant model may also include a convolutional neural network. Furthermore, the discriminant model may also include a neural network of multiple levels (e.g., more than 8 layers). That is, the discriminant model may also be generated by deep learning. The learned model generation unit 202 outputs the acquisition model (extraction model) and the discriminant model to the communication unit 201 at a specified time.

[0107] As described above, the measuring system 1 according to the present embodiment includes a server 200 that generates a discrimination model for discriminating the type of a product based on a product image including the product, and a measuring device 100 that discriminates the type of a product based on an object image including the product. The server 200 includes a learned model generation unit 202. The learned model generation unit 202 acquires a product image and product information related to the type of the product (acquisition step), divides the product image into a plurality of regions, and acquires a plurality of segmented images (segmentation step). The learned model generation unit 202 extracts a plurality of segmented images that satisfy a predetermined condition related to the amount of product captured from the plurality of segmented images (extraction step). The learned model generation unit 202 associates the extracted plurality of segmented images with the product information and performs machine learning to generate a discrimination model (generation step).

[0108] Thus, in the measuring system 1 according to the present embodiment, the learned model generation unit 202 of the server 200 extracts a plurality of segmented images that satisfy the prescribed conditions related to the amount of goods captured from the plurality of segmented images. The segmented images include images in which only goods are captured, images in which goods and objects other than goods (containers, backgrounds, etc.) are captured, and images in which only objects other than goods are captured. In the learned model generation unit 202, the segmented images that satisfy the prescribed conditions are extracted from such segmented images, and machine learning is performed based on the extracted segmented images. Thus, in the learned model generation unit 202, machine learning can be performed based on appropriate teaching images. Therefore, in the server 200, a discrimination model that improves the discrimination accuracy of goods can be generated.

[0109] In the weighing system 1 according to the present embodiment, the control device 3 includes a discriminating unit 33. The discriminating unit 33 divides the target image into a plurality of regions, acquires a plurality of segmented target images (a first acquisition step), and acquires a plurality of segmented target images that satisfy a predetermined condition related to the amount of product captured from the plurality of segmented target images (a second acquisition step). The discriminating unit 33 acquires a processing result processed using a discriminant model for the plurality of segmented target images acquired, and discriminates the type of the product based on the processing result (a discriminating step).

[0110] Thus, in the weighing system 1 according to the present embodiment, the discriminating unit 33 of the control device 3 extracts a plurality of segmented target images that satisfy the prescribed condition related to the amount of product taken in from the plurality of segmented target images. The discriminating unit 33 obtains the segmented target images that satisfy the prescribed condition from the segmented target images, and processes the obtained segmented target images using the discriminant model. Thus, in the weighing device 100, processing can be performed based on appropriate segmented target images. Therefore, in the weighing system 1, the accuracy of product discrimination is improved.

[0111] In the metering system 1 according to the present embodiment, the learned model generation unit 202 of the server 200 extracts segmented images in which the amount of items other than the product is less than a threshold value. In other words, the learned model generation unit 202 extracts segmented images in which the amount of items other than the product is greater than the threshold value. In this method, segmented images with a high occupancy rate of the product are extracted. That is, in the metering system 1, segmented images in which containers or backgrounds other than the product are captured can be excluded. Therefore, in the metering system 1, machine learning can be performed using segmented images in which the product is captured. Therefore, in the metering system 1, a discrimination model that improves the discrimination accuracy of the product can be generated.

[0112] In the measurement system 1 according to the present embodiment, the learned model generation unit 202 uses the extraction model generated by machine learning based on the image not containing the product to obtain the neuron value indicating the probability of being a segmented image not containing the product for each of the plurality of segmented images, and extracts the segmented images having the neuron value below the threshold. In this method, only the segmented images containing the product can be appropriately extracted.

[0113] In the measurement system 1 according to the present embodiment, the learned model generation unit 202 obtains a non-product image that does not include a product, and divides the non-product image into a plurality of regions to obtain a plurality of non-product segmented images. The learned model generation unit 202 performs machine learning based on the plurality of non-product segmented images to generate an extraction model. Thus, in the measurement system 1, an extraction model can be appropriately generated.

[0114] In the measurement system 1 according to the present embodiment, the learned model generation unit 202 divides the product image in such a manner that the regions of the plurality of segmented images are rectangular and all have the same size. The learned model generation unit 202 divides the non-product image in such a manner that the plurality of non-product segmented images have the same shape and size as the segmented image. In this method, since the segmented image has the same shape and size as the non-product segmented image, there is no need to perform conversion processing for converting the shape and size of the image. Therefore, in the measurement system 1, a reduction in processing load is achieved.

[0115] In the measuring system 1 according to the present embodiment, the learned model generation unit 202 segments the product image in such a way that one segmented image overlaps at least a part of the other segmented images. In this method, when segmenting the product image, even if one item included in the product is cut off in one segmented image, the other segmented image may include the item completely. Therefore, in the measuring system 1, since machine learning can be performed based on an appropriate teaching image, a discrimination model that can improve the discrimination accuracy of the product can be generated.

[0116] In the measurement system 1 according to the present embodiment, the learned model generation unit 202 sets the region for segmenting the product image so that the other regions are shifted by a predetermined amount in the X direction or Y direction, which is the arrangement direction of pixels of the product image, relative to one region. In this method, by shifting the region by a predetermined amount, the product image can be segmented so that each pixel is included in each region arranged in the shift direction the same number of times.

[0117] In the measurement system 1 according to the present embodiment, the learned model generation unit 202 divides the product image in such a way that the regions of the plurality of segmented images are rectangular and all have the same size. In this method, since all the segmented images have the same shape and the same size, there is no need to perform conversion processing for converting the shape and size of the image. Therefore, in the measurement system 1, the processing load is reduced.

[0118] In the weighing system 1 according to the present embodiment, the discrimination unit 33 obtains segmentation target images in which the amount of objects other than commodities captured is less than a threshold value. In this method, images with a high occupancy rate of commodities are obtained. That is, images in which containers other than commodities or backgrounds are captured can be excluded. Therefore, in the weighing system 1, the accuracy of commodity discrimination is improved.

[0119] In the weighing system 1 according to the present embodiment, the determination unit 33 uses the acquisition model generated by machine learning based on the image not containing the commodity to acquire the neuron value indicating the probability of being a segmentation target image not containing the commodity for each of the plurality of segmentation target images, and acquires the segmentation target image having the neuron value below the threshold. In this method, only the segmentation target image containing the commodity can be appropriately acquired.

[0120] In the weighing system 1 according to the present embodiment, the discrimination unit 33 processes the plurality of segmentation target images using the discrimination model to obtain a plurality of processing results, and discriminates the type of the commodity based on the plurality of processing results. In this method, the discrimination model is used to process each segmentation target image, and a plurality of processing results are obtained, thereby further improving the discrimination accuracy of the commodity.

[0121] In the weighing system 1 according to the present embodiment, the discriminating unit 33 weights the processing result according to the degree of the neuron value indicating the probability that the product is of one type, and discriminates the type of the product by majority decision of the weights assigned to the processing result. In this method, the product can be discriminated with higher accuracy.

[0122] In the measurement system 1 according to the present embodiment, the determination unit 33 divides the target image so that the regions of the plurality of segmentation target images are rectangular and all have the same size. In this method, since all the segmentation target images have the same shape and the same size, it is not necessary to perform conversion processing for converting the shape and size of the image. Therefore, in the measurement system 1, a reduction in processing load is achieved.

[0123] As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, Various changes are possible within the range which does not deviate from the summary of this invention.

[0124] In the above embodiment, the discrimination unit 33 divides the object image data G1 into a plurality of regions A as an example. However, the discrimination unit 33 may compare the product image with a base image that does not include the product, cut out a product region that includes the entire product and is smaller than the product image from the product image (cutting out step), divide the product region into a plurality of regions, and obtain a plurality of segmented images. Specifically, Fig.15 As shown in FIG. 1 , the discriminating unit 33 compares the target image data G1, which is a product image, with the base image, and cuts out the product area SA that contains the product S to a minimum from the target image data G1. The base image is an image that captures only the background B. The discriminating unit 33 divides the product area SA into a plurality of areas and obtains a plurality of divided images. In this method, since the product area SA excluding the portion other than the product S is used for segmentation in the target image data G1, it is not necessary to segment unnecessary areas. Therefore, it is possible to reduce the processing load.

[0125] In the above embodiment, the determination unit 33 has been described as an example in which the predetermined amount by which the other regions A are moved relative to one region A in order to set the region A of the segmentation target image data G1 is an amount corresponding to the width of the region A in the moving direction divided by a natural number greater than 2 (1 / 2, 1 / 3, etc.). However, the determination unit 33 may also set the predetermined amount to be one pixel. That is, the determination unit 33 may also set the region A of the segmentation target image data G1 by moving the other regions A relative to one region A by one pixel in the arrangement direction of the pixels of the target image data G1, that is, the X direction or the Y direction. The same is true when the learned model generation unit 202 sets the region A1 for segmenting the non-product image data G2 and when setting the region A2 for segmenting the product image data G3.

[0126] In the above embodiment, the neural network NW2 of the discrimination model inputs the pixel value of each pixel of each segmentation target image, and outputs information indicating the discrimination result of the commodity S for each segmentation target image. However, the discrimination model may also be a configuration of the neural network NW2 that includes inputting the pixel value of each pixel of each segmentation target image and outputting information indicating the discrimination result of the commodity S for each pixel.

[0127] In the above-mentioned embodiment, the form in which the discrimination unit 33 weights the size of the neuron value of each segmented object image and sorts the commodities by majority decision of the weights assigned to the discrimination results is described as an example. However, the sorting (discrimination) of commodities in the discrimination unit 33 may also be other methods. For example, the discrimination unit 33 may sort the commodities only by using the neuron values ​​above the threshold, or may prioritize the discrimination results of the segmented object images in the central area of ​​the object image data to sort the commodities. In addition, the discrimination unit 33 may not use the neuron values ​​below the threshold (commodity likelihood below the threshold) among the neuron values ​​of each segmented object image.

[0128] In addition, the determination unit 33 may also determine, for example, the neuron value (output value) y of a segmentation target image. 2 (Product S 2 The likelihood of the product is "0.9", and the neuron value y of other segmentation target images is 2 is "0.4", and the neuron value y of the above segmentation target image 5 (Product S 5 The likelihood of the product is "0.7", which is consistent with the product S 2 When the variance of the relevant neuron value is large, reduce the product S 2 That is, when there is a deviation (divergence) (neuron value dispersion) between the neuron value corresponding to a commodity in one segmentation target image and the neuron value corresponding to a commodity in another segmentation target image, and when the neuron value corresponding to the other commodity in one segmentation target image is greater than a prescribed value, the discrimination unit 33 reduces the weight of one commodity.

[0129] In the above embodiment, the discriminating unit 33 uses the acquisition model to obtain the segmentation target images whose pixel inclusion ratio is below the predetermined threshold value from the plurality of segmentation target images. However, the discriminating unit 33 may also extract the segmentation target images whose pixel inclusion ratio is below the predetermined threshold value from the plurality of segmentation target images based on the feature quantity of the pixels of the images not containing the products. In this method, the acquisition model is not used, and only the segmentation target images containing the products can be appropriately extracted.

[0130] In the above embodiment, the example in which the discrimination unit 33 obtains a segmentation target image in which the amount of items other than commodities captured is below a threshold is described. However, the discrimination unit 33 may also obtain information on whether or not there are items other than commodities (containers, backgrounds, etc.) captured in the outer edge (outer edge region) of the segmentation target image, and obtain the segmentation target image when no items other than commodities are captured. In this way, segmentation target images with a high occupancy rate of commodities can be extracted.

[0131] In the above-mentioned embodiment, the form in which the object image data includes one commodity S and the discrimination unit 33 discriminates the type of one commodity S is described as an example. However, the object image data may also include multiple commodities. In this case, the discrimination unit 33 discriminates the types of each of the multiple commodities. Specifically, the discrimination unit 33 extracts the contour line (edge) based on the difference between the object image data and the pre-stored basic image, and obtains the area containing the commodity based on the edge. At this time, even if there are more than two commodities due to a situation such as a partial overlap of the commodities, they are sometimes obtained as one area. Therefore, the discrimination unit 33 compares the area of ​​the area with the area of ​​the container, for example, to obtain the area of ​​each commodity (dividing the area for each commodity). As a method of obtaining the area of ​​each commodity, other methods may also be used. When obtaining multiple areas, the discrimination unit 33 discriminates the type of the commodity for each area.

[0132] In addition, in the case where a plurality of commodities are placed in a plate, for example, the discriminating unit 33 discriminates the types of the plurality of commodities respectively through the following processing. The discriminating unit 33 adopts a majority decision method in the processing results of a segmented object image and a plurality of segmented object images near the segmented object image, and uses the result as the result (the type of commodity) at the center coordinate of the region containing the above-mentioned segmented object image and the above-mentioned plurality of segmented object images (the processing results of each segmented object image are smoothed). The discriminating unit 33 performs the above-mentioned processing on all segmented object images to obtain the result at the center coordinate of each region. The discriminating unit 33 discriminates the types of each of the plurality of commodities by aggregating the regions where the same results (commodities) are densely concentrated in the obtained results into one. In addition, in the case where a plurality of commodities are included, as in the above-mentioned embodiment, it is used as a commodity discriminating device instead of the meter 100 for measuring and selling. In the case where a plurality of foods are placed on a tray, for example, the commodity discriminating device can discriminate the types of each of the plurality of foods, or check the contents of the bento.

[0133] In the above-mentioned embodiment, the learned model generation unit 202 divides the product image in a manner that the regions of the multiple segmented images are rectangular and all of the same size, and divides the non-product image in a manner that the multiple non-product segmented images are the same shape and the same size as the segmented image as an example. However, the shapes and sizes of the product image and the non-product segmented image may be different. For example, the resolution of the segmented image may be higher than that of the non-product segmented image. In this case, when the shape is set to a rectangle, it is preferred to make the ratio of the vertical and horizontal (aspect ratio) consistent. That is, it is preferred to form the segmented image and the non-product segmented image into a similar shape. As a result, there is no need to perform the process of converting the ratio of the image, thereby reducing the processing load.

[0134] In the above embodiment, the learned model generation unit 202 uses the extraction model to obtain segmented images with a pixel inclusion ratio below a predetermined threshold value from a plurality of segmented images. However, the learned model generation unit 202 may extract segmented images with a pixel inclusion ratio below a predetermined threshold value from a plurality of segmented images based on the feature quantity of pixels of images that do not contain products. In this method, the extraction model is not used, and only segmented images with products can be appropriately extracted.

[0135] In the above embodiment, the learned model generation unit 202 obtains a segmented image in which the amount of items other than goods captured is below a threshold value. However, the learned model generation unit 202 may also obtain information on whether items other than goods (containers, background, etc.) are captured in the outer edge (outer edge area) of the segmented image, and obtain the segmented image when no items other than goods are captured. In this way, a segmented image with a high occupancy rate of goods can be extracted.

[0136] In the above-mentioned embodiment, the learned model generation unit 202 obtains a plurality of segmented images that satisfy the prescribed conditions related to the amount of product S captured from a plurality of segmented images, and associates the extracted plurality of segmented images with product information to perform machine learning, thereby generating a discriminant model. However, the learned model generation unit 202 may also extract a segmented image in which the amount of capture other than the product is below a threshold and other segmented images in which the amount of capture other than the product is greater than the threshold from the plurality of segmented images, establish a correspondence between the product information and the segmented image, and establish a correspondence between the non-product information other than the product and the other segmented images, perform machine learning based on the extracted segmented image and the other segmented images, thereby generating a discriminant model. The discriminant unit 33, for example, uses the extraction model to extract other segmented images in which the amount of capture other than the product is greater than the threshold (the capture amount of the product is below the threshold).

[0137] Specifically, the learned model generation unit 202 obtains the neuron value output from the neural network of the extraction model, and extracts a segmented image in which the amount of input other than the product S is below the threshold and other segmented images in which the amount of input other than the product is more than the threshold based on the neuron value. That is, the learned model generation unit 202 classifies into a segmented image in which the amount of input other than the product S is below the threshold and other segmented images in which the amount of input other than the product is more than the threshold, and extracts one segmented image and other segmented images. The learned model generation unit 202 marks the product information (product name) on one segmented image, and marks non-product information indicating items other than the product (background, container, etc.) on other segmented images. The learned model generation unit 202 performs machine learning based on the one segmented image and the other segmented images to generate a discriminant model. The machine learning itself can be performed using a known machine learning algorithm.

[0138] In the above embodiment, the discriminator 33 obtains a plurality of segmented object images that satisfy a predetermined condition related to the amount of product S captured from a plurality of segmented object images, inputs the segmented object images into a discriminator model, obtains a processing result processed by the discriminator model, and discriminates the type of product S based on the processing result as an example. However, the discriminator 33 may also divide the object image data into a plurality of regions (segmented image acquisition step) and input a plurality of segmented object images into the discriminator model. That is, the discriminator 33 inputs all segmented object images including the background B other than the product S into the discriminator model. Based on the fact that the segmented object images are input into the neural network NW2 of the discriminator model, the discriminator 33 obtains a discrimination result including an output value output from the neural network NW2 for each segmented object image.

[0139] The discriminant unit 33 obtains the product likelihood indicating the probability of being a product and the non-product likelihood indicating the probability of being an item other than a product from the processing result processed using the discriminant model. As the processing result, for example, for each segmented image, it is obtained in the form of an output value such as 1: (0.02; 0.20, 0.31, 0.89, ...), 2: (0.89; 0.12, 0.04, 0.23, ...). In the processing result, (non-product likelihood; product likelihood) is shown. That is, in the processing result of the segmented image of "1" above, the non-product likelihood is "0.02" and the product likelihood is "0.20, 0.31, 0.89, ...". In this case, the processing result of the segmented image of "1" indicates that the possibility of being a product with a product likelihood of "0.89" is high. In the processing result of the segmented image of "2" above, the non-product likelihood is "0.89", and the product likelihood is "0.12, 0.04, 0.23, ...". In this case, the processing result of the segmented image of "2" above indicates that the possibility of being the background is high. The discrimination unit 33 extracts the product likelihood from the two types of likelihoods, and discriminates the type of the product based on the product likelihood. Specifically, first, the discrimination unit 33 removes the segmented images whose non-product likelihood is above the threshold (not used for discrimination). Next, the discrimination unit 33 extracts the product likelihood with the largest value for each segmented image from the processing results of the remaining segmented images. The discrimination unit 33 discriminates the type of the product by majority rule of the products represented by the extracted product likelihoods.

[0140] In the above embodiment, the measuring device 100 is described as an example in which the measuring device 100 includes the control device 3. However, the measuring device 100 may not include the control device 3. In this case, the measuring device 2 only needs to have the function of the control device 3. Alternatively, a single device having the functions of the measuring device 2 and the control device 3 may be provided.

[0141] In the above embodiment, the control device 3 of the weighing device 100 is provided with the discrimination unit 33, and the type of the commodity is discriminated in the control device 3. However, the control device 3 may not include the discrimination unit 33. For example, the type of the commodity may be discriminated by the server 200. In this case, the server 200 sends the discrimination result to the control device 3.

[0142] In the above embodiment, the metering system 1 is described as an example in which the metering system 1 includes the metering device 100 and the server 200. However, the server 200 may not be included. In this case, the metering device 100 only needs to include a learned model generation unit. Alternatively, the metering device 100 may also obtain a learned model generated by another device (computer) and store it in the storage unit.

[0143] In the above embodiment, the measuring system 1 including the measuring instrument 100 and the server 200 is described. However, the present invention may be only the server 200. That is, the present invention may be a device for generating a discrimination model for discriminating the type of a product based on a product image including the product.

[0144] In the above embodiment, the touch panel display 31 of the control device 3 is arranged on the holding portion 12 of the housing 10. However, the touch panel display 31 may be arranged at a position other than the holding portion 12. The touch panel display 31 is preferably arranged near the measuring device 2.

[0145] In the above embodiment, the scale 100 is described as an example in which the display device 4 is provided. However, for example, when the customer operates the scale 100, the display device 4 may not be provided.

[0146] In the above embodiment, the weighing device 2 and the control device 3 are provided in the frame 10 as an example. However, the weighing device 100 is not limited to this. The weighing device 2 (weighing table 21a) and the control device 3 may be provided separately, that is, a so-called separate scale.

[0147] In the above embodiment, an example in which the weight of the commodity S stored in the container P is measured has been described. However, the commodity S may be fruits and vegetables that are not stored in the container P.

Claims

1. A commodity discrimination method, using a discrimination model generated by a learning model generation method, to discriminate the type of the commodity based on an object image containing a commodity with an irregular shape, characterized in that: The learning completed model generation method comprises: An acquisition step of acquiring a product image including the product and product information related to the type of the product; a segmentation step of segmenting the product image into a plurality of regions to obtain a plurality of segmented images; an extracting step of extracting, from the plurality of segmented images, a segmented image in which the occupancy rate of the commodity in the segmented images is higher than a predetermined value; and a generating step, performing machine learning based on the plurality of segmented images extracted in the extracting step to generate a discrimination model for discriminating the type of the commodity, The commodity identification method comprises: A first acquisition step is to segment the object image into a plurality of regions to acquire a plurality of segmented object images; A second acquisition step of acquiring, from the plurality of segmentation target images, a segmentation image in which the occupancy rate of the commodity in the segmentation image is higher than a predetermined value; and The discrimination step is to obtain processing results of the plurality of segmentation target images obtained in the second acquisition step using the discrimination model, and discriminate the type of the commodity based on the processing results.

2. The commodity identification method according to claim 1, characterized in that: In the generating step, the segmented image extracted in the extracting step is associated with the product information to perform machine learning to generate the discrimination model.

3. The commodity identification method according to claim 2, characterized in that: In the extraction step, an extraction model generated by machine learning based on images that do not include the product is used to obtain non-product likelihoods representing the probabilities of the segmented images not including the product for each of the plurality of segmented images, and the segmented images whose non-product likelihoods are below a threshold are extracted.

4. The commodity identification method according to claim 3, characterized in that: The generation of the extraction model includes: The first step is to obtain a non-product image that does not contain the product; The second step is to segment the non-commodity image into a plurality of regions to obtain a plurality of non-commodity segmented images; and The third step is to perform machine learning based on the plurality of non-commodity segmented images to generate the extraction model.

5. The commodity identification method according to claim 4, characterized in that: In the segmentation step, the product image is segmented in such a way that the regions of each of the plurality of segmented images are rectangular and all have the same size; In the second step, the non-product image is segmented in such a manner that the plurality of non-product segmented images have the same shape and size as the segmented image.

6. The commodity identification method according to claim 1, characterized in that: In the extraction step, one of the segmented images in which the occupancy rate of the product in the segmented images is higher than a predetermined value and the other segmented images in which the occupancy rate of the product in the segmented images is less than a predetermined value are extracted from the plurality of segmented images, the product information is associated with one of the segmented images, and non-product information indicating that the product is other than the product is associated with the other segmented images. In the generating step, machine learning is performed based on the one segmented image extracted in the extracting step and the other segmented images to generate the discriminant model.

7. The commodity identification method according to claim 1, characterized in that: In the segmentation step, the product image is segmented so that one segmented image overlaps at least partially with the other segmented image.

8. The commodity identification method according to claim 7, characterized in that: In the segmentation step, the regions for segmenting the product image are set so that the other regions are shifted by a predetermined amount in a first direction, which is a direction in which pixels of the product image are arranged, or in a second direction orthogonal to the first direction, relative to one of the regions.

9. The commodity identification method according to claim 1, characterized in that: In the segmentation step, the product image is segmented so that the regions of the plurality of segmented images are rectangular and all have the same size.

10. The commodity identification method according to claim 1, characterized in that: The learning completed model generation method comprises: a cutting step of comparing the product image with a base image that does not include the product, and cutting out a product area that at least includes the entire product and is smaller than the product image from the product image, In the segmentation step, the commodity area is segmented into a plurality of the areas, and a plurality of the segmented images are acquired.

11. The commodity identification method according to claim 1, characterized in that: In the second acquisition step, an acquisition model generated by machine learning based on images that do not contain the product is used to obtain non-product likelihoods representing the probabilities of the segmented object images not containing the product for each of the plurality of segmented object images, and the segmented object images whose non-product likelihoods are below a threshold are obtained.

12. The commodity identification method according to claim 1, characterized in that: In the first acquisition step, the target image is segmented so that the regions of the plurality of segmentation target images are rectangular and all have the same size.

13. A commodity discrimination method, using a discrimination model generated by a learning model generation method, to discriminate the type of a commodity based on an object image containing a commodity with an irregular shape, characterized in that: The learning completed model generation method comprises: An acquisition step of acquiring a product image including the product and product information related to the type of the product; a segmentation step of segmenting the product image into a plurality of regions to obtain a plurality of segmented images; an extracting step of extracting, from the plurality of segmented images, a segmented image in which the occupancy rate of the commodity in the segmented images is higher than a predetermined value; and a generating step, performing machine learning based on the plurality of segmented images extracted in the extracting step to generate a discrimination model for discriminating the type of the commodity, In the extraction step, one of the segmented images in which the occupancy rate of the product in the segmented images is higher than a predetermined value and the other segmented images in which the occupancy rate of the product in the segmented images is less than a predetermined value are extracted from the plurality of segmented images, the product information is associated with one of the segmented images, and non-product information indicating that the product is other than the product is associated with the other segmented images. In the generating step, machine learning is performed based on the one segmented image and the other segmented images extracted in the extracting step to generate the discriminant model, The commodity identification method comprises: a segmented image acquisition step, dividing the object image into a plurality of regions to acquire a plurality of segmented object images; and A discrimination step, for the plurality of segmented object images acquired in the segmented image acquisition step, acquiring a commodity likelihood indicating the probability of being the commodity and a non-commodity likelihood indicating the probability of being other than the commodity from the processing results processed by the discrimination model, and discriminating the type of the commodity based on the commodity likelihood.

14. The commodity identification method according to claim 13, characterized in that: In the discrimination step, the plurality of segmentation target images are processed respectively using the discrimination model to obtain a plurality of processing results, and the types of the commodities are discriminated based on the plurality of processing results.

15. The commodity identification method according to claim 13, characterized in that: In the determination step, the processing result is weighted according to the degree of the commodity likelihood indicating the probability that the commodity belongs to one category, and the category of the commodity is determined by majority decision of the weights assigned to the processing result.

16. The commodity identification method according to claim 15, characterized in that: In the discrimination step, when a plurality of the commodities are included in the target image, the category of each of the plurality of commodities is discriminated based on the processing result of each region including each of the plurality of commodities.

17. A commodity discrimination device, using a discrimination model generated by a learned model generation device, to discriminate the type of a commodity based on an object image containing a commodity with an irregular shape, characterized in that: The learning completed model generating device comprises: an acquisition unit, which acquires a product image including the product and product information related to the type of the product; A segmentation unit, which segments the product image into a plurality of regions to obtain a plurality of segmented images; An extracting unit extracts, from the plurality of segmented images, a segmented image in which the occupancy rate of the product in the segmented images is higher than a predetermined value; as well as a generating unit, performing machine learning based on the plurality of segmented images extracted by the extracting unit to generate the discriminant model, The commodity identification device comprises: A first acquisition unit, which divides the object image into a plurality of regions and acquires a plurality of segmented object images; A second acquisition unit acquires, from the plurality of segmentation target images, a segmentation image in which the occupancy rate of the commodity in the segmentation image is higher than a predetermined value; as well as The discriminating unit acquires processing results of the plurality of segmentation target images acquired by the second acquiring unit using the discriminating model, and discriminates the type of the commodity based on the processing results.

18. A commodity identification system comprising: a device for generating a discrimination model for discriminating the type of a commodity from a commodity image including a commodity having an irregular shape; and a discrimination device for discriminating the type of the commodity from an object image including the commodity, characterized in that: The generating device comprises: an acquisition unit, which acquires the commodity image and commodity information related to the type of the commodity; A segmentation unit, which segments the product image into a plurality of regions to obtain a plurality of first segmented images; an extracting unit, for extracting, from the plurality of first segmented images, a first segmented image satisfying that an occupation rate of the commodity in the first segmented image is higher than a predetermined value; as well as a generating unit, which associates the plurality of first segmented images extracted by the extracting unit with the commodity information to perform machine learning to generate the discriminant model; The identification device comprises: A first acquisition unit, which divides the object image into a plurality of regions and acquires a plurality of second segmented images; a second acquisition unit, which acquires, from the plurality of second segmented images, a second segmented image in which the occupancy rate of the commodity in the second segmented image is higher than a predetermined value; as well as The discriminating unit acquires a processing result of the plurality of second segmented images acquired by the second acquiring unit using the discriminating model, and discriminates the type of the commodity based on the processing result.

19. A weighing device, using the commodity identification device according to claim 17, to identify the type of the commodity based on an object image containing the commodity and calculate the price of the commodity, characterized in that: have: a weighing section, measuring the weight of the goods; A photography department, photographing the products; a discriminating unit, for obtaining a processing result of the object image photographed by the photographing unit using the discriminating model, and discriminating the type of the commodity according to the processing result; as well as The calculation unit calculates the price of the commodity based on the weight of the commodity measured by the measuring unit and the type of the commodity determined by the determination unit.

Images