IMAGE ANALYSIS SYSTEM, IMAGE ANALYSIS METHOD, AND PROGRAM

Abstract

An image analysis system (1) comprises a reference area detection section (110) and a management unit identification section (120). The reference area detection section (110) can detect a reference area in a product from an image of the product by processing the image. The management unit identification section (120) identifies the management unit of the product using the ratio of the size of the reference area to the size of the image area for the product.

Description

[Technical field]

[0001] The present invention relates to a technique for identifying products using images. [Background technology]

[0002] Technology that identifies products by analyzing images of the product's exterior is used in stores. Some products sold in stores are sold in various sizes (net amounts). For such products, the store needs to manage them by size (net amount).

[0003] An example of a technique for identifying the same product of different sizes is disclosed in, for example, the following Patent Document 1. Patent Document 1 discloses a technique for estimating the actual size of a subject using the actual size ratio per pixel of a query image and the size of a region of the subject in the query image, and estimating the size of the subject in the query image based on the actual size of a subject included in an image of the same cluster as the subject among clusters included in a pre-created list of different sizes. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-095408 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology of Patent Document 1, the actual size ratio per pixel of an image can vary depending on the position when the image was taken (the relative position between the camera and the subject) and the camera settings. Therefore, unless an accurate actual size ratio is obtained for each image to be processed, the actual size of the product shown in the image cannot be reliably estimated.

[0006] The present invention has been made in view of the above problems. It is an object of the present invention to provide a technique for stably identifying the size (net amount) of a product based on an image. [Means for solving the problem]

[0007] The image analysis system according to the present disclosure includes: a reference area detection means for detecting a reference area in the product from an image of the product by processing the image; a management unit specifying means for specifying a management unit of the product using a ratio of the size of the reference area to the size of the image area of ​​the product; Equipped with.

[0008] The image analysis method according to the present disclosure includes: The computer processing an image of an item to detect a reference region within the item from the image; identifying a management unit of the product using a ratio of the size of the reference area to the size of the image area of ​​the product; This includes:

[0009] The program in this disclosure is Computer, a reference area detection means for detecting a reference area in the product from an image of the product by processing the image; a management unit specifying means for specifying a management unit of the product using a ratio of the size of the reference area to the size of the image area of ​​the product; Function as. Effect of the Invention

[0010] According to the present invention, the size (net amount) of a product can be stably identified based on an image. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of an image analysis system according to a first embodiment. [Diagram 2]1 is a block diagram illustrating an example of a hardware configuration of an information processing device having each functional component of an image analysis system. [Diagram 3] 4 is a flowchart illustrating a flow of processing executed by the image analysis system of the first embodiment. [Figure 4] 11 is a diagram useful in explaining an example of a specific operation of a management unit specification part; FIG. [Diagram 5] 13 is a diagram useful in explaining another example of a specific operation of the management unit specification unit; FIG. [Figure 6] FIG. 11 is a diagram illustrating an example of a functional configuration of an image analysis system according to a second embodiment. [Figure 7] FIG. 11 is a diagram showing an example of dictionary data used in the image analysis system of the second embodiment. [Figure 8] 13 is a flowchart showing an example of processing executed by the image analysis system of the second embodiment. [Figure 9] 13 is a flowchart showing an example of processing executed by the image analysis system of the second embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of output information by a product identification unit. [Figure 11] 13 is a flowchart showing another example of the processing executed by the image analysis system of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all drawings, similar components are given similar reference numerals, and descriptions will be omitted as appropriate. In addition, unless otherwise specified, in each block diagram, each block represents a functional configuration, not a hardware configuration. In addition, the direction of the arrows in the drawings is merely for making the flow of information easier to understand. Unless otherwise specified, the direction of the arrows in the drawings does not limit the direction of communication (one-way communication / two-way communication).

[0013] [First embodiment] <Example of functional configuration> Fig. 1 is a diagram showing an example of the functional configuration of an image analysis system according to a first embodiment. The image analysis system 1 shown in Fig. 1 includes a reference area detection unit 110 and a management unit identification unit 120. The reference area detection unit 110 obtains an image in which a product appears as an image to be processed, and processes the image to detect a reference area in the product appearing in the image. The management unit identification unit 120 identifies the management unit of the product using the ratio of the size of the reference area to the image area corresponding to the product.

[0014] Here, the term "management unit" in this disclosure refers to variations of a product of the same type (contents). For example, a store may carry a certain PET bottle beverage in a 350 ml container and a 500 ml container. In this case, variations in the net volume of the PET bottle beverage, such as "350 ml" and "500 ml," correspond to the "management unit" in this disclosure. Note that the management unit is not limited to variations in net volume. For example, product size variations that do not indicate a specific net volume, such as "Large (Large)," "M (Medium)," and "S (Small)," are also included in the "management unit" in this disclosure.

[0015] The "reference area" in this disclosure is used as information for identifying the "management unit" from the image. The "reference area" in this disclosure is any area including an appearance feature that can be used as a criterion for determining the management unit of the same type of product. Note that the appearance feature included in the reference area is preferably a feature that has almost no change in size even if the above-mentioned management unit is different. Specific examples of appearance features that can be included in the reference area include, but are not limited to, the cap part of the product, the sealing part of the product package, the logo of the company or brand, the product display mark, and the display of special precautions. The cap part of the product is a part provided at the mouth of the container that contains the product, and includes, for example, the cap of a plastic bottle, the measuring cap and the measuring nozzle of a liquid detergent container, and the like. The sealing part of the product package is, for example, a part that is subjected to a sealer processing to seal a bag container (package) that contains snacks and the like. The logo of the company or brand is a logo designed for the company that manufactures the product or the brand of the product (for example, the name of the product). The product display mark is a specific mark prescribed for each category, including, for example, the JAS (Japanese Agricultural Standard) mark, the fairness mark, the regional specialty product certification mark, the SQ (Safety & Quality) mark, the certification mark, the Frozen Noodle Association (RMK) certification mark, the specified health food mark, the functional food mark, the self-medication logo mark, the pharmaceutical mark, the recycle mark, etc. Also, the display of specific precautions is some kind of display that calls attention to the care when handling the product (for example, a "Do not mix" display on a chlorine-based detergent product). The reference area detection unit 110 is configured to detect an area that at least partially includes at least one of these marks from within the image area of ​​the product as the "reference area".

[0016] <Hardware configuration of image analysis system 1> Each functional component of the image analysis system 1 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further description will be given of a case where each functional component of the image analysis system 1 is realized by a combination of hardware and software in one information processing device.

[0017] 2 is a block diagram illustrating an example of a hardware configuration of an information processing device 10 having each functional component of the image analysis system 1. The information processing device 10 has a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0018] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0019] The processor 1020 is implemented by a central processing unit (CPU) or a graphics processing unit (GPU).

[0020] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0021] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the image analysis system 1 described in this specification. The processor 1020 loads the program modules onto the memory 1030 and executes them, thereby realizing each function of the image analysis system 1 described in this specification.

[0022] The input / output interface 1050 is an interface for connecting the information processing device 10 to peripheral devices. To the input / output interface 1050, input devices such as a keyboard, a mouse, and a touch panel, and output devices such as a display and a speaker can be connected.

[0023] The network interface 1060 is an interface for connecting the information processing device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method for connecting to the network via the network interface 1060 may be a wireless connection or a wired connection. As an example, the information processing device 10 can communicate with a terminal 20 carried by a store clerk and other external storage devices connected to the network via the network interface 1060.

[0024] The hardware configuration shown in FIG. 2 is merely an example. The hardware configuration of the image analysis system 1 according to the present disclosure is not limited to the example of FIG. 2. For example, the various functions of the image analysis system 1 according to the present disclosure may be implemented in a single information processing device, or may be distributed and implemented in multiple information processing devices. In addition, in the example of FIG. 2, the information processing device 10 having each function of the image analysis system 1 is depicted as a device different from the terminal 20 used by the store clerk, but all or part of the functions of the image analysis system 1 may be provided in the terminal 20 used by the store clerk.

[0025] <Processing flow> FIG. 3 is a flowchart illustrating the flow of processing executed by the image analysis system 1 of the first embodiment.

[0026] First, the reference area detection unit 110 acquires an image of a product captured by an imaging device (not shown) as an image to be processed (S102). The image of the product is captured, for example, by a camera mounted on a terminal carried by a store clerk (for example, terminal 20 shown in FIG. 2). The store clerk captures, for example, an image of the location where the product is displayed (such as a product shelf) using the camera function of the terminal. The reference area detection unit 110 can acquire the image of the product from the terminal, or from a server device (not shown) that collects and stores images generated by the terminal.

[0027] Then, the reference area detection unit 110 extracts image areas corresponding to each object from the acquired image (S104). In the following description, the image areas corresponding to each object are also referred to as "object areas". The reference area detection unit 110 can recognize each object (object area) in the image by using an object recognition model (not shown) learned by a machine learning algorithm such as Deep Learning. As an example, the object recognition model is stored in advance in the storage device 1040 of the information processing device 10 of FIG. 2. As another example, the object recognition model may be stored in an external device (not shown) communicably connected to the information processing device 10 of FIG. 2 via the network interface 1060.

[0028] Then, the reference area detection unit 110 detects a reference area from each of the object areas extracted from the image (S106). As an example, the reference area detection unit 110 uses a learning model (not shown) that has been trained to recognize various types of reference areas from the input image as described above. For example, this learning model may be stored in advance in the storage device 1040 of the information processing device 10 of FIG. 2, or may be stored in an external device (not shown) communicatively connected to the information processing device 10 of FIG. 2 via the network interface 1060. The reference area detection unit 110 can obtain information indicating the reference area in the object area provided as the input from the learning model by providing each object area (image area) extracted in the process of S104 as an input to the learning model.

[0029] The management unit identification unit 120 calculates the ratio of the size of the object area extracted in the process of S104 to the size of the reference area detected for that object area in the process of S106 (S108). For example, the management unit identification unit 120 may obtain the size (number of pixels) of the object area in the height direction and the size (number of pixels) of the reference area detected for that object area, and calculate the ratio. Then, the management unit identification unit 120 identifies the management unit of the product using the ratio of the size of the reference area to the object area calculated in the process of S108 (S110). The specific operation of the management unit identification unit 120 will be described below with reference to the drawings.

[0030] FIG. 4 is a diagram used to explain an example of the specific operation of the management unit identification unit 120. In FIG. 4, for the bottled beverage product "Tea A", a case is illustrated in which an area (the area indicated by diagonal lines in the figure) where the logo of the product brand is drawn on the package of the product is detected as a "reference area". Here, in the example of FIG. 4, the product "Tea A" is sold in two management units of "500 ml" and "350 ml", and it is assumed that the size of the logo of the product brand does not change in the product packages for each management unit. Therefore, as shown in the figure, for the 500-ml product P1 and the 350-ml product P2, the sizes of the logo areas of the product brand (that is, the reference areas) are the same. In particular, in the example of FIG. 4, the sizes in the height direction of the logo areas of the product brand are both "h3". On the other hand, between the 500-ml product P1 and the 350-ml product P2, since the management units are different, the sizes of the entire products (that is, the object areas in the image) are different. In particular, the size in the height direction of the entire product is "h1" for the 500-ml product P1 and "h2 (<h1)" for the 350-ml product P2. Note that the sizes "h1", "h2", and "h3" are obtained as numerical values (for example, the number of pixels in the height direction of each area) based on the number of pixels in the image to be processed, for example.

[0031] The management unit identification unit 120 calculates the ratio of the size of the logo area of the product brand (that is, the size of the reference area in the image) to the size of the entire product (that is, the size of the object area in the image). For example, for the object area corresponding to the 500-ml product P1, the management unit identification unit 120 can calculate a numerical value such as "h3 / h1" as the value of the ratio of the size in the height direction of the reference area occupying the object area. Also, for the object area corresponding to the 350-ml product P2, the management unit identification unit 120 can calculate a numerical value such as "h3 / h2" as the value of the ratio of the size in the height direction of the reference area occupying the object area. In this case, since the relationship "h2 <h1" holds, the value of the ratio calculated for the 500-ml product P1 is significantly smaller than the value of the ratio calculated for the 350-ml product P2.

[0032] 5 is a diagram useful for explaining another example of the specific operation of the management unit specification unit 120. In the example of FIG. 5, merchandise 5, a case is illustrated in which the area of ​​the cap of the PET bottle (the area indicated by diagonal lines in the figure) is detected as the "reference area". Here, PET bottle caps are basically manufactured to a common standard and have the same size. Therefore, the area of ​​the cap of the PET bottle can be used as the reference area, similar to the product brand logo area in the example of FIG. 4. In other words, in the example of FIG. 5, the management unit identification unit 120 executes the same process as that described using FIG. 4. Specifically, the management unit identification unit 120 can calculate a numerical value such as "h4 / h1" as the value of the ratio of the size in the height direction for the 500 ml product P1, and a numerical value such as "h4 / h2" as the value of the ratio of the size in the height direction for the 350 ml product P2. In this case as well, the "h2

[0033] ​In this way, the ratio of the size of the reference area to the object area is useful information for identifying the management unit of the product. For example, the management unit identification unit 120 can identify the management unit of the identified product by using the ratio of the size of the reference area to the object area in combination with the identification result of the product based on the image feature information of the object area. In this case, the identification result of the product corresponding to each object area is supplied, for example, from another processing unit (not shown) that executes the product identification process. The management unit identification unit 120 can identify the management unit of the product by comparing the size ratio calculated in the process of S108 with the size ratio similarly obtained from information indicating the appearance feature of the identified product that is registered in advance for each management unit (e.g., a front image of the product, etc.). In addition, even if there is no identification result of the product, there are cases in which the management unit can be identified using the size ratio of the reference area to the object area obtained by the process as described above. For example, for PET bottle drinks, the size of the product for each management unit (net amount) does not vary greatly depending on the manufacturer or the type of product. In addition, the size of the cap of the PET bottle drink does not vary greatly depending on the manufacturer or the type of product. Therefore, when an area corresponding to a plastic bottle cap is detected as a reference area, the size ratio of the reference area to the object area can be information indicating a management unit (350 ml container, 500 ml container, etc.). Therefore, in such a case, even if a product in an object area including the reference area is unidentified, the management unit identification unit 120 can identify the management unit based on the size ratio of the reference area to the object area.

[0034] <Examples of effects> The positional relationship between the camera and the subject (product) when the image to be processed is taken, the hardware characteristics of the camera, and the like may cause distortion of the subject (product) in the image. For example, when the camera is directed from above the subject (product), the subject (product) is photographed shrinking toward the bottom of the image. When using the actual size ratio per pixel as in the technology disclosed in Patent Document 1, this distortion causes an error between the actual size of the subject and the size of the subject obtained from the image. This may cause the management unit (size, etc.) of the product to be unable to be correctly identified from the image. On the other hand, the image analysis system 1 of this embodiment calculates the size ratio of the area detected from the image. By calculating the ratio in this way, the effect of distortion in the image of the subject is offset. As a result, it becomes possible to stably identify the management unit (size, net amount, etc.) of the product even from an image in which the subject product is distorted.

[0035] [Second embodiment] This embodiment has the same configuration as the first embodiment, except for the points described below.

[0036] <Example of functional configuration> Fig. 6 is a diagram showing an example of the functional configuration of an image analysis system according to the second embodiment. The image analysis system 1 shown in Fig. 6 includes a product identification unit 130 in addition to the reference area detection unit 110 and the management unit identification unit 120 described in the first embodiment.

[0037] The product identification unit 130 identifies products appearing in the image based on image feature information obtained from the product image acquired as the processing target and dictionary data including information indicating the appearance features of each product (hereinafter also referred to as "product appearance information"). The product identification unit 130 compares the image feature information obtained from the product image with the product appearance information of each product included in the dictionary data, and calculates the degree of match (degree of match of feature points) between the appearance features of the product appearing in the image and each product registered in the dictionary data. Then, the product identification unit 130 identifies products having appearance features that show a degree of match equal to or higher than a predetermined standard as candidates for products appearing in the image acquired as the processing target.

[0038] FIG. 7 is a diagram showing an example of dictionary data used in the image analysis system 1 of the second embodiment. In the dictionary data, various information related to a product is registered for each product (product in a management unit). For example, the dictionary data shown in FIG. 7 includes, for each product, product identification information, product name, management unit, product appearance information, and reference area type as various information related to the product. The product appearance information is any information indicating the appearance of the product. For example, the product appearance information may be a sample image obtained by photographing a product from the front, or may be image feature information generated based on such a sample image, or may be a combination of these. The reference area information is information for identifying the type of the reference area to be detected for the product. The dictionary data is stored, for example, in the storage device 1040 of the information processing device 10 of FIG. 2.

[0039] Here, as a result of the processing by the product identification unit 130, multiple products may be identified as candidates for products having appearance characteristics that show a degree of match equal to or higher than a predetermined standard. For example, consider a case where the product shown in the image is a plastic bottle drink (for convenience, referred to as "Tea A"), and "Tea A in a 350 ml container" and "Tea A in a 500 ml container" are handled in a store. These "Tea A in a 350 ml container" and "Tea A in a 500 ml container" are the same product in terms of the type of product, but are different products in terms of the management unit in the store. Therefore, information about each of these two products is registered in the dictionary data. Here, when the types of products are the same, their appearance characteristics (package designs) are basically similar. Therefore, when the product identification unit 130 identifies products based on images, "Tea A in a 350 ml container" and "Tea A in a 500 ml container" may each be identified as "products having appearance characteristics that show a degree of match equal to or higher than a standard value."

[0040] In the image analysis system 1 of this embodiment, as an example, when multiple products that are identical except for the management unit are identified in this way, the reference area detection unit 110 and the management unit identification unit 120 execute processing (the processing as described in the first embodiment). Then, the image analysis system 1 of this embodiment uniquely identifies the product appearing in the image to be processed based on the management target identified by the management unit identification unit 120.

[0041] <Processing flow> 8 and 9 are flowcharts showing an example of processing executed by the image analysis system 1 of the second embodiment.

[0042] First, the product specification unit 130 processes the image of the product captured by an imaging device (not shown). subjectThe image of the product is acquired as an image of the product (S202). The image of the product is captured, for example, by a camera mounted on a terminal carried by a store clerk (for example, terminal 20 shown in FIG. 2). The store clerk captures, for example, an image of the location where the product is displayed (such as a product shelf) using the camera function of the terminal. The product identification unit 130 can acquire the image of the product from the terminal, or from a server device (not shown) that collects and stores images generated by the terminal.

[0043] Then, the product identification unit 130 extracts image regions (object regions) corresponding to each object from the acquired image (S204). The product identification unit 130 can recognize each object (object region) in the image by using an object recognition model (not shown) learned by a machine learning algorithm such as Deep Learning. As an example, the object recognition model is stored in advance in the storage device 1040 of the information processing device 10 of FIG. 2. As another example, the object recognition model may be stored in an external device (not shown) communicatively connected to the information processing device 10 of FIG. 2 via the network interface 1060.

[0044] Then, the product identification unit 130 generates image feature information for each of the extracted object regions (S206). The product identification unit 130 can generate various image feature information from each object region using a known method.

[0045] Then, the product identification unit 130 compares the image feature information generated for each object region with the product appearance information of each product included in the dictionary data (S208). Based on the comparison result, the product identification unit 130 identifies, from among the products registered in the dictionary data, products having appearance features that show a degree of match with the image feature information obtained from a certain object region that is equal to or higher than a reference value (S210).

[0046] Then, the product identification unit 130 judges whether or not multiple products are identified as a result of the process of S208 (S212). Here, if only one product is identified as a product having an appearance characteristic that shows a degree of matching with the image feature information obtained from the object region equal to or higher than a reference value (S212: NO), the product identification unit 130 outputs information on the identified product (S226). On the other hand, if multiple products are identified as products having an appearance characteristic that shows a degree of matching with the image feature information obtained from the object region equal to or higher than a reference value (S212: YES), the product identification unit 130 further judges whether or not the multiple identified products are the same product (S214). Note that, in the process of S214, the product identification unit 130 judges whether or not they are the "same product" based on the "type (contents) of the product". In other words, the product identification unit 130 does not consider the difference or similarity of the "management target" for the multiple identified products. For example, two products that have the same contents but different net amounts (products of different sizes) are determined to be "the same product" in the process of S214.

[0047] If the identified multiple products are not the same product (S214: NO), the product identification unit 130 identifies one product based on the degree of match calculated for each product (S216). As an example, the product identification unit 130 can select one product with the highest degree of match from the identified multiple products and obtain the selected one product as the final identification result. Then, the product identification unit 130 outputs information on the identified product (S226). On the other hand, if the identified multiple products are the same product, that is, if the products are the same except for the management unit (S214: YES), the product identification unit 130 requests the reference area detection unit 110 and the management unit identification unit 120 to execute processing.

[0048] In response to a request from the commodity identification unit 130, the reference area detection unit 110 detects a reference area for each of the object areas extracted in the process of S204 (S218). Here, for example, as shown in FIG. 7, the type of the reference area to be detected may be determined for each commodity. The reference area detection unit 110 can identify information (reference area information) indicating the type of the reference area linked to the commodity identification result (commodity identification information) by the commodity identification unit 130, based on the commodity identification result (commodity identification information) of the commodity and dictionary data as shown in FIG. 7. Then, as described in the first embodiment, the reference area detection unit 110 detects a reference area corresponding to the identified type from within the object area.

[0049] When the reference area is detected by the reference area detection unit 110, the management unit identification unit 120 calculates the ratio of the size of the object area extracted in the process of S204 to the size of the reference area detected for that object area in the process of S218 (S220). This process is similar to the process of S108 in the flowchart of Fig. 3. Then, the management unit identification unit 120 uses the ratio of the size of the reference area to the object area calculated in the process of S220 to identify the management unit of the product in that object area (S222).

[0050] The product identification unit 130 finally identifies one product from among the multiple products that are identical except for the management unit, based on the identification result of the management unit by the management unit identification unit 120 (S224). Then, the product identification unit 130 outputs information on the one product finally identified (S226).

[0051] The above-mentioned processes of S218 to S226 will be described with a more specific example. For example, assume that the product identification unit 130 compares the image feature information of a certain object region with the product appearance information of each product in the dictionary data, and identifies two products, "Tea A in a 350 ml container" and "Tea A in a 500 ml container," as products that show a degree of match above a standard. These two products are the same in terms of product type, both being "Tea A," but differ in terms of the management unit, one being "350 ml" and the other being "500 ml."

[0052] In this case, the product identification unit 130 issues a processing execution request to the reference area detection unit 110 and the management unit identification unit 120. Upon receiving the request from the product identification unit 130, the reference area detection unit 110 detects a reference area from the object area from which such a product identification result has been obtained. Based on information such as that illustrated in FIG. 7, the reference area detection unit 110 can identify at least one of a "product brand logo" and a "plastic bottle cap" as the type of reference area corresponding to the identification result (tea A) of the product identification unit 130. Then, the reference area detection unit 110 detects at least one of an area corresponding to the "product brand logo" and an area corresponding to the "plastic bottle cap" from the object area as a reference area.

[0053] Then, the management unit specification unit 120 calculates the ratio of the size of the reference area detected by the reference area detection unit 110 to the object area. Then, the product specification unit 130 specifies the management unit of the product in the object area using the ratio of the size of the reference area to the object area. Here, the ratio of the size of the reference area (area corresponding to the cap of a plastic bottle or the logo of the product brand) to the object area can be calculated in advance, for example, based on a value actually measured for each product (product of the management unit). When such information is linked in advance to the product identification information of FIG. 7, for example, the management unit specification unit 120 can specify the management unit based on the similarity between the actual measurement value linked to the product specified by the product specification unit 130 and the ratio value calculated by the management unit specification unit 120. For example, for "Tea A in a 350 ml container" and "Tea A in a 500 ml container", the actual measurement results of "2:1" and "4:1" are obtained as the ratio of the height direction size of the product brand logo (reference area) to the entire area (object area) of each product container, and information on these actual measurement results is stored in advance. In this case, if the ratio of the size of the reference area (the logo of the product brand) to the object area calculated by the management unit specifying unit 120 is closer to "4:1" than "2:1", the management unit specifying unit 120 can specify the management unit of the product in the object area as "500 ml (500 ml container)". On the other hand, if the ratio of the size of the reference area (the logo of the product brand) to the object area calculated by the management unit specifying unit 120 is closer to "2:1" than "4:1", the management unit specifying unit 120 can specify the management unit of the product in the object area as "350 ml (350 ml container)". In addition, the management unit specifying unit 120 may calculate a theoretical value regarding the ratio of the size of the reference area to the object area using the appearance characteristic information of the product specified by the product specifying unit 130 (for example, a sample image of the product photographed from the front). Even in this case, the management unit specifying unit 120 can similarly specify the management unit based on the similarity of the ratio of the size of the reference area to the object area. Then, the product specifying unit 130 returns information on the management unit thus specified as a response to the request of the product specifying unit 130.

[0054] The product identification unit 130 finally identifies one product from among multiple products that are identical except for the management unit, based on the management unit identified by the management unit identification unit 120. For example, if the management unit identified by the management unit identification unit 120 is "350ml (350ml container)", the product identification unit 130 determines "Tea A in a 350ml container" as the final product identification result from the products "Tea A in a 350ml container" and "Tea A in a 500ml container" identified in the process of S210. Then, the product identification unit 130 outputs information on the finally identified product. The product identification unit 130 can output, for example, information as shown in FIG. 10 on an arbitrarily selected output device as a result obtained by performing the above-mentioned process on each object region detected from the image to be processed.

[0055] FIG. 10 is a diagram showing an example of output information by the product identification unit 130. FIG. 10 illustrates an example of an output screen when an image IMG capturing at least a part of a product shelf on which a plurality of products are displayed is acquired as an image to be processed. The product identification unit 130 superimposes a display d1 showing each object region extracted from the image IMG and a display d2 showing information on the product identified for each object region (e.g., product name, management unit, etc.) on the image IMG to generate data for the output screen. The product identification unit 130 transmits the output screen data thus generated to, for example, a terminal 20 used by a store clerk and displays it on the display of the terminal 20. The store clerk can easily grasp the display state on the product shelf (e.g., whether each product is displayed in the correct position, etc.) by referring to the screen shown in FIG. 10 displayed on the terminal 20.

[0056] <Modification> In the above example, the process of identifying the management unit is executed after the process of identifying the product, but the process of identifying the management unit may be executed before the process of identifying the product. In this case, the image analysis system 1 executes the process described below.

[0057] FIG. 11 is a flowchart showing another example of the processing executed by the image analyzing system 1 of the second embodiment.

[0058] The processes from S302 to S310 in Fig. 11 are similar to the processes from S102 to S110 in Fig. 3. Through these processes, a management unit is specified based on the ratio of the size of the reference area to the object area.

[0059] Based on the identified management unit, the product identification unit 130 selects product appearance information to be compared with the image feature information obtained from the image from among the product appearance information of each product registered in the dictionary data (S312). For example, the reference area detection unit 110 detects the area of ​​the "PET bottle cap" as a reference area from an object area, and the management unit identification unit 120 identifies a management unit of "350 ml" based on the ratio of the size of the reference area to the object area. In this case, the product identification unit 130 selects the product appearance information of a product whose management unit is "350 ml" as information to be compared when identifying a product. For example, when dictionary data such as that shown in FIG. 7 is stored, the product identification unit 130 selects the product appearance information of a product whose management unit is "350 ml" including at least the product appearance information linked to "product identification information: 0001" and the product appearance information linked to "product identification information: 0003" as a comparison target.

[0060] Then, the product identification unit 130 generates image feature information for each of the object regions extracted in the process of S304 (S314). Then, the product identification unit 130 calculates the degree of match between the image feature information of each object region generated in the process of S314 and each of the product appearance information selected in the process of S312, and identifies a product that shows a degree of match equal to or greater than a reference value for each object region (S316). Here, if multiple products that show a degree of match equal to or greater than the reference value are identified, the product identification unit 130 can select, for example, one product with the highest degree of match, and obtain the selected product as the final identification result. Then, the product identification unit 130 outputs information about the identified product (S318).

[0061] According to this modification, the management unit It is possible to narrow down (reduce) the amount of information to be compared based on the management unit. This is expected to have the effect of shortening the time required for the process of identifying products. In addition, by narrowing down the comparison targets using the management unit, it is expected to have the effect of preventing misrecognition between products of the same type but different sizes.

[0062] Although the embodiments of the present invention have been described above with reference to the drawings, the present invention should not be interpreted as being limited thereto, and various modifications, improvements, etc. can be made based on the knowledge of a person skilled in the art without departing from the gist of the present invention. In addition, the multiple components disclosed in the embodiments can be appropriately combined to form various inventions. For example, some components may be deleted from all the components shown in the embodiments, or components of different embodiments may be appropriately combined.

[0063] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that the content is not affected. In addition, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0064] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. a reference area detection means for detecting a reference area in the product from an image of the product by processing the image; a management unit specifying means for specifying a management unit of the product using a ratio of the size of the reference area to the size of the image area of ​​the product; An image analysis system comprising: 2. The method further comprises a product identification means for identifying a product having an appearance characteristic that shows a degree of match with the image characteristic information by comparing image characteristic information obtained from the image with product appearance information that shows an appearance characteristic of each product, When a comparison is made between the image characteristic information and the product appearance information and a plurality of products are identified that are identical except for the management unit, The reference area detection means detects the reference area from the image, The management unit specifying means specifies a management unit of the product. 1. The image analysis system according to claim 1. 3. The method further comprises a product identification means for identifying a product having an appearance characteristic that shows a degree of match with the image characteristic information by comparing image characteristic information obtained from the image with product appearance information that shows an appearance characteristic of each product, the product identification means selects product appearance information to be compared with the image feature information from the product appearance information of each of the products based on the identified management unit; 1. The image analysis system according to claim 1. 4. The reference area is determined for each product. 4. An image analysis system according to any one of 1 to 3. 5. the reference area detection means detects an area corresponding to at least one of a cap portion of a product, a sealing portion of a product package, a company or product brand logo, a product display mark, and a special warning display as the reference area; 5. An image analysis system according to any one of 1 to 4. 6. the management unit specifying means specifies the management unit using a ratio of a height direction size of the reference area to an image area of ​​the product. 5. An image analysis system according to any one of 1 to 4. 7. The computer processing an image of an item to detect a reference region within the item from the image; identifying a management unit of the product using a ratio of the size of the reference area to the size of the image area of ​​the product; The image analysis method includes: 8. The computer, By comparing image feature information obtained from the image with product appearance information indicating the appearance features of each product, products having appearance features that match the image feature information to a standard or higher are identified; When a comparison is made between the image characteristic information and the product appearance information and a plurality of products are identified that are identical except for the management unit, Detecting the reference region from the image; Identifying a management unit of the product; The image analysis method according to 7., further comprising: 9. The computer, By comparing image feature information obtained from the image with product appearance information indicating the appearance features of each product, products having appearance features that match the image feature information to a standard or higher are identified; selecting, based on the identified management unit, product appearance information to be compared with the image feature information from among the product appearance information of each of the products; The image analysis method according to 8., further comprising: 10. The reference area is determined for each product. 10. An image analysis method according to any one of 7. to 9. 11. The computer, Detecting an area corresponding to at least one of a cap portion of a product, a sealing portion of a product package, a company or product brand logo, a product display mark, and a special warning display as the reference area; The image analysis method according to any one of 7. to 10., 12. The computer, Identifying the management unit using a ratio of a height direction size of the reference area to an image area of ​​the product. The image analysis method according to any one of 7. to 10., 13. Computer, a reference area detection means for detecting a reference area in the product from an image of the product by processing the image; a management unit specifying means for specifying a management unit of the product using a ratio of the size of the reference area to the size of the image area of ​​the product; A program to function as a 14. The computer, and further functioning as a product identification means for identifying a product having an appearance characteristic that shows a degree of match with the image characteristic information or higher by comparing image characteristic information obtained from the image with product appearance information that shows an appearance characteristic of each product, When a comparison is made between the image characteristic information and the product appearance information and a plurality of products are identified that are identical except for the management unit, The reference area detection means detects the reference area from the image, The management unit specifying means specifies a management unit of the product. 13. The program described in. 15. The computer, and further functioning as a product identification means for identifying a product having an appearance characteristic that shows a degree of match with the image characteristic information or higher by comparing image characteristic information obtained from the image with product appearance information that shows an appearance characteristic of each product, the product identification means selects product appearance information to be compared with the image feature information from the product appearance information of each of the products based on the identified management unit; 13. The program described in. 16. The reference area is determined for each product. 16. The program according to any one of 13 to 15. 17. the reference area detection means detects an area corresponding to at least one of a cap portion of a product, a sealing portion of a product package, a company or product brand logo, a product display mark, and a special warning display as the reference area; 17. A program according to any one of 13. to 16. 18. the management unit specifying means specifies the management unit using a ratio of a height direction size of the reference area to an image area of ​​the product. 17. A program according to any one of 13. to 16. [Explanation of symbols]

[0065] 1. Image analysis system 110 Reference area detection unit 120 Management Unit Identification Department 130 Product Specification Department 10. Information processing device 1010 Bus 1020 Processor 1030 Memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface 20 Terminals

Claims

1. a reference area detection means for processing an image of a commodity to extract a predetermined partial appearance feature of the commodity, and detecting an area including the extracted partial appearance feature as a reference area from the image; a management unit specifying means for specifying a management unit indicating a variation in volume or size of the product by using a ratio of a size of the reference area to an image area of ​​the product; An image analysis system comprising:

2. The method further comprises a product identification means for identifying a product having an appearance characteristic that shows a degree of match with the image characteristic information by comparing image characteristic information obtained from the image with product appearance information that shows an appearance characteristic of each product, When a comparison is made between the image characteristic information and the product appearance information and a plurality of products are identified that are identical except for the management unit, The reference area detection means detects the reference area from the image, The management unit specifying means specifies a management unit of the product. The image analysis system according to claim 1 .

3. The method further comprises a product identification means for identifying a product having an appearance characteristic that shows a degree of match with the image characteristic information by comparing image characteristic information obtained from the image with product appearance information that shows an appearance characteristic of each product, the product identification means selects product appearance information to be compared with the image feature information from the product appearance information of each of the products based on the identified management unit; The image analysis system according to claim 1 .

4. The reference area is determined for each product. The image analysis system according to any one of claims 1 to 3.

5. the reference area detection means extracts at least one of a cap portion of a product, a sealing portion of a product package, a company or product brand logo, a product display mark, and a special warning display as the partial appearance feature; The image analysis system according to any one of claims 1 to 4.

6. the management unit specifying means specifies the management unit using a ratio of a height direction size of the reference area to an image area of ​​the product. The image analysis system according to any one of claims 1 to 4.

7. The computer extracting predetermined partial appearance features of the product by processing an image of the product, and detecting an area including the extracted partial appearance features from the image as a reference area; identifying a management unit that indicates a variation in volume or size of the product using a ratio of the size of the reference area to the image area of ​​the product; The image analysis method includes:

8. Computer, a reference area detection means for extracting predetermined partial appearance features of the product by processing an image of the product, and detecting an area including the extracted partial appearance features from the image as a reference area; a management unit specifying means for specifying a management unit indicating a variation in volume or size of the product by using a ratio of a size of the reference area to an image area of ​​the product; A program to function as a

Images