IMAGE ANALYSIS SYSTEM, IMAGE ANALYSIS METHOD, AND PROGRAM

Abstract

This image analysis system (1) comprises a commodity identification result acquisition unit (110) and a correction necessity assessment unit (120). The commodity identification result acquisition unit (110) acquires an identification result for each of a plurality of commodities appearing in an image. When the identification result of a first commodity among the plurality of commodities differs from the identification result of a second commodity adjacent to the first commodity, the correction necessity assessment unit (120) calculates a first degree of coincidence of the image region of the first commodity and the image region of the second commodity. The correction necessity assessment unit (120) assesses, on the basis of the first degree of coincidence, whether or not the identification result of the first commodity or the identification result of the second commodity needs to be corrected.

Description

[Technical field]

[0001] The present invention relates to a technique for identifying products using images. [Background technology]

[0002] There is a technology that performs image processing on an image of a place where products are displayed, such as products in a store, to identify the products displayed in that place. With such a technology, there are cases where the product identification fails or the product is erroneously identified due to various factors. Therefore, a technology that improves the accuracy of product identification is desired.

[0003] An example of a technology for improving the accuracy of identifying products is disclosed in, for example, the following Patent Document 1. Patent Document 1 discloses a technology for detecting a product area of ​​each product from an image of a product shelf on which multiple products are arranged, and determining the validity of the product recognition result for a target product area based on the relevance of the product recognition result between the target product area and an adjacent product area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 107157 Summary of the Invention [Problem to be solved by the invention]

[0005] Basically, image-based processing requires a large amount of calculations, so when performing image processing on the entire image to improve the accuracy of identifying products, there is a risk that the processing time will increase beyond an acceptable range.

[0006] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide a technique for improving the accuracy of identifying products using images while suppressing the overall amount of processing. [Means for solving the problem]

[0007] The image analysis system according to the present disclosure includes: A product identification result acquisition means for acquiring an identification result for each of a plurality of products appearing in an image; a correction necessity determination means for, when a classification result of a first product among the plurality of products differs from a classification result of a second product adjacent to the first product, calculating a first degree of coincidence which is a degree of coincidence between an image area of ​​the first product and an image area of ​​the second product, and determining whether or not the classification result of the first product or the classification result of the second product needs to be corrected based on the first degree of coincidence; Equipped with.

[0008] The image analysis method according to the present disclosure includes: The computer Obtain the identification results for each of the multiple products in the image, If an identification result of a first product among the plurality of products differs from an identification result of a second product adjacent to the first product, a first degree of match is calculated, which is a degree of match between an image area of ​​the first product and an image area of ​​the second product; determining whether or not the identification result of the first product or the identification result of the second product needs to be corrected based on the first degree of match; This includes:

[0009] The program in this disclosure is The computer is caused to carry out the image analysis method described above. Effect of the Invention

[0010] According to the present invention, it is possible to improve the accuracy of identifying products using images while suppressing the overall amount of processing. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of an image analysis system according to a first embodiment. [Diagram 2]1 is a block diagram illustrating a hardware configuration of an information processing device having each functional component of an image analysis system. [Diagram 3] 4 is a flowchart illustrating a flow of processing executed by the image analysis system of the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an image provided to the image analysis system as a processing target. [Diagram 5] 5 is a diagram illustrating an example of the image area of ​​each product shown in the image of FIG. 4. [Figure 6] 5 is a diagram showing an example of the classification results for each of a plurality of products shown in the image of FIG. 4. [Figure 7] FIG. 11 is a diagram illustrating an example of a functional configuration of an image analysis system according to a second embodiment. [Figure 8] 10 is a flowchart illustrating a flow of processing executed by an image analysis system according to a second embodiment. [Figure 9] FIG. 2 is a diagram showing an example of an image provided to the image analysis system as a processing target. [Figure 10] 10 is a diagram illustrating an example of a detection result of a placement member in the image of FIG. 9. FIG. [Figure 11] 13A and 13B are diagrams illustrating an example of a processing result obtained by a product identification result acquisition unit and a placement member detection unit. [Figure 12] FIG. 13 is a diagram illustrating an example of a functional configuration of an image analysis system according to a third embodiment. [Figure 13] 13 is a flowchart illustrating the flow of processing executed by the image analysis system of the third embodiment. [Figure 14] FIG. 13 is a diagram illustrating an example of information output by a correction necessity determining unit according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all drawings, similar components are given similar reference numerals, and descriptions will be omitted as appropriate. In addition, unless otherwise specified, in each block diagram, each block represents a functional configuration, not a hardware configuration. In addition, the direction of the arrows in the drawings is merely for making the flow of information easier to understand. Unless otherwise specified, the direction of the arrows in the drawings does not limit the direction of communication (one-way communication / two-way communication).

[0013] [First embodiment] <Example of functional configuration> Fig. 1 is a diagram illustrating an example of the functional configuration of an image analysis system according to the first embodiment. The image analysis system 1 illustrated in Fig. 1 includes a commodity identification result acquisition unit 110, a correction necessity determination unit 120, and a commodity identification result correction unit 130.

[0014] The commodity identification result acquisition unit 110 acquires an image in which a plurality of commodities are shown. The commodity identification result acquisition unit 110 also acquires the identification results of each of the plurality of commodities shown in the image. For example, the commodity identification result acquisition unit 110 can acquire image areas corresponding to each commodity shown in the image and the identification results of the commodities in each image area by inputting the acquired image to a commodity recognition model that has been trained in advance to be able to identify various commodities. In addition, if the commodity recognition process has already been performed on the acquired image by an external device (not shown), the commodity identification result acquisition unit 110 may acquire the image processing results by the external device (information indicating the image area of ​​each commodity and the identification results of the commodities in each image area) together with the image to be processed.

[0015] The correction necessity determination unit 120 determines whether or not the recognition result of each product needs to be corrected based on the information acquired by the product recognition result acquisition unit 110. Basically, products of the same type are displayed together in a store. Therefore, it can be said that the characteristics (appearance characteristics) of the image areas corresponding to each product are basically similar, except for places where the product type actually changes. Using this characteristic, the correction necessity determination unit 120 determines whether or not the recognition result needs to be corrected by comparing image areas in parts where the product recognition results differ.

[0016] First, the correction necessity determination unit 120 identifies a pair of products that have different product identification results and are adjacent to each other from among multiple products shown in the image. The correction necessity determination unit 120 can identify such a pair of products based on, for example, information acquired by the product identification result acquisition unit 110 (the position of the image area of ​​each product and the product identification result in each image area). In the following description, for convenience, one of the product pair is also referred to as the "first product" and the other as the "second product". Then, when the identification result of the first product and the identification result of the second product are different, the correction necessity determination unit 120 calculates the degree of agreement between the image area corresponding to the first product and the image area corresponding to the second product. For example, the correction necessity determination unit 120 can calculate the degree of agreement between the image areas by extracting various feature amounts from each image area and comparing the feature amounts extracted from each image area. In the following description, the degree of agreement calculated here is also referred to as the "first degree of agreement". Then, the correction necessity determining unit 120 determines whether or not the classification result of the first product or the classification result of the second product needs to be corrected based on the calculated first degree of coincidence.

[0017] Here, as described above, it can be said that the features of the image regions corresponding to each product are basically similar, except for the places where the product type actually changes. Therefore, if the first degree of coincidence is equal to or greater than the reference value for determining that "the two are similar", the first product and the second product are considered to be the same product in reality, and there is a possibility that either the identification result of the first product or the identification result of the second product is incorrect. Therefore, in this case, the correction necessity determination unit 120 determines that the identification result of the first product or the identification result of the second product needs to be corrected. On the other hand, if the first degree of coincidence does not meet the reference value for determining that "the two are similar", the first product and the second product are considered to be different products in reality, and there is a low possibility that the identification result of the first product and the identification result of the second product are incorrect. Therefore, in this case, the correction necessity determination unit 120 determines that there is no need to correct the identification result of the first product or the identification result of the second product.

[0018] As described above, the product identification result correction unit 130 corrects one of the first product identification result and the second product identification result in response to determining that the first product identification result or the second product identification result needs to be corrected based on the first degree of coincidence. The specific method will be described later, but the product identification result correction unit 130 aligns the identification result of either the first product or the second product with the identification result of the other product.

[0019] <Hardware configuration of image analysis system 1> Each functional component of the image analysis system 1 may be realized by hardware that realizes each functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a further description will be given of a case where each functional component of the image analysis system 1 is realized by a combination of hardware and software in one information processing device.

[0020] 2 is a block diagram illustrating an example of a hardware configuration of an information processing device 10 having each functional component of the image analysis system 1. The information processing device 10 has a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0021] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0022] The processor 1020 is implemented by a central processing unit (CPU) or a graphics processing unit (GPU).

[0023] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0024] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the image analysis system 1 described in this specification. The processor 1020 loads the program modules onto the memory 1030 and executes them to realize each function of the image analysis system 1 described in this specification (such as the commodity identification result acquisition unit 110, the correction necessity determination unit 120, and the commodity identification result correction unit 130).

[0025] The input / output interface 1050 is an interface for connecting the information processing device 10 to peripheral devices. To the input / output interface 1050, input devices such as a keyboard, a mouse, and a touch panel, and output devices such as a display and a speaker can be connected.

[0026] The network interface 1060 is an interface for connecting the information processing device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method for connecting to the network via the network interface 1060 may be a wireless connection or a wired connection. As an example, the information processing device 10 can communicate with a terminal 20 carried by a store clerk and other external devices connected to the network via the network interface 1060.

[0027] The hardware configuration shown in FIG. 2 is merely an example. The hardware configuration of the image analysis system 1 according to the present disclosure is not limited to the example of FIG. 2. For example, the various functions of the image analysis system 1 according to the present disclosure may be implemented in a single information processing device, or may be distributed and implemented in multiple information processing devices. In addition, in the example of FIG. 2, the information processing device 10 having each function of the image analysis system 1 is depicted as a device different from the terminal 20 used by the store clerk, but all or part of the functions of the image analysis system 1 may be provided in the terminal 20 used by the store clerk.

[0028] <Processing flow> FIG. 3 is a flowchart illustrating the flow of processing executed by the image analysis system 1 of the first embodiment.

[0029] First, the product identification result acquisition unit 110 acquires an image of a product captured by an imaging device (not shown) as an image to be processed (S102). The product image is captured, for example, by a camera mounted on a terminal carried by a store clerk (for example, terminal 20 shown in FIG. 2). The store clerk captures, for example, an image of the location where the product is displayed (such as a product shelf) using the camera function of the terminal. The product identification result acquisition unit 110 can acquire the product image from the terminal, or from a server device (not shown) that collects and stores images generated by the terminal.

[0030] Then, the commodity identification result acquisition unit 110 extracts image areas corresponding to individual objects (commodity) from the acquired image (S104). The commodity identification result acquisition unit 110 can recognize individual objects (objects assumed to be some commodity) in the image by using an object recognition model (not shown) learned by a machine learning algorithm such as Deep Learning. Here, "recognition" includes identifying the position of the image area corresponding to the object (e.g., position coordinates in an image coordinate system). As an example, the object recognition model is stored in advance in, for example, the storage device 1040 of the information processing device 10 of FIG. 2. As another example, the object recognition model may be stored in an external device (not shown) communicably connected to the information processing device 10 of FIG. 2 via the network interface 1060.

[0031] Then, the commodity identification result acquisition unit 110 uses each of the extracted image regions to identify a commodity corresponding to each image region (S106). For example, the commodity identification result acquisition unit 110 generates image feature information from each image region using a known method, and calculates a similarity between the image feature information generated from each image region and master information of each commodity (feature information to be compared when identifying each commodity). Note that the master information of each commodity is stored, for example, in the storage device 1040 of the information processing device 10 in FIG. 2 or an external storage device (not shown) communicably connected via the network interface 1060. Then, the commodity identification result acquisition unit 110 identifies a commodity corresponding to each image region based on the similarity calculated using the image feature information of each image region.

[0032] The above-mentioned processes of S104 and S106 may be executed by an external device (not shown). In this case, the commodity identification result acquisition unit 110 acquires, in the process of S102, the image to be processed and the result of processing the image (information including the detected position of the image area of ​​each commodity in the image and the identification result of the commodity in each image area) from the external device.

[0033] The correction necessity determination unit 120 identifies the first and second products to be processed based on the position of each image region and the product identification results corresponding to each image region (S108). For example, the correction necessity determination unit 120 compares the product identification results for two image regions adjacent to each other in either the up, down, left, or right direction. If the product identification results for the two image regions are different, the correction necessity determination unit 120 can identify the two image regions as image regions corresponding to the first and second products to be processed, respectively.

[0034] Then, the correction necessity determination unit 120 calculates the degree of coincidence (first degree of coincidence) between the two identified image regions, i.e., the image region of the first product and the image region of the second product (S110). The correction necessity determination unit 120 extracts, for example, various image feature information that may indicate the external appearance features of the products from each image region using a known method. Then, the correction necessity determination unit 120 compares the image feature information extracted from the image region of the first product and the image feature information extracted from the image region of the second product. Product A first degree of match between the two image regions is calculated by comparing the image feature information extracted from the image regions.

[0035] Then, the correction necessity determination unit 120 determines whether the first degree of coincidence calculated in the process of S110 is equal to or greater than a predetermined reference value (S112). The predetermined reference value is a value indicating a criterion for determining that objects (products) shown in each image area are identical in appearance, and an appropriate value is set in advance. The predetermined reference value is set in advance, for example, in the memory 1030 or the storage device 1040 of the information processing device 10 in FIG. 2. The correction necessity determination unit 120 can refer to this reference value and determine whether the degree of coincidence calculated in S110 is equal to or greater than the reference value.

[0036] If the first degree of coincidence calculated in the process of S110 is less than a predetermined reference value (S112: NO), it can be said that there is a high probability that the first product and the second product are actually different products. In this case, the correction necessity determination unit 120 determines that correction is not necessary and does not execute the process of S114 described below. On the other hand, if the first degree of coincidence calculated in the process of S110 is equal to or greater than a predetermined reference value (S112: YES), it can be said that there is a high probability that the first product and the second product are actually the same product, and that there is an error in the identification result of one of them. Therefore, the correction necessity determination unit 120 determines that correction is necessary. In this case, the product identification result correction unit 130 corrects the identification result regarding the first product and the identification result regarding the second product (S114).

[0037] The above-mentioned processing will be described in more detail with reference to the drawings. Note that the processing described below is merely an example, and the processing of the image analysis system 1 according to the present disclosure is not limited to the content described below.

[0038] Fig. 4 is a diagram showing an example of an image provided as a processing target to the image analysis system 1. When the commodity identification result acquisition unit 110 acquires an image as shown in Fig. 4, it detects image regions corresponding to each object (commodity) in the image as shown in Fig. 5, for example, by using an object recognition model stored in the storage device 1040 of the information processing device 10 in Fig. 2.

[0039] FIG. 5 is a diagram illustrating image areas of each commodity shown in the image of FIG. 4. The commodity identification result acquisition unit 110 identifies a plurality of image areas in the image, as shown by dotted rectangles in FIG. 5. In the following description, when distinguishing between the image areas, reference numerals 50-1 to 50-4 are used as shown in the figure. At this time, the commodity identification result acquisition unit 110 generates, for example, information indicating the shape of the image area for each of the image areas 50-1 to 50-4 (e.g., information indicating the positional information of each vertex on the image and the connection between each vertex), and stores the information in a predetermined storage area (e.g., the storage device 1040 of the information processing device 10 in FIG. 2). Based on the information stored in this way, the correction necessity determination unit 120 can identify the positional relationship of the image areas corresponding to each of the plurality of objects (commodity) in the image.

[0040] Furthermore, the product identification result acquisition unit 110 generates image feature information for each of the detected multiple image regions as shown in Fig. 5, and identifies the product corresponding to each region using the image feature information for each of the multiple image regions. For example, the product identification result acquisition unit 110 can compare the image feature information for each of the multiple image regions with master information for each product, and identify the most similar product from the comparison result (degree of agreement with the master information). The product identification result acquisition unit 110 stores the result of the product identified using each image region in a predetermined storage area (e.g., storage device 1040 of the information processing device 10 in Fig. 2) in association with information indicating the position of each of the above-mentioned image regions.

[0041] Then, the correction necessity determination unit 120 uses the result of the processing by the product identification result acquisition unit 110 to compare the product identification results between two adjacent image regions, for example, starting from the left side of the image. Here, it is assumed that the information shown in FIG. 6 is obtained as a result of the processing by the product identification result acquisition unit 110. FIG. 6 is a diagram showing an example of the identification results of each of the multiple products shown in the image of FIG. 4. In the example of FIG. 6, only the product in the image region 50-2 is identified as "Drink B", and the products in the other image regions are all identified as "Drink A". In this case, the correction necessity determination unit 120 can first specify the pair of the image region 50-1 and the image region 50-2 as the processing target based on the positional relationship between the image region 50-1 and the image region 50-2 and the product identification results of each of these regions. Then, the correction necessity determination unit 120 calculates a first degree of coincidence indicating the degree of coincidence of the image regions (appearance features of the products) between the image region 50-1 and the image region 50-2.

[0042] If the first degree of coincidence is equal to or greater than the reference value, the correction necessity determination unit 120 recognizes that the product in the image area 50-1 and the product in the image area 50-2 are the same product, and determines that the product identification result needs to be corrected. In this case, the correction necessity determination unit 120 requests the product identification result correction unit 130 to execute a correction process. On the other hand, if the first degree of coincidence is less than the reference value, the correction necessity determination unit 120 recognizes that the product in the image area 50-1 and the product in the image area 50-2 are different products, and determines that the product identification result does not need to be corrected. Here, referring to FIG. 5, the products placed at positions corresponding to the image area 50-1 and the image area 50-2 are both the same product (drink A) in reality. Therefore, the correction necessity determination unit 120 can find a relatively large number of common points between the image area 50-1 and the image area 50-2 in terms of the image area (appearance features of the product). As a result, a first degree of match that is equal to or greater than a reference value is calculated for image area 50-1 and image area 50-2, and it is predicted that the correction necessity determination unit 120 will determine that the product identification result for image area 50-1 or the product identification result for image area 50-2 needs to be corrected.

[0043] The product identification result correction unit 130 corrects the product identification result for one of the two target image regions in response to a request from the correction necessity determination unit 120. As an example, the product identification result correction unit 130 can obtain the degree of match with the master information used to identify the product in each image region as the second degree of match, and determine the classification result with the higher second degree of match as the product identification result for the two image regions. For example, the product identification result correction unit 130 can obtain the degree of match (second degree of match) with the master information of the product "Drink A" for the image region 50-1 from the result of the product identification process by the product identification result acquisition unit 110. Similarly, the product identification result correction unit 130 can obtain the degree of match (second degree of match) with the master information of the product "Drink B" for the image region 50-2 from the result of the product identification process by the product identification result acquisition unit 110. Furthermore, the commodity identification result correction unit 130 may compare the image feature information obtained from these image regions with the master information of the commodities identified by the commodity identification result acquisition unit 110, and recalculate the degree of agreement (second degree of agreement). Then, if the second degree of agreement for the image region 50-1 is higher than the second degree of agreement for the image region 50-2, the commodity identification result acquisition unit 110 may recalculate the degree of agreement for the image region 50- 2 Product identification results (beverages B ) to the image area 50- 1 On the other hand, if the second degree of agreement for the image region 50-2 is higher than the second degree of agreement for the image region 50-1, the product identification result acquisition unit 110 corrects the identification result of the product in the image region 50-1 (drink A) to the same identification result as the identification result of the product in the image region 50-2 (drink B). Note that, as shown in FIG. 5, the product placed at the positions corresponding to the image region 50-1 and the image region 50-2 is actually "drink A". Therefore, the second degree of agreement for the image region 50-1 (degree of agreement with the master information of "drink A") is higher than the second degree of agreement for the image region 50-2 (degree of agreement with the master information of "drink B"), and it is predicted that the product identification result correction unit 130 will correct the identification result of the product in the image region 50-2 from "drink B" to "drink A".

[0044] As another example, the commodity identification result correction unit 130 may use information on a commodity in an image area 50-3 that is adjacent to the commodity in the image area 50-2 at a position different from that of the commodity in the image area 50-1 and has the same classification result as the image area 50-1. In this case, the commodity identification result correction unit 130 first calculates the degree of coincidence between the image area 50-2 and the image area 50-3 as a second degree of coincidence. If the second degree of coincidence is equal to or greater than a reference value, it can be said that the commodity in the image area 50-2 and the commodity in the image area 50-3 are highly likely to be the same commodity in reality. Here, from the result of the determination based on the first degree of coincidence by the correction necessity determination unit 120, it can be said that the commodity in the image area 50-1 and the commodity in the image area 50-2 are also highly likely to be the same commodity in reality. From these points, the commodity identification result correction unit 130 can determine that the same commodity is arranged side by side in the image areas 50-1 to 50-3. In this case, for example, since there are many identification results for the product "Drink A" in the area where identical products are arranged side by side, the product identification result correction unit 130 corrects the identification result for the product "Drink B" in the image area 50-2 to the identification result for the product "Drink A."

[0045] <Examples of effects> In this way, by comparing the image regions of two products displayed nearby and determining whether the products are the same, it is possible to improve the accuracy of product identification using images. Also, by narrowing down the target to the area where two products with different identification results are adjacent to each other, it is expected that the overall processing amount for the process of improving product identification accuracy can be reduced.

[0046] [Second embodiment] This embodiment has the same configuration as the first embodiment, except for the points described below.

[0047] <Functional configuration> 7 is a diagram illustrating an example of a functional configuration of an image analysis system according to the second embodiment. In this embodiment, the correction necessity determination unit 120 further includes a placement member detection unit 122. The placement member detection unit 122 acquires information indicating an image area of ​​a placement member (e.g., a shelf board of a product shelf) on which a product is placed.

[0048] Here, a product shelf generally has multiple shelves, and different types of products are often arranged on each shelf. Therefore, even if the identification results differ for two products that are adjacent to each other vertically across a shelf, there is a high possibility that the identification results for each product are correct. On the other hand, the same products may be stacked vertically on each shelf. In an area where the same products are stacked vertically in this way, if the identification results differ for two products that are adjacent to each other vertically, the identification results are likely to be incorrect.

[0049] The correction necessity determination unit 120 of this embodiment is configured to be able to recognize an area where products are stacked vertically (hereinafter, also referred to as a "stacked area"). Specifically, the correction necessity determination unit 120 identifies the position of the placement member in the image to be processed based on information acquired by the placement member detection unit 122. By identifying the position of the placement member in the image to be processed, the correction necessity determination unit 120 is able to determine the stacked area from the positional relationship between the image area corresponding to each product and the image area of ​​the placement member. Then, the correction necessity determination unit 120 preferentially checks the direction perpendicular to the placement member (the stacked direction of the products) and identifies the first product and the second product. When there are two products adjacent to each other without a placement member between them, the correction necessity determination unit 120 preferentially identifies the two products as the first product and the second product.

[0050] <Processing flow> Fig. 8 is a flowchart illustrating the flow of processing executed by the image analysis system 1 of the second embodiment. The following mainly describes the differences from the flowchart in Fig. 3.

[0051] Before identifying the first product and the second product to be processed, the correction necessity determination unit 120 identifies the position of the placement member in the image (S202). For example, the placement member detection unit 122 can detect the area of ​​the placement member from the image to be processed using a machine learning model capable of detecting the area of ​​the placement member (shelf board) of the product. Such a machine learning model is constructed by performing training using learning data to which information indicating the area of ​​the placement member of the product is previously given, and is stored in the storage device 1040 of the information processing device 10 in FIG. 2, or the like.

[0052] For example, assume that an image as shown in FIG. 9 is provided to the image analysis system 1 as a processing target. FIG. 9 is a diagram showing an example of an image provided to the image analysis system 1 as a processing target. The placement member detection unit 122 can obtain a result as shown in FIG. 10 by using, for example, the machine learning model described above. FIG. 10 is a diagram showing an example of a detection result of a placement member in the image of FIG. 9. The placement member detection unit 122 acquires position information of an image area surrounded by a dotted line in FIG. 10, and stores the position information in, for example, the memory 1030 or the storage device 1040 of the information processing device 10. The correction necessity determination unit 120 can identify the position of the placement member in the image based on the stored information.

[0053] The correction necessity determination unit 120 identifies a piled-up area in the image based on the position information of the image area of ​​the placement member acquired in the process of S202 and the position information of the image area of ​​each commodity acquired in the process of S104 (S204). The correction necessity determination unit 120 can detect an area (piled-up area) where multiple commodities are adjacent to each other in the vertical direction without a placement member between them, based on the vertical position information of each image area acquired by the commodity identification result acquisition unit 110 and the placement member detection unit 122.

[0054] For example, assume that the processing result as shown in FIG. 11 is obtained as a processing result by the commodity identification result acquisition unit 110 and the placement member detection unit 122. FIG. 11 is a diagram showing an example of the processing result by the commodity identification result acquisition unit 110 and the placement member detection unit 122. When the processing result as shown in FIG. 11 is obtained, only one commodity image area exists on the upper placement member in a direction perpendicular to the placement member. In this case, the correction necessity determination unit 120 can determine that the upper area of ​​the upper placement member is an area where no commodity is piled up (non-piled area). On the other hand, on the lower placement member, two commodity image areas exist adjacent to each other without a placement member in the direction perpendicular to the placement member. In this case, the correction necessity determination unit 120 can determine that the area between the upper placement member and the lower placement member is an area where multiple commodities are piled up (piled area).

[0055] Then, the correction necessity determination unit 120 identifies the first product and the second product as described in the first embodiment (S108). In the stacked area identified in S204, the correction necessity determination unit 120 checks the product identification results along the stacked product direction (vertical direction) and identifies the first product and the second product. In the example of FIG. 11, the product identification results are different for the two image areas indicated by diagonal lines. In this case, the correction necessity determination unit 120 identifies the two products corresponding to these image areas as the first product and the second product. On the other hand, in an area that is not a stacked area (non-stacked area), the correction necessity determination unit 120 identifies the first product and the second product based on the comparison result in the horizontal direction.

[0056] Then, the correction necessity determination unit 120 performs a correction process by comparing the image area of ​​the first product and the image area of ​​the second product. and the image area of Based on the degree of match, the product identification result correction unit 130 is requested to correct the product identification result (S110, S112). In response to the request from the correction necessity determination unit 120, the product identification result correction unit 130 corrects either the first product identification result or the second product identification result (S114). These processes are similar to the processes described in the first embodiment.

[0057] In this embodiment, an area (stacked area) where products are stacked and displayed is identified by detecting the product placement member. Then, in the stacked area, it is determined whether or not the product identification results differ along the direction in which the products are stacked. Here, essentially, identical products are stacked and displayed. Therefore, by comparing the product identification results along the product stacking direction in the stacked area, it is possible to more efficiently detect areas where products have been incorrectly identified.

[0058] [Third embodiment] <Functional configuration> 12 is a diagram illustrating a functional configuration of an image analysis system according to the third embodiment. The image analysis system 1 of this embodiment has the same configuration as the first or second embodiment, except that it does not have a commodity identification result correction unit 130. That is, the commodity identification result acquisition unit 110 and the correction necessity determination unit 120 of this embodiment have the functions described in the first or second embodiment.

[0059] <Hardware configuration> The image analysis system 1 of this embodiment can be realized by a hardware configuration (e.g., FIG. 2) similar to that of the first or second embodiment. For example, the storage device 1040 in FIG. 2 stores program modules that realize the functions of the image analysis system 1, including the commodity identification result acquisition unit 110 and the correction necessity determination unit 120. The processor 1020 in FIG. 2 reads each program module into the memory 1030 and executes it, thereby realizing the function corresponding to the read program module.

[0060] <Processing flow> Fig. 13 is a flowchart illustrating the flow of processing executed by the image analysis system 1 of the third embodiment. The processing of S302 to S312 in the flowchart of Fig. 13 is similar to the processing of S102 to S112 in Fig. 3. The following mainly describes processing that differs from the first and second embodiments. Although not shown, the image analysis system 1 of this embodiment may be configured to further execute the processing described in the second embodiment (e.g., the processing of S202 and S204 in the flowchart of Fig. 8).

[0061] As described in the first embodiment, when the classification results for two adjacent image regions are different, the correction necessity determination unit 120 specifies the two image regions as image regions corresponding to the first and second products to be processed (S308). Then, when the first degree of agreement calculated by comparing the two specified image regions is equal to or greater than a predetermined reference value (S312: YES), the correction necessity determination unit 120 determines that the classification result of the product needs to be corrected. In this case, the correction necessity determination unit 120 outputs information indicating that the classification result of the product needs to be corrected (S314). For example, the correction necessity determination unit 120 outputs information indicating that the classification result of the product needs to be corrected to a processing unit (not shown in this embodiment) corresponding to the classification result correction unit 130. As a result, as described in the other embodiments, the classification result of the product that may be erroneous is corrected. In addition, the correction necessity determination unit 120 may output information indicating that the classification result of the product needs to be corrected, for example, on the screen of the terminal 20 for the store clerk. For example, assume that an image as shown in Fig. 4 is given as an input image, and information as shown in Fig. 6 is obtained as a result of identifying a plurality of products shown in the image. In this case, the correction necessity determination unit 120 of this embodiment may be configured to output information as shown in Fig. 14.

[0062] FIG. 14 is a diagram showing an example of information output by the correction necessity determination unit 120 of the third embodiment. In the example of FIG. 14, the correction necessity determination unit 120 specifies two image areas (image area 50-1, image area 50-2) corresponding to the first product located at the leftmost position of the input image (drink A) and the second product located at the leftmost position of the input image (drink B) as processing targets because the identification result of these products is different. Here, if the degree of coincidence between the image area 50-1 and the image area 50-2 is equal to or higher than a reference, the correction necessity determination unit 120 changes the display mode of the image area and the identification result of the product related to the image area, as shown in the figure. Specifically, the correction necessity determination unit 120 highlights the frame or background indicating the image area, or highlights the identification result of the image area. In addition, the correction necessity determination unit 120 may output a message prompting the user to confirm the product identification results of the two image areas that are the processing targets.

[0063] The image analysis system 1 may further include a function of accepting input of information for correcting the commodity identification result from a user. For example, when an image as shown in FIG. 14 is displayed on the screen of the terminal 20 for the clerk, the clerk who uses the terminal 20 executes an input operation to select a target image area using an input device of the terminal 20. In response to the clerk executing the input operation to select an image area, the image analysis system 1 displays on the screen a form for inputting information for correcting the commodity identification result of the selected image area. Then, the clerk inputs correction information (correct commodity identification result) into the form displayed on the screen, thereby correcting the commodity identification result of the selected image area. For example, the clerk can determine from the image of FIG. 14 that the commodity identification result of the image area 50-2 is incorrect. In this case, the clerk can select the image area 50-2 as the target and execute an input to correct the commodity identification result to "drink A". As a result, the commodity identification result (drink B) corresponding to the image area 50-2 of FIG. 14 is corrected to the commodity identification result (drink A) input by the clerk.

[0064] <Examples of effects> In this embodiment, the image areas of two products displayed nearby are compared, and information indicating that the product identification result needs to be corrected is output according to the result of judging whether the products are the same or not. Such information can be used as a trigger for processing to improve the product identification accuracy, as described in the other embodiments above. In addition, since the target is narrowed down to the area where two products with different identification results are adjacent to each other, it is expected that the overall processing amount for processing to improve the product identification accuracy can be reduced.

[0065] <Other features> Furthermore, when the determination result by the correction necessity determination unit 120 is displayed on the screen of the store clerk's terminal and the store clerk performs an input operation to correct the product identification result, the image analysis system 1 may generate learning data that combines the image region selected as the target with the information input by the correction process (a label indicating correct information). By feeding back such learning data to the product identification model that generated the product identification result acquired by the product identification result acquisition unit 110, the identification accuracy of the product identification model can also be improved.

[0066] Although the embodiments of the present invention have been described above with reference to the drawings, the present invention should not be interpreted as being limited thereto, and various modifications, improvements, etc. can be made based on the knowledge of a person skilled in the art without departing from the gist of the present invention. In addition, the multiple components disclosed in the embodiments can be appropriately combined to form various inventions. For example, some components may be deleted from all the components shown in the embodiments, or components of different embodiments may be appropriately combined.

[0067] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that the content is not affected. In addition, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0068] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. A product identification result acquisition means for acquiring an identification result for each of a plurality of products appearing in an image; a correction necessity determination means for, when a classification result of a first product among the plurality of products differs from a classification result of a second product adjacent to the first product, calculating a first degree of coincidence which is a degree of coincidence between an image area of ​​the first product and an image area of ​​the second product, and determining whether or not the classification result of the first product or the classification result of the second product needs to be corrected based on the first degree of coincidence; An image analysis system comprising: 2. and a product identification result correction means for correcting one of the first product identification result and the second product identification result in response to a determination that the first product identification result or the second product identification result needs to be corrected based on the first degree of coincidence. 1. The image analysis system according to claim 1. 3. The commodity identification result correction means acquiring a second degree of match for each of the first product and the second product, the second degree of match indicating a degree of match with the master information of each of the first product and the second product; determining a classification result of the first product and the second product, whichever has a higher second degree of match, as a classification result of the first product and the second product; 2. The image analysis system according to claim 1. 4. The commodity identification result correction means if there is a third product that is adjacent to the first product at a position different from that of the second product and has the same classification result as the second product, calculate a second degree of match indicating a degree of match between the image area of ​​the first product and the image area of ​​the third product; If the second degree of match is equal to or greater than a reference level, correct the first product identification result. 2. The image analysis system according to claim 1. 5. The correction necessity determination means Acquire information indicating an image area of ​​a placement member on which the product is placed; Identifying two products adjacent to each other without the placement member in a direction perpendicular to the placement member as the first product and the second product; 5. An image analysis system according to any one of 1 to 4. 6. The computer Obtain the identification results for each of the multiple products in the image, If an identification result of a first product among the plurality of products differs from an identification result of a second product adjacent to the first product, a first degree of match is calculated, which is a degree of match between an image area of ​​the first product and an image area of ​​the second product; determining whether or not the identification result of the first product or the identification result of the second product needs to be corrected based on the first degree of match; The image analysis method includes: 7. The computer, correcting one of the first product identification result and the second product identification result in response to a determination that correction of the first product identification result or the second product identification result is necessary based on the first degree of match; The image analysis method according to 6., further comprising: 8. The computer, acquiring a second degree of match for each of the first product and the second product, the second degree of match indicating a degree of match with the master information of each of the first product and the second product; determining a classification result of the first product and the second product, whichever has a higher second degree of match, as a classification result of the first product and the second product; The image analysis method according to 7., further comprising: 9. The computer, if there is a third product that is adjacent to the first product at a position different from that of the second product and has the same classification result as the second product, calculate a second degree of match indicating a degree of match between the image area of ​​the first product and the image area of ​​the third product; If the second degree of match is equal to or greater than a reference level, correct the first product identification result. The image analysis method according to 7., further comprising: 10. The computer, Acquire information indicating an image area of ​​a placement member on which the product is placed; Identifying two products adjacent to each other without the placement member in a direction perpendicular to the placement member as the first product and the second product; 10. An image analysis method according to any one of 6. to 9. 11. A program for causing a computer to execute the image analysis method according to any one of 6. to 10. [Explanation of symbols]

[0069] 1. Image analysis system 10. Information processing device 1010 Bus 1020 Processor 1030 Memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface 110 Product identification result acquisition unit 120 Correction necessity determination section 122 Placed member detection unit 130 Product identification result correction department 20 Terminals

Claims

1. A product identification result acquisition means for acquiring an identification result for each of a plurality of products appearing in an image; a correction necessity determination means for calculating a first degree of coincidence, which is a degree of coincidence between an image area of ​​the first product and an image area of ​​the second product, when an identification result of a first product among the plurality of products differs from an identification result of a second product adjacent to the first product, and determining whether or not the identification result of the first product or the identification result of the second product needs to be corrected based on the first degree of coincidence; An image analysis system comprising:

2. and a product identification result correction means for correcting one of the first product identification result and the second product identification result in response to a determination that the first product identification result or the second product identification result needs to be corrected based on the first degree of coincidence. The image analysis system according to claim 1 .

3. The commodity identification result correction means acquiring a second degree of match for each of the first product and the second product, the second degree of match indicating a degree of match with master information of each of the first product and the second product; determining a classification result of the first product and the second product, whichever has a higher second degree of match, as a classification result of the first product and the second product; The image analysis system according to claim 2 .

4. The commodity identification result correction means if there is a third product that is adjacent to the first product at a position different from that of the second product and has the same classification result as the second product, calculate a second degree of match indicating a degree of match between an image area of ​​the first product and an image area of ​​the third product; If the second degree of match is equal to or greater than a reference level, correcting the identification result of the first product. The image analysis system according to claim 2 .

5. The correction necessity determination means Acquire information indicating an image area of ​​a placement member on which the product is placed; Two products that are adjacent to each other in a direction perpendicular to the placement member without the placement member in between are identified as the first product and the second product. The image analysis system according to any one of claims 1 to 4.

6. The computer Obtain the identification results for each of the multiple products in the image, If an identification result of a first product among the plurality of products differs from an identification result of a second product adjacent to the first product, a first degree of match is calculated, which is a degree of match between an image region of the first product and an image region of the second product; determining whether or not the identification result of the first product or the identification result of the second product needs to be corrected based on the first degree of coincidence; The image analysis method includes:

7. The computer, correcting one of the first product identification result and the second product identification result in response to a determination that correction of the first product identification result or the second product identification result is necessary based on the first degree of match; The image analysis method according to claim 6, further comprising:

8. The computer, acquiring a second degree of match for each of the first product and the second product, the second degree of match indicating a degree of match with master information of each of the first product and the second product; determining a classification result of the first product and the second product, whichever has a higher second degree of match, as a classification result of the first product and the second product; The image analysis method according to claim 7, further comprising:

9. The computer, Acquire information indicating an image area of ​​a placement member on which the product is placed; Two products that are adjacent to each other in a direction perpendicular to the placement member without the placement member in between are identified as the first product and the second product. The image analysis method according to any one of claims 6 to 8.

10. A program for causing a computer to execute the image analysis method according to any one of claims 6 to 9.

Images