Image processing device, image processing method, and program

Abstract

According to the present invention, an image processing unit (120) processes an image of a plurality of articles (for example, products) arranged on a shelf to specify an article name (for example, product name) of each of the plurality of articles shown in the image. Here, specifying the article name also includes specifying identification information associated with the article name. When the article whose article name could not be specified (hereinafter referred to as unspecified article) by the image processing unit (120) satisfies a specific condition, the article estimation unit (130) estimates the article name of the unspecified article as the article name of the article located next to the unspecified article.

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] The way in which items are arranged in a store affects the sales of the items. For this reason, the way in which items are arranged is often planned in advance. In this case, the person in charge of items checks that the arrangement of the items is as planned.

[0003] As a technology to assist this confirmation work, Patent Document 1 describes a method of recognizing displayed items by processing images of the displayed items. Patent Document 1 further describes that when an item is not recognized, items around the unrecognized item that have already been recognized are set as candidates for the unrecognized item. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2016 / 063484 Summary of the Invention [Problem to be solved by the invention]

[0005] The present inventor has considered how to estimate the name of an item with high accuracy when an item is not recognized by image processing. An object of the present invention is to estimate the name of an item with high accuracy when an item is not recognized by image processing. [Means for solving the problem]

[0006] According to the present invention, there is provided an image processing means for processing an image of a plurality of items arranged on a shelf to identify the names of the respective items shown in the image; an item estimation means for estimating, when a specific condition is satisfied, the name of an unidentified item, which is an item whose name could not be identified by the image processing means, to be the name of the item located adjacent to the unidentified item; An image processing apparatus is provided, comprising:

[0007] According to the present invention, a computer By processing an image of a plurality of items arranged on a shelf, the names of the respective items appearing in the image are identified; An image processing method is provided in which, for an unidentified item, the name of which cannot be identified, is estimated to be the name of an item located adjacent to the unidentified item when certain conditions are satisfied.

[0008] According to the present invention, a computer includes: A function of processing an image of a plurality of items arranged on a shelf to identify the names of the respective items shown in the image; A function of estimating the name of an unidentified item, which is an item whose name could not be identified, to be the name of an item located adjacent to the unidentified item when a specific condition is satisfied; A program will be provided to help students develop these skills. Effect of the Invention

[0009] According to the present invention, when an item is not recognized by image processing, the name of the item can be estimated with high accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing apparatus according to an embodiment. [Diagram 2] FIG. 4 illustrates an example of a data configuration of a result storage unit. [Diagram 3] FIG. 2 is a block diagram illustrating a hardware configuration of an image processing device. [Figure 4]11 is a flowchart illustrating an example of a process performed by the image processing device. [Diagram 5] 5 is a flowchart showing a first example of step S30 in FIG. 4. [Figure 6] FIG. 11 is a diagram showing a first example of a specific condition in step S120. [Figure 7] FIG. 11 is a diagram showing a second example of the specific condition in step S120. [Figure 8] FIG. 13 is a diagram showing a third example of a specific condition in step S120. [Figure 9] FIG. 11 is a diagram showing a fourth example of a specific condition in step S120. [Figure 10] 5 is a flowchart showing a second example of step S30 in FIG. 4. [Figure 11] FIG. 11 is a diagram showing an example of a screen displayed on the user terminal after step S132 in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are given the same reference numerals and the description will be omitted as appropriate.

[0012] FIG. 1 is a diagram showing an example of a functional configuration of an image processing device 10 according to an embodiment. The image processing device 10 includes an image processing unit 120 and an item estimation unit 130. The image processing unit 120 processes an image of a plurality of items (e.g., products) arranged on a shelf, thereby identifying the item names (e.g., product names) of the plurality of items shown in the image. Here, identifying the item names includes identifying identification information associated with the item names. When a specific condition is satisfied for an item (hereinafter, referred to as an unidentified item) whose name could not be identified by the image processing unit 120, the item estimation unit 130 estimates the name of the unidentified item to be the name of an item located next to the unidentified item. The image processing device 10 will be described in detail below.

[0013] The image processing device 10 includes an acquisition unit 110, a reference data storage unit 140, a result storage unit 142, and an output unit 150, in addition to the image processing unit 120 and the article estimation unit 130 described above.

[0014] The acquisition unit 110 acquires an image to be processed by the image processing unit 120, i.e., an image including multiple items, from the user terminal 20. The user terminal 20 is, for example, a portable terminal carried by a user of the image processing device 10, and has a photographing function. The user takes an image of a shelf in a store to generate an image. This image includes multiple items placed on the shelf. The user terminal 20 transmits the generated image to the image processing device 10. The user terminal 20 transmits, together with this image, information identifying the store in which the shelf that was the subject of the photograph is installed (hereinafter referred to as store identification information), and the conditions under which the image was generated (hereinafter referred to as generation conditions). The generation conditions include at least the date and time when the image was generated. The acquisition unit 110 also receives the store identification information and the generation conditions.

[0015] The image processing unit 120 processes the image transmitted by the user terminal 20 to identify the names of the items included in the image. For example, the image processing unit 120 identifies the regions of each of the multiple items included in the image, and performs feature matching for each of the multiple identified regions to identify the names of each of the multiple items. The reference data storage unit 140 stores the features for each item and the features for identifying the area of ​​the item used here. The image processing unit 120 then stores information including the identification results in the result storage unit 142.

[0016] The image processing unit 120 may identify the name of the article using the results of machine learning. In this case, the results of machine learning are stored in the reference data storage unit 140.

[0017] Furthermore, if the image processing unit 120 estimates the presence of an item but is unable to identify the name of the item (for example, if there is no item name with a score above a reference value), it outputs information indicating the area of ​​the item, i.e., the unidentified item, as well as the names and positions of items located around the unidentified item, to the item estimation unit 130.

[0018] As described above, when a specific condition is satisfied, the item estimation unit 130 estimates the name of an unidentified item to be the name of an item located next to the unidentified item. A specific example of the specific condition will be described later. The item estimation unit 130 stores the estimation result in the result storage unit 142.

[0019] The output unit 150 outputs the estimation result by the item estimation unit 130 to the user terminal 20. Here, when the output unit 150 receives information from the user terminal 20 indicating that the estimation result by the item estimation unit 130 is correct, the output unit 150 may store the estimation result by the item estimation unit 130, i.e., the name of the unidentified item, in the result storage unit 142. In this case, the item estimation unit 130 does not need to store the estimation result in the result storage unit 142.

[0020] Furthermore, the output unit 150 reads out information stored in the result storage unit 142 and outputs it to the user terminal 20, as necessary. Note that the output unit 150 may output the data stored in the result storage unit 142 to a terminal other than the user terminal 20.

[0021] 2 is a diagram showing an example of the data configuration of the result storage unit 142. The result storage unit 142 stores the image acquired by the acquisition unit 110, the conditions under which the image was generated (e.g., date and time), as well as the position of an item included in the image and the name of the item. Furthermore, when the name of the item has been estimated by the item estimation unit 130, the result storage unit 142 also stores information indicating this in association with the item.

[0022] If the item name of the unidentified item cannot be estimated by the item estimation unit 130, the result storage unit 142 stores the position of the unidentified item but does not store the name of the unidentified item.

[0023] 3 is a block diagram illustrating an example of a hardware configuration of the image processing device 10. The image processing device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0024] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0025] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0026] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0027] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the image processing device 10 (e.g., the acquisition unit 110, the image processing unit 120, the item estimation unit 130, and the output unit 150). The processor 1020 loads each of these program modules onto the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 also functions as various storage units (e.g., the reference data storage unit 140 and the result storage unit 142).

[0028] The input / output interface 1050 is an interface for connecting the image processing device 10 to various input / output devices.

[0029] The network interface 1060 is an interface for connecting the image processing device 10 to other devices (eg, the user terminal 20) on the network.

[0030] FIG. 4 is a flowchart showing an example of processing performed by the image processing device 10.

[0031] First, the user of the user terminal 20 photographs a shelf in a desired store (e.g., the store where the user works or supervises) and transmits the image generated by photographing to the image processing device 10 together with the store-specific information of the store and the image generation conditions (e.g., the generation date and time). This process may be performed each time an image is generated. Alternatively, the user of the user terminal 20 may photograph different locations on the shelf and transmit multiple images generated by photographing these locations together to the image processing device 10.

[0032] When the user terminal 20 transmits the image and the store identification information, the acquisition unit 110 of the image processing device 10 acquires the image, the image generation conditions, and the store identification information (step S10). The image processing unit 120 of the image processing device 10 then processes the image acquired by the acquisition unit 110 to identify the area of ​​the item (step S20). If the image contains multiple items, the image processing unit 120 identifies the areas of each of these multiple items. Next, the image processing unit 120 and the item estimation unit 130 process the identified area of ​​the item to identify the name of the item (step S30). Next, the image processing unit 120 and the item estimation unit 130 use the name of the item identified in step S30 as the Result storage unit 142 (step S40).

[0033] If multiple shelf label areas are identified in step S20, the process shown in step S30 is performed for each of the multiple shelf label areas. The process shown in step S20 may also be performed using the results of machine learning.

[0034] Fig. 5 is a flow chart showing a first example of step S30 in Fig. 4. The image processing unit 120 compares the feature amount stored in the result storage unit 142 with the area of ​​the item identified in step S20, and calculates a score for each item name (step S100). The process shown in step S100 may be performed using the results of machine learning. Then, the item name corresponding to the feature amount whose score is equal to or greater than a reference value and is the maximum is identified as the name of the item located in that area (step S110: Yes), and the identified item name is associated with information identifying the area and the image to be processed, and stored in the result storage unit 142 (step S140).

[0035] If there is no feature value whose score is equal to or greater than the reference value (step S110: No), the article estimation unit 130 determines whether the region satisfies a specific condition (step S120). Specific examples of the specific condition will be described later with reference to Figs. 6 to 9.

[0036] If the specific condition is met (step S120: Yes), the item estimation unit 130 estimates the name of the item in that area (unidentified item) to be the name of the item located next to the unidentified item (step S130). The item estimation unit 130 then associates the estimated item name with the information identifying the area and the image to be processed, and stores the associated item in the result storage unit 142 (step S140). At this time, the item estimation unit 130 also stores information indicating that the item name has been estimated by the item estimation unit 130 in the result storage unit 142.

[0037] If the specific condition is not satisfied (step S120: No), the product estimation unit 130 associates information identifying the region with the image to be processed and stores the information in the result storage unit 142 (step S140). In this case, the product name field is essentially left blank.

[0038] 6 is a diagram showing a first example of the specific condition in step S120. In the example shown in this figure, multiple items 40 are placed on the same shelf 30. The specific condition is that the items 40 on both sides of an unidentified item 40a have the same product name.

[0039] 7 is a diagram showing a second example of the specific condition in step S120. In the example shown in this figure, the specific condition is that the unidentified item 40a is located at the end of the shelf 30. In other words, the specific condition is that the distance between the unidentified item 40a and the side 32 of the product shelf is equal to or less than a reference value, and that no other items exist between the side 32 and the unidentified item 40a. The reason why the example shown in this figure is set as the specific condition is that the area of ​​the image corresponding to the unidentified item 40a may be blown out due to the side 32 reflecting light from a light source in the store or the side 32 having lighting embedded therein.

[0040] FIG. 8 is a diagram showing a third example of the specific condition in step S120. In the example shown in this figure, the specific condition is when there is a pattern in the arrangement of the names of multiple items 40, and when, based on this pattern, an unidentified item 40a is presumed to have the same name as an adjacent item 40.

[0041] FIG. 9 is a diagram showing a fourth example of the specific condition in step S120. In the example shown in this figure, the image includes a first shelf 30a and a shelf one level above the first shelf 30a (hereinafter, referred to as a second shelf 30b). The specific condition is that the difference in the horizontal positions of a first unidentified item 40a-1 placed on the first shelf 30a and a second unidentified item 40a-2 placed on the second shelf is equal to or less than a reference value, that is, the first unidentified item 40a-1 and the second unidentified item 40a-2 are placed at approximately the same position in the horizontal direction. As an example, the specific condition is that the unidentified items 40a-1 and 40a-2 are located at the ends of the shelves 30a and 30b, respectively. The reason why the example shown in this figure is set as the specific condition is the same as the example shown in FIG. 7.

[0042] Fig. 10 is a flowchart showing a second example of step S30 in Fig. 4. In the example shown in this figure, the processes from step S100 to step S130 and the process shown in step S140 are similar to the first example shown in Fig. 5.

[0043] When the item estimation unit 130 estimates the name of the unidentified item to be the name of the item located next to the unidentified item (step S130), the output unit 150 outputs the item name estimated by the item estimation unit 130 to the user terminal 20 together with an image of the unidentified item (step S132). Note that it is preferable that the item estimation unit 130 does not store the estimation result of the item name of the unidentified item 40a in the result storage unit 142.

[0044] The user terminal 20 displays the item name received from the output unit 150 (i.e., the item name estimated by the item estimation unit 130) together with an image of the unidentified item. When the user of the user terminal 20 confirms that the item name is correct, the user performs an input indicating that to the user terminal 20. The user terminal 20 then transmits information indicating that the item name is correct to the image processing device 10.

[0045] When the output unit 150 receives information from the user terminal 20 indicating that the estimation result by the item estimation unit 130 is correct (step S134), it stores the estimation result by the item estimation unit 130, i.e., the name of the unidentified item, in the result storage unit 142 (step S140).

[0046] If the product name received from the output unit 150 (i.e., the product name estimated by the product estimation unit 130) is incorrect, the user of the user terminal 20 preferably inputs the correct product name and transmits it to the image processing device 10. In this case, the output unit 150 may store the correct product name transmitted from the user terminal 20 in the result storage unit 142.

[0047] Fig. 11 is a diagram showing an example of a screen displayed on the user terminal 20 after step S132 in Fig. 10. In the example shown in this figure, the user terminal 20 displays the entire image processed by the image processing unit 120, or an image obtained by cutting out an area of ​​the image including the unidentified item 40a. This image includes information that can identify the unidentified item 40a (for example, a frame surrounding the unidentified item 40a). Furthermore, the user terminal 20 also displays the estimated name of the unidentified item 40a. This allows the user of the user terminal 20 to confirm the image of the unidentified item 40a and the estimated result of the name of the unidentified item 40a in association with each other.

[0048] As described above, according to this embodiment, the image processing unit 120 of the image processing device 10 identifies the names of items on the shelf by image processing. Then, when there is an unidentified item 40a that the image processing unit 120 has not been able to identify, the item estimation unit 130 estimates the name of the unidentified item 40a to be the name of the item 40 located next to the unidentified item 40a when a specific condition is satisfied. 。 As a result, if an item is not recognized by image processing, the name of that item can be estimated with high accuracy.

[0049] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0050] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that the content is not affected. In addition, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0051] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. An image processing means for processing an image of a plurality of items arranged on a shelf to identify each of the plurality of items appearing in the image; an item estimation means for estimating, when a specific condition is satisfied, the name of an unidentified item, which is an item whose name could not be identified by the image processing means, to be the name of the item located adjacent to the unidentified item; An image processing device comprising: 2. In the image processing device according to the above item 1, The image processing device, wherein the specified condition is that the items on both sides of the unidentified item have the same item name. 3. In the image processing device according to the above item 1, The image processing device, wherein the specific condition is that the unidentified item is located at the edge of the shelf. 4. In the image processing device according to the above item 1, The image shows a first shelf and a second shelf one level above the first shelf, The image processing device, wherein the specific condition is a case in which a difference in lateral position between a first unidentified item placed on the first shelf and a second unidentified item placed on the second shelf is equal to or smaller than a reference value. 5. In the image processing device according to the above item 1, The specific condition is that there is a pattern in the arrangement of the names of the plurality of items, and that, in that pattern, the unidentified item has the same name as the adjacent item. 6. The computer By processing an image of a plurality of items arranged on a shelf, the names of the respective items appearing in the image are identified; An image processing method in which, for an unidentified item, the name of which cannot be identified, is estimated to be the name of an item located next to the unidentified item when certain conditions are satisfied. 7. In the image processing method according to claim 6, The image processing method according to the present invention, wherein the specified condition is that the items on both sides of the unidentified item have the same name. 8. In the image processing method according to the above item 6, An image processing method, wherein the specific condition is that the unidentified item is located at the edge of the shelf. 9. In the image processing method according to claim 6, The image shows a first shelf and a second shelf one level above the first shelf, An image processing method, wherein the specific condition is a case in which a difference in lateral position between a first unidentified item placed on the first shelf and a second unidentified item placed on the second shelf is equal to or smaller than a reference value. 10. In the image processing method according to claim 6, The image processing method, wherein the specific condition is that there is a pattern in the arrangement of the names of the plurality of items, and that, in that pattern, the unidentified item has the same name as the adjacent item. 11. On the computer, A function of processing an image of a plurality of items arranged on a shelf to identify the names of the respective items shown in the image; A function of estimating the name of an unidentified item, which is an item whose name could not be identified, to be the name of an item located next to the unidentified item when a specific condition is satisfied; A program that allows you to have 12. In the program according to 11 above, The program wherein the specific condition is that the names of the items on either side of the unidentified item are the same. 13. In the program according to 11 above, The program, wherein the specific condition is when the unspecified item is located at the edge of the shelf. 14. In the program according to 11 above, The image shows a first shelf and a second shelf one level above the first shelf, The program, wherein the specific condition is a case in which a difference in lateral position between a first unidentified item placed on the first shelf and a second unidentified item placed on the second shelf is less than or equal to a reference value. 15. In the program according to 11 above, The program, wherein the specific condition is that there is a pattern in the arrangement of the names of the plurality of items, and that, in that pattern, the unidentified item has the same name as the adjacent item.

[0052] This application claims priority based on Japanese Patent Application No. 2019-167973, filed on September 17, 2019, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0053] 10 Image processing device 20 User terminal 30 Shelves 30a 1st shelf 30b Second Shelf 32 Side 40 Goods 40a Unspecified items 110 Acquisition Department 120 Image Processing Unit 130 Goods Estimation Department 140 Reference data storage unit 142 Result storage section 150 Output section

Claims

1. An image processing means for processing an image of a plurality of items arranged on a shelf to identify the names of the respective items appearing in the image; an item estimation means for estimating, when a specific condition is satisfied, the name of an unidentified item, which is an item whose name could not be identified by the image processing means, to be the name of the item located adjacent to the unidentified item; Equipped with The image processing device, wherein the specified condition is that the items on both sides of the unidentified item have the same item name.

2. An image processing means for processing an image of a plurality of items arranged on a shelf to identify the names of the respective items appearing in the image; an item estimation means for estimating, when a specific condition is satisfied, the name of an unidentified item, which is an item whose name could not be identified by the image processing means, to be the name of the item located adjacent to the unidentified item; Equipped with The image processing device, wherein the specific condition is that the unidentified item is located at the edge of the shelf.

3. An image processing means for processing an image of a plurality of items arranged on a shelf to identify the names of the respective items appearing in the image; an item estimation means for estimating, when a specific condition is satisfied, the name of an unidentified item, which is an item whose name could not be identified by the image processing means, to be the name of the item located adjacent to the unidentified item; Equipped with the specific condition is a case where there is a pattern in the number of consecutive articles with the same name obtained based on the identification result of the names of the plurality of articles, The item estimation means is an image processing device that estimates the name of the unidentified item using the name of one of two items located on either side of the unidentified item that maintains the regularity.

4. The computer By processing an image of a plurality of items arranged on a shelf, the names of the respective items appearing in the image are identified; When a specific condition is satisfied, the name of the unspecified item, which is the item whose name could not be identified, is estimated to be the name of the item located next to the unspecified item, The image processing method according to the present invention, wherein the specified condition is that the items on both sides of the unidentified item have the same name.

5. A computer comprising: By processing an image of a plurality of items arranged on a shelf, the names of the respective items appearing in the image are identified; When a specific condition is satisfied, the name of the unspecified item, which is the item whose name could not be identified, is estimated to be the name of the item located next to the unspecified item, An image processing method, wherein the specific condition is that the unidentified item is located at the edge of the shelf.

6. A computer comprising: By processing an image of a plurality of items arranged on a shelf, the names of the respective items appearing in the image are identified; When a specific condition is satisfied, the name of the unspecified item, which is the item whose name could not be identified, is estimated to be the name of the item located next to the unspecified item, the specific condition is a case where there is a pattern in the number of consecutive articles with the same name obtained based on the identification result of the names of the plurality of articles, The computer estimates the name of the unidentified item using the name of one of two items located on either side of the unidentified item that maintains the regularity. Image processing methods.

7. On the computer, A function of processing an image of a plurality of items arranged on a shelf to identify the names of the respective items shown in the image; A function of estimating the name of an unidentified item, which is an item whose name could not be identified, to be the name of an item located next to the unidentified item when a specific condition is satisfied; Let them have The program wherein the specific condition is that the names of the items on either side of the unidentified item are the same.

8. A computer comprising: A function of processing an image of a plurality of items arranged on a shelf to identify the names of the respective items shown in the image; A function of estimating the name of an unidentified item, which is an item whose name could not be identified, to be the name of an item located next to the unidentified item when a specific condition is satisfied; Let them have The program, wherein the specific condition is when the unspecified item is located at the edge of the shelf.

9. A computer comprising: A function of processing an image of a plurality of items arranged on a shelf to identify the names of the respective items shown in the image; A function of estimating the name of an unidentified item, which is an item whose name could not be identified, to be the name of an item located next to the unidentified item when a specific condition is satisfied; Let them have the specific condition is a case where there is a pattern in the number of consecutive articles with the same name obtained based on the identification result of the names of the plurality of articles, The function of estimating the name of the item estimates the name of the unidentified item using the name of one of two items located on both sides of the unidentified item that maintains the regularity. program.

Images