Detection device, detection method, and program

Abstract

A detection unit (11) in a detection device (10) applies a first model to a captured image so as to detect a first image region of the captured image. A detection unit (12) applies a second model to the captured image so as to detect a second image region of the captured image. The second model identifies the second image region, which corresponds to an image of an article group disposed at the front side of a target image. The first model identifies the first image region, which is based on an article main occupation space in a target image. An index calculation unit (14) calculates an index pertaining to a deviation of the first image region and the second image region from each other in each shelf image region. A determination unit (15) determines a detection result to be employed for each shelf image region from the first image region and the second image region in each shelf image region on the basis of the index that has been calculated for each shelf image region.

Description

DETECTION APPARATUS, DETECTION METHOD, AND NON-TRANSITORY COMPUTER-READABLE MEDIUM

[0001] The present disclosure relates to a detection apparatus, a detection method, and a non-transitory computer-readable medium.

[0002] A technology has been proposed that uses a single trained discrimination model to detect an area in an image of a group of items such as merchandise where a group of items is continuously present (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-117531

[0004] The present inventors have found that a single discrimination model may not be able to achieve sufficient accuracy in detecting an image region. That is, a discrimination model typically has targets that it is good at detecting and targets that it is not good at detecting. Therefore, the present inventors have found that the accuracy in detecting an image region can be improved by applying a first model and a second model with different characteristics to an image of a group of items.

[0005] One objective of the present disclosure is to provide a detection device, a detection method, and a non-transitory computer-readable medium that can improve the detection accuracy of an image region. It should be noted that this objective is only one of multiple objectives that multiple embodiments disclosed in this specification aim to achieve. Other objectives or problems and novel features will become apparent from the description of this specification or the accompanying drawings.

[0006] In one aspect, the detection device comprises: a first detection unit that detects a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; a second detection unit that detects a second image area in the captured image using a second model that identifies a second image area that corresponds to an image of a group of items arranged at the front in the target image; an identification unit that identifies a plurality of shelf level image areas in the captured image that respectively correspond to a plurality of shelf levels of the item shelf; an index calculation unit that calculates an index related to the deviation between the first image area and the second image area in each shelf level image area; and a determination unit that determines a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area.

[0007] In another aspect, the detection method includes: detecting a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; detecting a second image area in the captured image using a second model that identifies a second image area that corresponds to an image of a group of items arranged at the front in the target image; identifying a plurality of shelf level image areas in the captured image that respectively correspond to a plurality of shelf levels of the item shelf; calculating an index related to the deviation between the first image area and the second image area in each shelf level image area; and determining a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area.

[0008] In another aspect, a non-transitory computer-readable medium stores a program that causes a detection device to execute processes including: detecting a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; detecting a second image area in the captured image using a second model that identifies a second image area that corresponds to an image of a group of items arranged in the front in the target image; identifying a plurality of shelf level image areas in the captured image that respectively correspond to a plurality of shelf levels of the item shelf; calculating an index related to the deviation between the first image area and the second image area for each shelf level image area; and determining a detection result to be adopted for each shelf level image area from the first image area and the second image area for each shelf level image area based on the index calculated for each shelf level image area.

[0009] The present disclosure provides a detection device, a detection method, and a non-transitory computer-readable medium that can improve the accuracy of detecting an image region.

[0010] FIG. 1 is a block diagram showing an example of a detection device in a first embodiment; FIG. 2 is a flowchart showing an example of a processing operation of the detection device in the first embodiment; FIG. 3 is a block diagram showing an example of a detection device in a second embodiment; FIG. 4 is a diagram showing an example of an item shelf image; FIG. 5 is a diagram showing an example of a second image area; FIG. 6 is a diagram showing an example of a first image area; FIG. 7 is a block diagram showing an example of a detection device in a third embodiment; FIG. 8 is a diagram showing an example of an integrated image; and FIG. 9 is a diagram showing an example of the hardware configuration of the detection device.

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements are denoted by the same reference numerals, and redundant description will be omitted.

[0012] First Embodiment Overview of the Detection Device The detection device 10 in the first embodiment detects an image area corresponding to an image of a group of items in a captured image, for example, by using a first model and a second model that have different detection characteristics. The captured image is, for example, an image of an item shelf (hereinafter, sometimes referred to as an item shelf image).

[0013] The above-mentioned "second model" is a model that identifies an image area (hereinafter sometimes referred to as a "second image area" or a "second item group image area") that corresponds to an image of an item group placed in the foreground in the target image. Note that hereinafter, the "item group image" may be referred to as an "item group image."

[0014] For example, the second model may be a trained model that has been trained using training data including the following images: - An image of one entire side of a single item. - An item shelf image in which an image area corresponding to an item group image of an item group arranged in the front row of each shelf in the item shelf image is designated as an "item group image area."

[0015] In addition, for each item in the group of items arranged in the front row of a shelf, most of one side of the item (for example, more than half of one side) is usually shown in the item shelf image.

[0016] By learning using such learning data, the second model can accurately detect image areas corresponding to the group of products located at the front of each shelf, but may not be able to accurately detect image areas corresponding to the group of products located behind the group of products located at the front and with most of one side hidden.

[0017] Furthermore, the above-mentioned "first model" is a model that identifies an image area (hereinafter, sometimes referred to as the "first image area" or "first item group image area") based on the space that is primarily occupied by the item group in the target image (hereinafter, sometimes referred to as the "item main occupied space").

[0018] For example, the first model may be a trained model that has been trained using training data including the following images: - An image capturing the entirety of one side of a single item. - An item shelf image in which an image area corresponding to an item group image of a group of items lined up in the front row of each shelf in the item shelf image is designated as an "item group image area." - An item shelf image in which an image area corresponding to an "image equivalent to the true background (hereinafter sometimes referred to as a "true background image")" in the item shelf image is designated as a "true background image area."

[0019] Here, the "true background image area" includes an image area corresponding to the image of the back panel, side panel, or shelf of the goods shelf that is shown in the goods shelf image without being hidden by the shadow of the goods.

[0020] By learning using such training data, the first model has the characteristic of being able to accurately detect image areas that are likely to be background images (e.g., image areas corresponding to large empty spaces where no items are placed) and image areas corresponding to the above-mentioned space primarily occupied by items. However, because the first model learns contradictory information, namely, "item group image areas" and "true background image areas," for areas that cannot be designated as either of these, there is a possibility that the model will detect areas close to "item group image areas" as "item group image areas" and areas close to "true background image areas" as "true background image areas." As a result, there is a possibility that the model will detect an image area corresponding to a narrow empty space sandwiched between two "item group image areas" as part of the item group image area. This is because it is believed that in many cases, images corresponding to narrow empty spaces do not provide enough information to detect that the image is a background image.

[0021] Unlike the learning data of the first model, the learning data of the second model does not include any shelf images for which the "true background image region" is specified, or if it does include any, the number of such images is small.

[0022] As described above, the detection device 10 detects an image area corresponding to an image of a group of items in a captured image using the "first model" and the "second model" that have different detection characteristics, thereby improving the detection accuracy of the image area.

[0023] <Configuration example of detection device> Fig. 1 is a block diagram showing an example of a detection device in the first embodiment. In Fig. 1, the detection device 10 has a detection unit (first detection unit) 11, a detection unit (second detection unit) 12, an identification unit 13, an index calculation unit 14, and a determination unit 15. The detection device 10 acquires a captured image. This captured image is, for example, a shelf image of a shelf. The following description will be given on the assumption that the captured image is a shelf image. This shelf has multiple shelves.

[0024] The detection unit (first detection unit) 11 detects a "first image region" in the captured image by applying a first model to the captured image.

[0025] The detection unit (second detection unit) 12 detects a "second image area" in the captured image by applying a second model to the captured image.

[0026] The identification unit 13 identifies a plurality of "shelf level image areas" corresponding to the plurality of shelf levels in the captured image. A shelf level image area corresponding to one shelf level is, for example, an image area corresponding to the space between the shelf board of the shelf level and the shelf board of the shelf level immediately above the one shelf level.

[0027] The index calculation unit 14 calculates an "index related to the deviation" between the first image area and the second image area in each shelf image area. A specific example of the "index related to the deviation" will be described in the second embodiment.

[0028] The determination unit 15 determines the detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the above-mentioned indicators calculated for each shelf level image area.

[0029] <Example of Operation of Detection Device> A description will be given of an example of processing operations of the detection device 10 having the above configuration. Fig. 2 is a flowchart showing an example of processing operations of the detection device in the first embodiment.

[0030] The detection unit 11 detects a first image region in the captured image by applying a first model to the captured image (step S101).

[0031] The detection unit 12 detects a second image region in the captured image by applying the second model to the captured image (step S102).

[0032] The identifying unit 13 identifies a plurality of shelf level image areas corresponding to the plurality of shelf levels in the captured image (step S103).

[0033] The index calculation unit 14 calculates an index related to the deviation between the first image area and the second image area in each shelf image area (step S104).

[0034] The determination unit 15 determines the detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area (step S105).

[0035] As described above, according to the first embodiment, the detection device 10 detects an item group image area in a captured image using a "first model" and a "second model" that have different detection characteristics. This allows one model to compensate for the weaknesses of the other model in detecting objects, thereby improving the detection accuracy of the item group image area.

[0036] Furthermore, the index calculation unit 14 in the detection device 10 calculates an "index related to the deviation" between the first image area and the second image area in each shelf level image area. The determination unit 15 determines the detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area.

[0037] This configuration of the detection device 10 can improve the detection accuracy of the item group image area. That is, if the above-mentioned indicator indicates a large deviation between the first image area and the second image area in a shelf level image area, it is highly likely that items located further back are also captured in the shelf level image area. When the second model is applied to such a shelf level image area, it is possible that the image area corresponding to items located further back cannot be accurately detected. On the other hand, when the first model is applied to such a shelf level image area, it is highly likely that the image area corresponding to items located further back can be accurately detected. Therefore, in such cases, the detection device 10 can use the first image area obtained by applying the first model to the shelf level image area as the detection result. On the other hand, when the above-mentioned indicator indicates a small deviation between the first image area and the second image area in a shelf level image area, it is highly likely that few items located further back are captured in the shelf level image area. When the second model is applied to such a shelf level image area, it is highly likely that the image area corresponding to the product group located in the front can be accurately detected. Therefore, in such a case, the detection device 10 can use the second image area obtained by applying the second model to the one shelf image area as the detection result. Therefore, one model can compensate for the weak detection of the detection target of the other model, thereby improving the detection accuracy of the item group image area.

[0038] Second Embodiment The second embodiment relates to an embodiment that further embodies the contents of the first embodiment.

[0039] Fig. 3 is a block diagram showing an example of a detection device according to the second embodiment. In Fig. 3, the detection device 20 includes a detection unit (first detection unit) 21, a detection unit (second detection unit) 22, an identification unit 23, an index calculation unit 24, and a determination unit 25. The detection device 20 acquires a captured image. This captured image is, for example, a shelf image of a shelf. The following description will be given on the assumption that the captured image is a shelf image. This shelf has multiple shelves.

[0040] The detector 22, like the detector 12 of the first embodiment, detects a second image area in the captured image (i.e., the item shelf image) by applying a second model to the captured image.

[0041] Fig. 4A is a diagram showing an example of an item shelf image, and Fig. 4B is a diagram showing an example of a second image area.

[0042] Figure 4A shows an image of a shelf where bottled beverages are displayed. The shelf shown in the image of Figure 4A has four shelves. On the top three shelves, the bottles are arranged upright. On the bottom shelf, the bottles are arranged horizontally and stacked. On the top three shelves, the space above the bottles is narrow, while on the bottom shelf, the space above the bottles is wide. Therefore, in the image of the top three shelves, most of the bottles at the back are hidden by the bottles in front, while in the image of the bottom shelf, even the bottles at the back are visible.

[0043] The result of applying the second model to the shelf image of Figure 4A is shown in Figure 4B. In Figure 4B, the shaded area corresponds to the second image area.

[0044] 4B, the second model can accurately detect the item group image area corresponding to the plastic bottles in the image of the top three shelves. As a result, the second model can accurately detect even an image area corresponding to a narrow empty space sandwiched between two item group image areas.

[0045] On the other hand, as can be seen from Figure 4B, in the image of the bottom shelf, the item group image area corresponding to the group of PET bottles located at the front can be detected, but the item group image area corresponding to the group of PET bottles located at the back cannot be detected.

[0046] The detection unit 21, like the detection unit 11 of the first embodiment, detects a "first image region" in the captured image by applying a first model to the captured image.

[0047] 4C is a diagram showing an example of the first image area. In FIG. 4C, the shaded area in the shelf image area SA14 corresponds to the first image area. As can be seen from FIG. 4C, model 1 can accurately detect not only the item group image area corresponding to the group of plastic bottles located at the front in the shelf image area SA14 (i.e., the image of the bottom shelf), but also the item group image area corresponding to the group of plastic bottles located at the back. Note that, as can be seen from FIGS. 4B and 4C, model 1 may detect an image area corresponding to a narrow empty space SP1 sandwiched between two item group image areas as part of the item group image area.

[0048] Returning to the explanation of FIG. 3, the identifying unit 23, like the identifying unit 13 of the first embodiment, identifies a plurality of "shelf level image areas" corresponding to the plurality of shelf levels in the captured image.

[0049] For example, as can be seen in Figure 4B, the lower line defining the second image area in the image of each shelf appears as a straight line that is approximately parallel to the shelf. In other words, by identifying this straight line, it is possible to identify the line that corresponds to the surface of the plastic bottle that is in contact with the shelf.

[0050] Therefore, the identification unit 23 may identify the lower line that defines the second image area in the image of each shelf level, and identify the image area sandwiched between two adjacent lines as the ``shelf level image area.''

[0051] Alternatively, for example, the identification unit 23 may directly identify the front image of the shelf by pattern matching or the like. This front image of the shelf can also be identified as a line corresponding to the surface of the plastic bottle in contact with the shelf. The identification unit 23 may then identify the image area sandwiched between two adjacent lines as a "shelf level image area." Note that image areas SA11, SA12, SA13, and SA14, each surrounded by a frame in FIG. 4B, are examples of shelf level image areas.

[0052] The index calculation unit 24, like the index calculation unit 14 of the first embodiment, calculates an "index related to the deviation" between the first image area and the second image area in each shelf image area.

[0053] For example, the index calculation unit 24 may calculate the ratio of the area of ​​the first image area to the area of ​​the second image area in each shelf level image area as the "deviation-related index." For example, in the examples of FIGS. 4B and 4C , for shelf level image areas SA11, SA12, and SA13, the ratio of the area of ​​the first image area to the area of ​​the second image area is close to 1. This is because the areas of the second image area and the first image area are approximately equal in shelf level image areas SA11, SA12, and SA13. On the other hand, for shelf level image area SA14, the ratio of the area of ​​the first image area to the area of ​​the second image area is greater than the ratios for shelf level image areas SA11, SA12, and SA13. This is because the difference between the area of ​​the second image area and the area of ​​the first image area is large in shelf level image area SA14.

[0054] Similar to the determination unit 15 of the first embodiment, the determination unit 25 determines the detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the above-mentioned indicators calculated for each shelf level image area.

[0055] For example, the determiner 25 determines the detection result to be adopted for each shelf level image area based on the ratio of the area of ​​the first image area to the area of ​​the second image area in each shelf level image area. Specifically, for shelf level image areas where the ratio is equal to or greater than a threshold, the determiner 25 determines that the first image area is to be adopted as the detection result. That is, for shelf level image area SA14, the determiner 25 determines that the first image area is to be adopted as the detection result. On the other hand, for shelf level image areas where the ratio is less than the threshold, the determiner 25 determines that the second image area is to be adopted as the detection result. That is, for shelf level image areas SA11, SA12, and SA13, the determiner 25 determines that the second image area is to be adopted as the detection result.

[0056] Third Embodiment The third embodiment relates to identifying free space.

[0057] 5 is a block diagram showing an example of a detection device according to the third embodiment. In FIG. 5, the detection device 30 includes a detection unit (first detection unit) 11, a detection unit (second detection unit) 12, an identification unit 13, an index calculation unit 14, a determination unit 15, an integration unit 31, and a space identification unit 32.

[0058] The integration unit 31 integrates the detection results used for each shelf level image area to obtain an "integrated image." For example, in the case of Figures 4B and 4C above, the second image areas in shelf level image areas SA11, SA12, and SA13 detected by the second model and the first image area in shelf level image area SA14 detected by the first model are integrated to form an "integrated image." Figure 6 is a diagram showing an example of an integrated image.

[0059] The space identifying unit 32 identifies empty spaces on each shelf level where no items are placed, based on the integrated image. For example, the space identifying unit 32 may identify empty spaces by subtracting the integrated image from the shelf level image area. In Figure 6, for example, the areas surrounded by rectangular frames (spaces SP1, SP2, SP3, and SP4) correspond to empty spaces.

[0060] As described above, the detection device 30 in the third embodiment identifies vacant space based on an integrated image obtained by applying the first model and the second model to the shelf image areas, which are the detection targets that the first model and the second model are each good at detecting, and thereby can identify vacant space with high accuracy.

[0061] Although the description herein has been given on the assumption that the integrating unit 31 and the space identifying unit 32 are applied to the detection device 10 of the first embodiment, the present disclosure is not limited to this. The integrating unit 31 and the space identifying unit 32 may also be applied to the detection device 20 of the second embodiment.

[0062] <Other Embodiments> Fig. 7 is a diagram illustrating an example of the hardware configuration of a detection device. In Fig. 7, the detection device 100 includes a processor 101 and a memory 102. The processor 101 may be, for example, a microprocessor, a microprocessing unit (MPU), or a central processing unit (CPU). The processor 101 may include multiple processors. The memory 102 is configured by a combination of volatile memory and non-volatile memory. The memory 102 may include storage located remotely from the processor 101. In this case, the processor 101 may access the memory 102 via an I / O interface (not shown).

[0063] The detection devices 10, 20, and 30 of the first to third embodiments may each have the hardware configuration shown in FIG. 7 . The detection units 11 and 21, the detection units 12 and 22, the identification units 13 and 23, the index calculation units 14 and 24, the determination units 15 and 25, the integration unit 31, and the space identification unit 32 of the detection devices 10, 20, and 30 of the first to third embodiments may be implemented by the processor 101 reading and executing a program stored in the memory 102. The program can be stored using various types of non-transitory computer-readable media and supplied to the detection devices 10, 20, and 30. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, and hard disk drives) and magneto-optical recording media (e.g., magneto-optical disks). Further examples of non-transitory computer-readable media include CD-ROMs (Read Only Memory), CD-Rs, and CD-R / Ws. Further, examples of non-transitory computer-readable media include semiconductor memory. Semiconductor memory includes, for example, mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash ROM, and random access memory (RAM). The program may also be provided to the detection devices 10, 20, and 30 by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media may provide the program to the detection devices 10, 20, and 30 via wired communication paths such as electrical wires and optical fibers, or wireless communication paths.

[0064] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0065] REFERENCE SIGNS LIST 10 Detection device 11 Detection unit (first detection unit) 12 Detection unit (second detection unit) 13 Identification unit 14 Index calculation unit 15 Determination unit 20 Detection device 21 Detection unit (first detection unit) 22 Detection unit (second detection unit) 23 Identification unit 24 Index calculation unit 25 Determination unit 30 Detection device 31 Integration unit 32 Space identification unit

Claims

1. a first detection unit that detects a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; a second detection unit that detects the second image area in the captured image using a second model that identifies a second image area corresponding to an image of a group of items arranged in front in the target image; an identification unit that identifies a plurality of shelf level image areas in the captured image that correspond to a plurality of shelf levels of the item shelf; an index calculation unit that calculates an index related to a deviation between the first image area and the second image area in each shelf image area; a determination unit that determines a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; A detection device comprising:

2. the index calculation unit calculates, as the index, a ratio of an area of ​​the first image area to an area of ​​the second image area in each shelf image area; the determination unit determines a detection result to be adopted for each shelf level image area based on the ratio. The detection device according to claim 1 .

3. The determination unit For a shelf image area where the ratio is equal to or greater than a threshold, the first image area is determined to be the detection result to be adopted, For a shelf image area where the ratio is less than the threshold, the second image area is determined to be the detection result to be adopted. The detection device according to claim 2.

4. the identification unit identifies a plurality of lines corresponding to surfaces of items in contact with shelves of the item shelf in the second image area, and identifies the plurality of shelf level image areas by dividing the captured image by the plurality of lines. The detection device according to claim 1 .

5. and further comprising an integration unit that integrates the detection results employed for each shelf image region to obtain an integrated image. The detection device according to claim 1 .

6. a space identifying unit that identifies an empty space on each shelf where no article is placed based on the integrated image; The detection device according to claim 5.

7. Detecting a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; Detecting a second image region in the captured image using a second model that identifies a second image region corresponding to an image of a group of items located in front of the target image; Identifying a plurality of shelf level image areas in the captured image, each of which corresponds to a plurality of shelf levels of the item shelf; Calculating an index related to a deviation between the first image area and the second image area in each shelf image area; determining a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; A detection method comprising:

8. The calculating step includes calculating, as the index, a ratio of an area of ​​the first image area to an area of ​​the second image area in each shelf image area, The determining step includes determining a detection result to be adopted for each shelf image region based on the ratio. The detection method according to claim 7.

9. Detecting a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; Detecting a second image region in the captured image using a second model that identifies a second image region corresponding to an image of a group of items located in front of the target image; Identifying a plurality of shelf level image areas in the captured image, each of which corresponds to a plurality of shelf levels of the item shelf; Calculating an index related to a deviation between the first image area and the second image area in each shelf image area; determining a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; A program that causes a detection device to execute a process including the steps of:

10. The calculating step includes calculating, as the index, a ratio of an area of ​​the first image area to an area of ​​the second image area in each shelf image area, The determining step includes determining a detection result to be adopted for each shelf image region based on the ratio. The program according to claim 9.