Detection device, detection method, and program

Abstract

A detection unit (11) in a detection device (10) applies a first model to a captured image so as to detect a first image region of the captured image. A detection unit (12) applies a second model to the captured image so as to detect a second image region of the captured image. The second model identifies the second image region, which corresponds to an image of an article group disposed at the front side of a target image. The first model identifies the first image region, which is based on an article main occupation space in a target image. An index calculation unit (14) calculates an index pertaining to a deviation of the first image region and the second image region from each other in each shelf image region. A determination unit (15) determines a detection result to be employed for each shelf image region from the first image region and the second image region in each shelf image region on the basis of the index that has been calculated for each shelf image region.

Description

[Technical Field]

[0001] The present disclosure relates to a detection apparatus, a detection method, and a non-transitory computer-readable medium. [Background technology]

[0002] A technology has been proposed that uses one trained discrimination model to detect an area in an image of a group of items such as merchandise where a group of items is continuously present (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-117531 Summary of the Invention [Problem to be solved by the invention]

[0004] The inventors have found that a single discrimination model may not be able to achieve sufficient accuracy in detecting an image region. That is, a discrimination model usually has targets that it is good at detecting and targets that it is not good at detecting. For this reason, the inventors have found that the accuracy in detecting an image region can be improved by applying a first model and a second model with different characteristics to an image of a group of items.

[0005] One objective of the present disclosure is to provide a detection device, a detection method, and a non-transitory computer-readable medium that can improve the accuracy of detecting an image region. It should be noted that this objective is only one of multiple objectives that multiple embodiments disclosed in this specification aim to achieve. Other objectives or problems and novel features will become apparent from the description of this specification or the accompanying drawings. [Means for solving the problem]

[0006] In one embodiment, the sensing device comprises: a first detection unit that detects a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; a second detection unit that detects the second image area in the captured image using a second model that identifies a second image area corresponding to an image of a group of items arranged in front in the target image; an identification unit that identifies a plurality of shelf level image areas in the captured image that correspond to a plurality of shelf levels of the item shelf; an index calculation unit that calculates an index related to a deviation between the first image area and the second image area in each shelf image area; a determination unit that determines a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; It is equipped with:

[0007] In another aspect, the detection method comprises: Detecting a first image area in a captured image of an item shelf using a first model that identifies a first image area based on an item main occupied space that is mainly occupied by an item group in the target image; Detecting a second image region in the captured image using a second model that identifies a second image region corresponding to an image of a group of items located in front of the target image; Identifying a plurality of shelf level image areas in the captured image, each of which corresponds to a plurality of shelf levels of the item shelf; Calculating an index related to a deviation between the first image area and the second image area in each shelf image area; determining a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; Includes:

[0008] In another aspect, the non-transitory computer readable medium comprises: Detecting a first image area in a captured image of an item shelf using a first model that identifies a first image area based on an item main occupied space that is mainly occupied by an item group in the target image; Detecting a second image region in the captured image using a second model that identifies a second image region corresponding to an image of a group of items located in front of the target image; Identifying a plurality of shelf level image areas in the captured image, each of which corresponds to a plurality of shelf levels of the item shelf; Calculating an index related to a deviation between the first image area and the second image area in each shelf image area; determining a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; The program for causing the detection device to execute the process including the above is stored. [Effects of the Invention]

[0009] The present disclosure provides a detection device, a detection method, and a non-transitory computer-readable medium that can improve the accuracy of detecting an image region. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating an example of a detection device according to a first embodiment. [Figure 2] 4 is a flowchart showing an example of a processing operation of the detection device in the first embodiment. [Figure 3] FIG. 10 is a block diagram illustrating an example of a detection device according to a second embodiment. [Figure 4A] FIG. 10 is a diagram showing an example of an item shelf image. [Figure 4B] FIG. 10 is a diagram illustrating an example of a second image area. [Figure 4C] FIG. 2 is a diagram showing an example of a first image area. [Figure 5] FIG. 10 is a block diagram illustrating an example of a detection device according to a third embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of an integrated image. [Figure 7] FIG. 2 illustrates an example of a hardware configuration of a detection device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements are denoted by the same reference numerals, and redundant description will be omitted.

[0012] First Embodiment <Detection device overview> The detection device 10 in the first embodiment detects an image area corresponding to an image of a group of items in a captured image, for example, by using a first model and a second model that have different detection characteristics. The captured image is, for example, an image of an item shelf (hereinafter, sometimes referred to as an item shelf image).

[0013] The above-mentioned "second model" is a model that identifies an image area (hereinafter sometimes referred to as a "second image area" or "second item group image area") that corresponds to an image of an item group placed in the foreground in the target image. Note that hereinafter, the "item group image" may be referred to as an "item group image."

[0014] For example, the second model may be a trained model trained using training data including the following images: - An image showing the entire side of a single item. - An item shelf image in which an image area corresponding to an item group image of an item group arranged in the front row of each shelf in the item shelf image is designated as an "item group image area."

[0015] Note that, for each item in the group of items arranged in the front row of a shelf, most of one side of the item (for example, more than half of one side) is usually shown in the item shelf image.

[0016] By learning using such training data, the second model can accurately detect image areas corresponding to the group of products located at the front of each shelf, but may not be able to accurately detect image areas corresponding to the group of products located behind the group of products at the front and with most of one side hidden.

[0017] Furthermore, the above-mentioned "first model" is a model that identifies an image area (hereinafter sometimes referred to as "first image area" or "first item group image area") based on the space primarily occupied by the item group in the target image (hereinafter sometimes referred to as "item main occupied space").

[0018] For example, the first model may be a trained model trained using training data including the following images: - An image showing the entire side of a single item. - An item shelf image in which an image area corresponding to an item group image of an item group arranged in the front row of each shelf in the item shelf image is designated as an "item group image area." - A goods shelf image in which an image area corresponding to an "image equivalent to the true background (hereinafter sometimes referred to as a "true background image")" in the goods shelf image is designated as a "true background image area."

[0019] Here, the "true background image area" includes an image area corresponding to the image of the back panel, side panel, or shelf of the goods shelf that is shown in the goods shelf image without being hidden by the shadow of the goods.

[0020] By learning using such training data, the first model has the characteristic of being able to accurately detect image areas that are likely to be background images (e.g., image areas corresponding to large empty spaces where no items are placed) and image areas corresponding to the above-mentioned space mainly occupied by items. However, because the first model learns contradictory information such as "item group image areas" and "true background image areas," for areas that cannot be designated as either, there is a possibility that the model will detect areas close to "item group image areas" as "item group image areas" and areas close to "true background image areas" as "true background image areas." As a result, there is a possibility that the model will detect an image area corresponding to a narrow empty space sandwiched between two "item group image areas" as part of the item group image area. This is because it is thought that there are many cases in which an image corresponding to a narrow empty space does not provide enough information to detect it as a background image.

[0021] Unlike the learning data of the first model, the learning data of the second model does not include any article shelf images for which the "true background image region" is specified, or if it does include any, the number of such images is small.

[0022] As described above, the detection device 10 detects an image area corresponding to an image of a group of items in a captured image using the "first model" and the "second model" that have different detection characteristics, thereby improving the detection accuracy of the image area.

[0023] <Detection device configuration example> FIG. 1 is a block diagram showing an example of a detection device according to the first embodiment. In FIG. 1, the detection device 10 includes a detection unit (first detection unit) 11, a detection unit (second detection unit) 12, an identification unit 13, an index calculation unit 14, and a determination unit 15. The detection device 10 acquires a captured image. This captured image is, for example, an image of a shelf where a shelf is captured. The following description will be given on the assumption that the captured image is an image of a shelf. This shelf has multiple shelves.

[0024] The detection unit (first detection unit) 11 detects a "first image region" in the captured image by applying a first model to the captured image.

[0025] The detection unit (second detection unit) 12 detects a "second image area" in the captured image by applying a second model to the captured image.

[0026] The identification unit 13 identifies a plurality of "shelf level image areas" corresponding to the plurality of shelf levels in the captured image. A shelf level image area corresponding to one shelf level is, for example, an image area corresponding to the space between the shelf board of that shelf level and the shelf board of the shelf level immediately above that shelf level.

[0027] The index calculation unit 14 calculates an "index related to the deviation" between the first image area and the second image area in each shelf image area. A specific example of the "index related to the deviation" will be described in the second embodiment.

[0028] The determination unit 15 determines the detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the above-mentioned index calculated for each shelf level image area.

[0029] <Example of detection device operation> An example of the processing operation of the detection device 10 having the above configuration will be described below. Fig. 2 is a flowchart showing an example of the processing operation of the detection device in the first embodiment.

[0030] The detection unit 11 detects a first image region in the captured image by applying a first model to the captured image (step S101).

[0031] The detection unit 12 detects a second image region in the captured image by applying the second model to the captured image (step S102).

[0032] The identifying unit 13 identifies a plurality of shelf level image areas corresponding to the plurality of shelf levels in the captured image (step S103).

[0033] The index calculation unit 14 calculates an index related to the deviation between the first image area and the second image area in each shelf image area (step S104).

[0034] The determination unit 15 determines the detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area (step S105).

[0035] As described above, according to the first embodiment, the detection device 10 detects an item group image area in a captured image using a "first model" and a "second model" that have different detection characteristics. This allows one model to compensate for the weaknesses of the other model in detecting objects, thereby improving the detection accuracy of the item group image area.

[0036] Furthermore, the index calculation unit 14 in the detection device 10 calculates an "index related to the deviation" between the first image area and the second image area in each shelf level image area. The determination unit 15 determines the detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area.

[0037] This configuration of the detection device 10 can improve the detection accuracy of the item group image area. That is, if the above index indicates a large deviation between the first image area and the second image area in a shelf level image area, it is highly likely that items located at the back are also captured in the shelf level image area. When the second model is applied to such a shelf level image area, it is possible that the image area corresponding to items located at the back cannot be accurately detected. On the other hand, when the first model is applied to such a shelf level image area, it is highly likely that the image area corresponding to items located at the back can be accurately detected. Therefore, in such cases, the detection device 10 can use the first image area obtained by applying the first model to the shelf level image area as the detection result. On the other hand, when the above index indicates a small deviation between the first image area and the second image area in a shelf level image area, it is highly likely that few items located at the back are captured in the shelf level image area. When the second model is applied to such a shelf level image area, it is highly likely that the image area corresponding to the product group located at the front can be accurately detected. Therefore, in such a case, the detection device 10 can use the second image area obtained by applying the second model to the one shelf image area as the detection result. Therefore, one model can compensate for the weak detection of the detection target of the other model, thereby improving the detection accuracy of the item group image area.

[0038] Second Embodiment The second embodiment relates to an embodiment that embodies the contents of the first embodiment more specifically.

[0039] Fig. 3 is a block diagram showing an example of a detection device according to the second embodiment. In Fig. 3, the detection device 20 includes a detection unit (first detection unit) 21, a detection unit (second detection unit) 22, an identification unit 23, an index calculation unit 24, and a determination unit 25. The detection device 20 acquires a captured image. This captured image is, for example, a shelf image of a shelf. The following description will be given on the assumption that the captured image is a shelf image. This shelf has a plurality of shelves.

[0040] The detector 22, like the detector 12 of the first embodiment, detects a second image area in the captured image (that is, the item shelf image) by applying a second model to the captured image.

[0041] Fig. 4A is a diagram showing an example of an article shelf image, and Fig. 4B is a diagram showing an example of a second image area.

[0042] FIG. 4A shows an image of a shelf on which bottled beverages are displayed. The shelf shown in the image of FIG. 4A has four shelves. On the top three shelves, the plastic bottles are arranged upright. On the bottom shelf, the plastic bottles are arranged lying on their sides and stacked. On the top three shelves, the space above the plastic bottles is narrow, while on the bottom shelf, the space above the plastic bottles is wide. For this reason, in the image of the top three shelves, most of the plastic bottles at the back are hidden by the plastic bottles in front, while in the image of the bottom shelf, even the plastic bottles at the back are visible.

[0043] The result of applying the second model to the shelf image of Figure 4A is shown in Figure 4B. In Figure 4B, the shaded area corresponds to the second image area.

[0044] As can be seen in Figure 4B, the second model can accurately detect the item group image area corresponding to the plastic bottles in the image of the top three shelves. As a result, the second model can accurately detect even image areas corresponding to narrow empty spaces sandwiched between two item group image areas.

[0045] On the other hand, as can be seen from Figure 4B, in the image of the bottom shelf, the item group image area corresponding to the group of plastic bottles located at the front can be detected, but the item group image area corresponding to the group of plastic bottles located at the back cannot be detected.

[0046] Similar to the detection unit 11 of the first embodiment, the detection unit 21 detects a "first image region" in the captured image by applying a first model to the captured image.

[0047] FIG. 4C is a diagram showing an example of the first image area. In FIG. 4C, the shaded area in the shelf image area SA14 corresponds to the first image area. As can be seen from FIG. 4C, in the shelf image area SA14 (i.e., the image of the lowest shelf), model 1 can accurately detect not only the item group image area corresponding to the group of plastic bottles located at the front, but also the item group image area corresponding to the group of plastic bottles located at the back. Note that, as can be seen from FIGS. 4B and 4C, model 1 may detect an image area corresponding to a narrow empty space SP1 sandwiched between two item group image areas as part of the item group image area.

[0048] Returning to the explanation of FIG. 3, the identification unit 23, like the identification unit 13 of the first embodiment, identifies a plurality of "shelf level image areas" corresponding to the plurality of shelf levels in the captured image.

[0049] For example, as can be seen in Figure 4B, the lower line defining the second image area in the image of each shelf appears as a straight line that is approximately parallel to the shelf. In other words, by identifying this straight line, it is possible to identify the line that corresponds to the surface of the plastic bottle that is in contact with the shelf.

[0050] Therefore, the specifying unit 23 may specify the lower line that defines the second image area in the image of each shelf level, and specify the image area sandwiched between two adjacent lines as the "shelf level image area."

[0051] Alternatively, for example, the identification unit 23 may directly identify the front image of the shelf by pattern matching or the like. This front image of the shelf can also be identified as a line corresponding to the surface of the plastic bottle in contact with the shelf. The identification unit 23 may then identify the image area sandwiched between two adjacent lines as a "shelf level image area." Note that image areas SA11, SA12, SA13, and SA14, each surrounded by a frame in FIG. 4B, are examples of shelf level image areas.

[0052] The index calculation unit 24, like the index calculation unit 14 of the first embodiment, calculates an "index related to the deviation" between the first image area and the second image area in each shelf image area.

[0053] For example, the index calculation unit 24 may calculate the ratio of the area of ​​the first image region to the area of ​​the second image region in each shelf level image region as the "deviation-related index." For example, in the example of FIGS. 4B and 4C, for shelf level image regions SA11, SA12, and SA13, the ratio of the area of ​​the first image region to the area of ​​the second image region is close to 1. This is because the areas of the second image region and the first image region are approximately equal in shelf level image regions SA11, SA12, and SA13. On the other hand, for shelf level image region SA14, the ratio of the area of ​​the first image region to the area of ​​the second image region is greater than the ratios for shelf level image regions SA11, SA12, and SA13. This is because the difference between the area of ​​the second image region and the area of ​​the first image region is large in shelf level image region SA14.

[0054] Similar to the determination unit 15 of the first embodiment, the determination unit 25 determines the detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the above-mentioned indicators calculated for each shelf level image area.

[0055] For example, the determination unit 25 determines the detection result to be adopted for each shelf level image area based on the ratio of the area of ​​the first image area to the area of ​​the second image area in each shelf level image area. Specifically, for shelf level image areas where the ratio is equal to or greater than a threshold, the determination unit 25 determines that the first image area should be adopted as the detection result. That is, for shelf level image area SA14, the determination unit 25 determines that the first image area should be adopted as the detection result. On the other hand, for shelf level image areas where the ratio is less than the threshold, the determination unit 25 determines that the second image area should be adopted as the detection result. That is, for shelf level image areas SA11, SA12, and SA13, the determination unit 25 determines that the second image area should be adopted as the detection result.

[0056] <Third embodiment> The third embodiment relates to identifying free space.

[0057] Fig. 5 is a block diagram showing an example of a detection device according to the third embodiment. In Fig. 5, the detection device 30 includes a detection unit (first detection unit) 11, a detection unit (second detection unit) 12, an identification unit 13, an index calculation unit 14, a determination unit 15, an integration unit 31, and a space identification unit 32.

[0058] The integrating unit 31 integrates the detection results used for each shelf level image area to obtain an "integrated image." For example, in the case of Figures 4B and 4C above, the second image area in shelf level image areas SA11, SA12, and SA13 detected by the second model and the first image area in shelf level image area SA14 detected by the first model are integrated to form an "integrated image." Figure 6 is a diagram showing an example of an integrated image.

[0059] The space identifying unit 32 identifies empty spaces on each shelf where no items are placed, based on the integrated image. For example, the space identifying unit 32 may identify empty spaces by subtracting the integrated image from the shelf image area. In FIG. 6, for example, the areas surrounded by rectangular frames (spaces SP1, SP2, SP3, and SP4) correspond to empty spaces.

[0060] As described above, the detection device 30 in the third embodiment identifies vacant space based on an integrated image obtained by applying the first model and the second model to the shelf image areas, which are the detection targets that the first model and the second model are each good at detecting, and thus can identify vacant space with high accuracy.

[0061] Although the description has been given here on the assumption that the integrating unit 31 and the space identifying unit 32 are applied to the detection device 10 of the first embodiment, the present disclosure is not limited to this. The integrating unit 31 and the space identifying unit 32 may also be applied to the detection device 20 of the second embodiment.

[0062] <Other embodiments> FIG. 7 is a diagram illustrating an example of the hardware configuration of a detection device. In FIG. 7, the detection device 100 includes a processor 101 and a memory 102. The processor 101 may be, for example, a microprocessor, a microprocessing unit (MPU), or a central processing unit (CPU). The processor 101 may include multiple processors. The memory 102 is configured by a combination of volatile memory and non-volatile memory. The memory 102 may include storage located remotely from the processor 101. In this case, the processor 101 may access the memory 102 via an I / O interface (not shown).

[0063] The detection devices 10, 20, and 30 of the first to third embodiments may each have the hardware configuration shown in FIG. 7 . The detection units 11 and 21, the detection units 12 and 22, the identification units 13 and 23, the index calculation units 14 and 24, the determination units 15 and 25, the integration unit 31, and the space identification unit 32 of the detection devices 10, 20, and 30 of the first to third embodiments may be realized by the processor 101 reading and executing a program stored in the memory 102. The program can be stored using various types of non-transitory computer-readable media and provided to the detection devices 10, 20, and 30. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, and hard disk drives) and magneto-optical recording media (e.g., magneto-optical disks). Further examples of non-transitory computer-readable media include CD-ROMs (Read Only Memory), CD-Rs, and CD-R / Ws. Further, examples of non-transitory computer-readable media include semiconductor memories. Semiconductor memories include, for example, mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash ROM, and random access memory (RAM). The program may also be provided to the detection devices 10, 20, and 30 by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media may provide the program to the detection devices 10, 20, and 30 via wired communication paths such as electrical wires and optical fibers, or wireless communication paths.

[0064] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention. [Explanation of symbols]

[0065] 10. Detection Device 11 Detection unit (first detection unit) 12 Detection unit (second detection unit) 13 Specific section 14 Indicator calculation section 15 Decision Section 20 Detection Device 21 Detection unit (first detection unit) 22 Detection unit (second detection unit) 23 Specific section 24 Indicator calculation section 25 Decision Section 30 Detection Device 31 Integration Department 32 Space Identification Section

Claims

1. a first detection unit that detects a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; a second detection unit that detects the second image area in the captured image using a second model that identifies a second image area corresponding to an image of a group of items arranged in front in the target image; an identification unit that identifies a plurality of shelf level image areas in the captured image that correspond to a plurality of shelf levels of the item shelf; an index calculation unit that calculates an index related to a deviation between the first image area and the second image area in each shelf image area; a determination unit that determines a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; A detection device comprising:

2. the index calculation unit calculates, as the index, a ratio of an area of ​​the first image area to an area of ​​the second image area in each shelf image area; the determination unit determines a detection result to be adopted for each shelf level image area based on the ratio. The detection device according to claim 1 .

3. The determination unit For a shelf image area where the ratio is equal to or greater than a threshold, the first image area is determined to be the detection result to be adopted, For a shelf image area where the ratio is less than the threshold, the second image area is determined to be the detection result to be adopted. The detection device according to claim 2.

4. the identification unit identifies a plurality of lines corresponding to surfaces of items in contact with shelves of the item shelf in the second image area, and identifies the plurality of shelf level image areas by dividing the captured image by the plurality of lines. The detection device according to claim 1 .

5. and further comprising an integration unit that integrates the detection results employed for each shelf image region to obtain an integrated image. The detection device according to claim 1 .

6. a space identifying unit that identifies an empty space on each shelf where no article is placed based on the integrated image; The detection device according to claim 5.

7. Detecting a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; Detecting a second image region in the captured image using a second model that identifies a second image region corresponding to an image of a group of items located in front of the target image; Identifying a plurality of shelf level image areas in the captured image, each of which corresponds to a plurality of shelf levels of the item shelf; Calculating an index related to a deviation between the first image area and the second image area in each shelf image area; determining a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; A detection method comprising:

8. The calculating step includes calculating, as the index, a ratio of an area of ​​the first image area to an area of ​​the second image area in each shelf image area, The determining step includes determining a detection result to be adopted for each shelf image region based on the ratio. The detection method according to claim 7.

9. Detecting a first image area in a captured image of an item shelf using a first model that identifies a first image area based on a main item occupied space that is mainly occupied by a group of items in the target image; Detecting a second image region in the captured image using a second model that identifies a second image region corresponding to an image of a group of items located in front of the target image; Identifying a plurality of shelf level image areas in the captured image, each of which corresponds to a plurality of shelf levels of the item shelf; Calculating an index related to a deviation between the first image area and the second image area in each shelf image area; determining a detection result to be adopted for each shelf level image area from the first image area and the second image area in each shelf level image area based on the index calculated for each shelf level image area; A program that causes a detection device to execute a process including the steps of:

10. The calculating step includes calculating, as the index, a ratio of an area of ​​the first image area to an area of ​​the second image area in each shelf image area, The determining step includes determining a detection result to be adopted for each shelf image region based on the ratio. The program according to claim 9.

Images