Detection device, detection method, and program

Abstract

A detection device (10), wherein a determination unit (15) determines, on the basis of an orientation in which an article is placed on a shelf stage corresponding to each shelf stage image region, a use model for when the shelf stage image regions are defined as image regions being processed, from among a plurality of models. Each of the plurality of models is used for detecting an article group image region. The plurality of models have different detection characteristics from each other. The detection unit (14) applies the use model to the image regions being processed, thereby detecting the article group image region in the image regions being processed.

Description

DETECTION APPARATUS, DETECTION METHOD, AND NON-TRANSITORY COMPUTER-READABLE MEDIUM

[0001] The present disclosure relates to a detection apparatus, a detection method, and a non-transitory computer-readable medium.

[0002] A technology has been proposed that uses a single trained discrimination model to detect an area in an image of a group of items such as merchandise where a group of items is continuously present (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-117531

[0004] The present inventors have found that a single discrimination model may not be able to achieve sufficient accuracy in detecting an image region. That is, a discrimination model typically has targets that it is good at detecting and targets that it is not good at detecting. Therefore, the present inventors have found that the accuracy in detecting an image region can be improved by applying a first model and a second model with different characteristics to an image of a group of items.

[0005] One objective of the present disclosure is to provide a detection device, a detection method, and a non-transitory computer-readable medium that can improve the detection accuracy of an image region. It should be noted that this objective is only one of multiple objectives that multiple embodiments disclosed in this specification aim to achieve. Other objectives or problems and novel features will become apparent from the description of this specification or the accompanying drawings.

[0006] In one aspect, the detection device comprises: a first detection unit that detects a single item image of a single item in a captured image of an item shelf; a first identification unit that identifies a plurality of shelf level image areas in the captured image that correspond respectively to a plurality of shelf levels of the item shelf; a determination unit that determines the orientation in which an item is placed on a shelf level corresponding to each shelf level image area based on the single item image detected in each shelf level image area; a determination unit that determines a model to be used when each shelf level image area is set as a processing target image area from a plurality of models, each model being a model for detecting an item group image area and having different detection characteristics from one another, based on the orientation in which the item is placed; and a second detection unit that detects an item group image area that corresponds to an image of the item group in the processing target image area by applying the usage model to the processing target image area.

[0007] In another aspect, the detection method includes: detecting a single item image of a single item in a captured image of an item shelf; identifying a plurality of shelf level image areas in the captured image, each corresponding to a plurality of shelf levels of the item shelf; determining the posture in which the item is placed on the shelf level corresponding to each shelf level image area, based on the single item image detected in each shelf level image area; determining a model to be used when each shelf level image area is set as a processing target image area, from a plurality of models, each model being a model for detecting an item group image area and having different detection characteristics from one another, based on the posture in which the item is placed; and detecting an item group image area corresponding to an image of the item group in the processing target image area by applying the usage model to the processing target image area.

[0008] In another aspect, a non-transitory computer-readable medium stores a program that causes a detection device to execute processes including: detecting a single item image of a single item in a captured image of an item shelf; identifying a plurality of shelf level image areas in the captured image, each corresponding to a plurality of shelf levels of the item shelf; determining the orientation in which the item is placed on the shelf level corresponding to each shelf level image area, based on the single item image detected in each shelf level image area; determining a model to be used when each shelf level image area is set as a processing target image area, from a plurality of models, each model being a model for detecting an item group image area and having different detection characteristics, based on the orientation in which the item is placed; and detecting an item group image area that corresponds to an image of the item group in the processing target image area by applying the usage model to the processing target image area.

[0009] The present disclosure provides a detection device, a detection method, and a non-transitory computer-readable medium that can improve the accuracy of detecting an image region.

[0010] FIG. 1 is a block diagram showing an example of a detection device in a first embodiment; FIG. 2 is a flowchart showing an example of a processing operation of the detection device in the first embodiment; FIG. 3 is a block diagram showing an example of a detection device in a second embodiment; FIG. 4 is a diagram showing an example of an item shelf image; FIG. 5 is a diagram showing an example of a second image area; FIG. 6 is a diagram showing an example of a first image area; FIG. 7 is a block diagram showing an example of a detection device in a third embodiment; FIG. 8 is a diagram showing an example of an integrated image; and FIG. 9 is a diagram showing an example of the hardware configuration of the detection device.

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements are denoted by the same reference numerals, and redundant description will be omitted.

[0012] First Embodiment

[0013] <Configuration example of detection device> Fig. 1 is a block diagram showing an example of a detection device in the first embodiment. In Fig. 1, the detection device 10 has a detection unit (first detection unit) 11, an identification unit (first identification unit) 12, a determination unit 13, a detection unit (second detection unit) 14, and a determination unit 15. This captured image is, for example, an image of a shelf taken of an item shelf. The following description will be given on the assumption that the captured image is an image of a shelf. This shelf has multiple shelves.

[0014] The detection unit 11 detects an "image of an individual item" of an individual item in a captured image. For example, the detection unit 11 may detect an "image of an individual item" by using "AI (Artificial Intelligence) that detects an individual item."

[0015] The identification unit 12 identifies a plurality of "shelf level image areas" corresponding to the plurality of shelf levels in the captured image. A shelf level image area corresponding to one shelf level is, for example, an image area corresponding to the space between the shelf board of the shelf level and the shelf board of the shelf level immediately above the one shelf level.

[0016] The determination unit 13 determines the posture in which the item is placed on the shelf corresponding to each shelf image area based on the image of the single item detected in each shelf image area. The posture in which the item is placed on the shelf includes, for example, an upright posture (hereinafter sometimes referred to as a "first posture") and a lying posture (hereinafter sometimes referred to as a "second posture").

[0017] The determination unit 15 determines, from among the plurality of models, a "model to be used" when each shelf image area is set as a "processing target image area" for the detection unit 14, based on the "posture of the items placed on the shelf" determined by the determination unit 13. Each of the plurality of models is a model for detecting the item group image area. Furthermore, the plurality of models have different detection characteristics.

[0018] The detection unit 14 detects an "item group image area" corresponding to the image of the item group in the "processing target image area" by applying the "usage model" to the "processing target image area." The detection unit 14 sets each of the multiple shelf image areas identified by the identification unit 12 as the "processing target image area."

[0019] <Example of Operation of Detection Device> A description will be given of an example of processing operations of the detection device 10 having the above configuration. Fig. 2 is a flowchart showing an example of processing operations of the detection device in the first embodiment.

[0020] The detection unit 11 detects an image of an article alone in a captured image (step S101).

[0021] The identifying unit 12 identifies a plurality of shelf level image areas corresponding to the plurality of shelf levels in the captured image (step S102).

[0022] The determination unit 13 determines the posture of the item placed on the shelf corresponding to each shelf image area based on the image of the single item detected in each shelf image area (step S103).

[0023] Based on the determined "posture in which the item is placed on the shelf," the determination unit 15 determines, from among a plurality of models, a model to be used when each shelf image area is set as the processing target image area (step S104).

[0024] The detection unit 14 detects an item group image area in the processing target image area by applying the usage model to the processing target image area (step S105).

[0025] As described above, according to the first embodiment, the detection device 10 detects an item group image area in a captured image using multiple models with different detection characteristics. This allows one model to compensate for the weaknesses of the other model in detecting objects, thereby improving the detection accuracy of the item group image area.

[0026] Furthermore, in the detection device 10, the determination unit 15 determines, from among a plurality of models, a model to be used when each shelf level image area is set as the processing target image area, based on the orientation of the items placed on the shelf level corresponding to each shelf level image area. Each of the plurality of models is a model for detecting an item group image area. Furthermore, the plurality of models differ from one another in detection characteristics. The detection unit 14 detects the item group image area in the processing target image area by applying the usage model to the processing target image area.

[0027] This configuration of the detection device 10 makes it possible to apply a model corresponding to the arrangement of items on each shelf to the shelf image area, thereby improving the detection accuracy of the item group image area.

[0028] Second Embodiment The second embodiment relates to an embodiment that further embodies the contents of the first embodiment.

[0029] FIG. 3 is a block diagram showing an example of a detection device according to the second embodiment. In FIG. 3, the detection device 20 includes a detection unit (first detection unit) 21, an identification unit (first identification unit) 22, a determination unit 23, a detection unit (second detection unit) 24, a determination unit 25, an identification unit (second identification unit) 26, and an average value calculation unit 27. The detection device 20 acquires a captured image, similar to the detection device 10 of the first embodiment. This captured image is, for example, a shelf image of a shelf. The following description will be given on the assumption that the captured image is a shelf image. This shelf has multiple shelves.

[0030] The detector (first detector) 21 detects a "single item image" of a single item in a captured image, similar to the detector 11 of the first embodiment.

[0031] FIG. 4A is a diagram showing an example of a shelf image. FIG. 4A shows a shelf image of a shelf on which bottled beverages are displayed. The shelf shown in the shelf image of FIG. 4A has four shelf levels. On the top three shelf levels, plastic bottles are arranged upright. On the bottom shelf level, plastic bottles are arranged lying on their sides and stacked. On the top three shelf levels, the space above the plastic bottles is narrow, while on the bottom shelf level, the space above the plastic bottles is wide. Therefore, in the image of the top three shelf levels, most of the plastic bottles at the back are hidden by the plastic bottles in front, while in the image of the bottom shelf, even the plastic bottles at the back are visible.

[0032] For example, each image area surrounded by a rectangular frame BB in Fig. 4A corresponds to an "image of an item alone." This rectangular frame may be a so-called bounding box.

[0033] Returning to the explanation of FIG. 3 , the identification unit 22 (first identification unit), like the identification unit 12 of the first embodiment, identifies a plurality of "shelf level image areas" corresponding to a plurality of shelf levels in the captured image. For example, the identification unit 22 may identify a line corresponding to a surface of an item that contacts a shelf board of an item shelf in an image of the item alone. The identification unit 22 may then identify a plurality of shelf level image areas by dividing the captured image along the identified line. In other words, the identification unit 22 may identify an image area sandwiched between two adjacent lines as a "shelf level image area."

[0034] Alternatively, for example, the identification unit 22 may directly identify the front image of the shelf by pattern matching or the like. This front image of the shelf can also be identified as a line corresponding to the surface of the plastic bottle in contact with the shelf. The identification unit 22 may then identify the image area sandwiched between two adjacent lines as a "shelf level image area." Note that image areas SA11, SA12, SA13, and SA14, each surrounded by a dashed line frame in FIG. 4A, are examples of shelf level image areas.

[0035] The average value calculation unit 27 calculates the average value of the length corresponding to the height direction of the item shelf for at least one single item image in each shelf image area. That is, in the example of Fig. 4A, the average values ​​for the top three shelf levels tend to be large because the plastic bottles are placed upright on these shelves. On the other hand, the average value for the bottom shelf level tends to be small because the plastic bottles are placed horizontally.

[0036] The specifying unit (second specifying unit) 26 specifies the length of each shelf level image area corresponding to the height direction of the article shelf as the height (α) of each shelf level image area.

[0037] Similar to the determination unit 13 of the first embodiment, the determination unit 23 determines the posture of an item placed on a shelf corresponding to each shelf image area based on the image of the item alone detected in each shelf image area.

[0038] For example, the determination unit 23 calculates a "reference value" for each shelf level image area by multiplying the height (α) of each shelf level image area by a predetermined ratio (e.g., 0.7). The determination unit 23 then compares the calculated "reference value" for each shelf level image area with the average value of the lengths of the individual item images corresponding to each shelf level image area calculated by the average value calculation unit 27, thereby determining the orientation of the item placed on the shelf level corresponding to each shelf level image area. For example, if the average value for the target shelf level image area is equal to or greater than the reference value for the target shelf level image area, the determination unit 23 may determine that the item is placed in an upright orientation (i.e., the first orientation) on the shelf level corresponding to the target shelf level image area. On the other hand, if the average value for the target shelf level image area is less than the reference value for the target shelf level image area, the determination unit 23 may determine that the item is placed in a horizontal orientation (i.e., the second orientation) on the shelf level corresponding to the target shelf level image area.

[0039] Similar to the determination unit 15 of the first embodiment, the determination unit 25 determines, from among a plurality of models, a "use model" to be used when each shelf image area is set as the "image area to be processed" of the detection unit 24, based on the "posture in which the item is placed on the shelf" determined by the determination unit 23.

[0040] For example, the determination unit 25 determines the "second model" as the "usage model" for a shelf image area corresponding to a shelf where it is determined that the posture on which an item is placed is an upright posture, whereas the determination unit 25 determines the "first model" as the "usage model" for a shelf image area corresponding to a shelf where it is determined that the posture on which an item is placed is a lying posture.

[0041] Here, the first model and the second model will be described.

[0042] The "second model" is a model that identifies an image area (hereinafter sometimes referred to as the "second image area" or the "second item group image area") that corresponds to the image of the item group placed in the foreground in the target image. Note that, hereinafter, the "item group image" may be referred to as the "item group image."

[0043] For example, the second model may be a trained model that has been trained using training data including the following images: - An image of one entire side of a single item. - An item shelf image in which an image area corresponding to an item group image of an item group arranged in the front row of each shelf in the item shelf image is designated as an "item group image area."

[0044] In addition, for each item in the group of items arranged in the front row of a shelf, most of one side of the item (for example, more than half of one side) is usually shown in the item shelf image.

[0045] By learning using such learning data, the second model can accurately detect image areas corresponding to groups of items located at the front of each shelf, but may not be able to accurately detect image areas corresponding to groups of items located behind the groups of items located at the front and with most of one side hidden.

[0046] Furthermore, the above-mentioned "first model" is a model that identifies an image area (hereinafter, sometimes referred to as the "first image area" or "first item group image area") based on the space that is primarily occupied by the item group in the target image (hereinafter, sometimes referred to as the "item main occupied space").

[0047] For example, the first model may be a trained model that has been trained using training data including the following images: - An image capturing the entirety of one side of a single item. - An item shelf image in which an image area corresponding to an item group image of a group of items lined up in the front row of each shelf in the item shelf image is designated as an "item group image area." - An item shelf image in which an image area corresponding to an "image equivalent to the true background (hereinafter sometimes referred to as a "true background image")" in the item shelf image is designated as a "true background image area."

[0048] Here, the "true background image area" includes an image area corresponding to the image of the back panel, side panel, or shelf of the goods shelf that is shown in the goods shelf image without being hidden by the shadow of the goods.

[0049] By learning using such training data, the first model has the characteristic of being able to accurately detect image areas that are likely to be background images (e.g., image areas corresponding to large empty spaces where no items are placed) and image areas corresponding to the above-mentioned space primarily occupied by items. However, because the first model learns contradictory information, namely, "item group image areas" and "true background image areas," for areas that cannot be designated as either of these, there is a possibility that the model will detect areas close to "item group image areas" as "item group image areas" and areas close to "true background image areas" as "true background image areas." As a result, there is a possibility that the model will detect an image area corresponding to a narrow empty space sandwiched between two "item group image areas" as part of the item group image area. This is because it is believed that in many cases, images corresponding to narrow empty spaces do not provide enough information to detect that the image is a background image.

[0050] Unlike the learning data of the first model, the learning data of the second model does not include any shelf images for which the "true background image region" is specified, or if it does include any, the number of such images is small.

[0051] Similar to the detection unit 14 of the first embodiment, the detection unit (second detection unit) 24 detects an "item group image area" corresponding to an image of an item group in the "image area to be processed" by applying a "usage model" to the "image area to be processed."

[0052] For example, the detection unit 24 detects the "second image area" by applying the second model to a shelf image area corresponding to a shelf where it has been determined that the posture on which an item is placed is an upright posture. The detection unit 24 also detects the "first image area" by applying the first model to a shelf image area corresponding to a shelf where it has been determined that the posture on which an item is placed is a lying posture.

[0053] Fig. 4B is a diagram showing an example of the second image area. Fig. 4B shows the result of applying the second model to the item shelf image of Fig. 4A. In Fig. 4B, the shaded area corresponds to the second image area. Note that, in reality, the second model is not applied to shelf level image area SA14, where items are placed in a horizontal orientation. However, for reference, Fig. 4B also shows the result of applying the second model to shelf level image area SA14.

[0054] 4B, the second model can accurately detect the item group image area corresponding to the plastic bottles in the image of the top three shelves. As a result, the second model can accurately detect even an image area corresponding to a narrow empty space sandwiched between two item group image areas.

[0055] On the other hand, as can be seen from Figure 4B, in the image of the bottom shelf, the item group image area corresponding to the group of PET bottles located at the front can be detected, but the item group image area corresponding to the group of PET bottles located at the back cannot be detected.

[0056] 4C is a diagram showing an example of the first image area. In FIG. 4C, the shaded area in shelf level image area SA14 corresponds to the first image area. Note that, in reality, the first model is not applied to shelf level image areas SA11, SA12, and SA13 where the position in which an item is placed is an upright position. However, for reference, the results of applying the second model to shelf level image areas SA11, SA12, and SA13 are also shown in FIG. 4C.

[0057] As can be seen in Figure 4C, model 1 is able to accurately detect not only the item group image area corresponding to the group of plastic bottles located at the front in shelf image area SA14 (i.e., the image of the bottom shelf), but also the item group image area corresponding to the group of plastic bottles located at the back. Note that, as can be seen in Figures 4B and 4C, model 1 may detect image area SP1 corresponding to a narrow empty space sandwiched between two item group image areas as part of the item group image area.

[0058] As described above, according to the second embodiment, the determination unit 25 in the detection device 20 determines the "second model" as the "usage model" for a shelf image area corresponding to a shelf where it is determined that the posture on which an item is placed is an upright posture. Furthermore, the determination unit 25 determines the "first model" as the "usage model" for a shelf image area corresponding to a shelf where it is determined that the posture on which an item is placed is a lying posture. The second model is a model that identifies a second image area corresponding to an image of a group of items placed at the front in the target image. The first model is a model that identifies a first image area based on the space mainly occupied by the items in the target image.

[0059] This configuration of the detection device 20 can improve the detection accuracy of the item group image area. That is, for example, in a shelf image area corresponding to a shelf where the placement orientation of an item is determined to be lying on its side, it is highly likely that even items located at the back are included. When the second model is applied to such a shelf image area, it is possible that the image area corresponding to items located at the back cannot be accurately detected. On the other hand, when the first model is applied to such a shelf image area, it is highly likely that the image area corresponding to items located at the back can be accurately detected. In contrast, when the placement orientation of an item is determined to be upright, it is highly likely that very few items located at the back are included. When the second model is applied to such a shelf image area, it is highly likely that the image area corresponding to the group of products located at the front can be accurately detected. Therefore, the detection accuracy of the item group image area can be improved because one model can compensate for the weaknesses of the other model in detecting detection targets.

[0060] <Modification of Second Embodiment> In the above description, it has been described that the determination unit 23 determines that the position on the shelf corresponding to the target shelf level image area is an upright position (i.e., the first position) when the average value for the target shelf level image area is equal to or greater than the reference value for the target shelf level image area. Furthermore, it has been described that the determination unit 23 determines that the position on the shelf corresponding to the target shelf level image area is a lying position (i.e., the second position) when the average value for the target shelf level image area is less than the reference value for the target shelf level image area. However, the present disclosure is not limited to this.

[0061] For example, the determination unit 23 may determine the orientation of an item placed on a shelf corresponding to each shelf image area based on the ratio of the length to the width of the image of the item alone detected in each shelf image area. For example, if the height direction of an item shelf is defined as the length and the direction perpendicular to that as the width, the length of the image of the item alone corresponding to an upright plastic bottle is longer than the width, so the value of length / width is greater than 1. On the other hand, the length of the image of the item alone corresponding to a plastic bottle laid on its side is shorter than the width, so the value of length / width is less than 1. Therefore, the orientation of an item placed on a shelf corresponding to each shelf image area can be determined based on the ratio of the length to the width of the image of the item alone detected in each shelf image area.

[0062] Third Embodiment The third embodiment relates to identifying free space.

[0063] Fig. 5 is a block diagram showing an example of a detection device according to the third embodiment. In Fig. 5, the detection device 30 includes a detection unit (first detection unit) 11, an identification unit (first identification unit) 12, a determination unit 13, a detection unit (second detection unit) 14, a determination unit 15, an integration unit 31, and a space identification unit 32.

[0064] The integrating unit 31 obtains an "integrated image" by integrating a first image area in the image area to be processed detected by the detecting unit 14 with a second image area in the image area to be processed detected by the detecting unit 11. For example, in the case of Figures 4B and 4C above, the second image area in the shelf level image areas SA11, SA12, SA13 detected by the second model and the first image area in the shelf level image area SA14 detected by the first model are integrated to form an "integrated image." Figure 6 is a diagram showing an example of an integrated image.

[0065] The space identifying unit 32 identifies empty spaces on each shelf level where no items are placed, based on the integrated image. For example, the space identifying unit 32 may identify empty spaces by subtracting the integrated image from the shelf level image area. In Figure 6, for example, the areas surrounded by rectangular frames (spaces SP1, SP2, SP3, and SP4) correspond to empty spaces.

[0066] As described above, the detection device 30 in the third embodiment identifies vacant space based on an integrated image obtained by applying the first model and the second model to the shelf image areas, which are the detection targets that the first model and the second model are each good at detecting, and thereby can identify vacant space with high accuracy.

[0067] Although the description herein has been given on the assumption that the integrating unit 31 and the space identifying unit 32 are applied to the detection device 10 of the first embodiment, the present disclosure is not limited to this. The integrating unit 31 and the space identifying unit 32 may also be applied to the detection device 20 of the second embodiment.

[0068] <Other Embodiments> Fig. 7 is a diagram illustrating an example of the hardware configuration of a detection device. In Fig. 7, the detection device 100 includes a processor 101 and a memory 102. The processor 101 may be, for example, a microprocessor, a microprocessing unit (MPU), or a central processing unit (CPU). The processor 101 may include multiple processors. The memory 102 is configured by a combination of volatile memory and non-volatile memory. The memory 102 may include storage located remotely from the processor 101. In this case, the processor 101 may access the memory 102 via an I / O interface (not shown).

[0069] The detection devices 10, 20, and 30 of the first to third embodiments may each have the hardware configuration shown in FIG. 7 . The detection units 11 and 21, the identification units 12 and 22, the determination units 13 and 23, the detection units 14 and 24, the determination units 15 and 25, the identification unit 26, the average value calculation unit 27, the integration unit 31, and the space identification unit 32 of the detection devices 10, 20, and 30 of the first to third embodiments may be realized by the processor 101 reading and executing a program stored in the memory 102. The program can be stored using various types of non-transitory computer-readable media and supplied to the detection devices 10, 20, and 30. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, and hard disk drives) and magneto-optical recording media (e.g., magneto-optical disks). Further examples of non-transitory computer-readable media include CD-ROMs (Read Only Memory), CD-Rs, and CD-R / Ws. Further, examples of non-transitory computer-readable media include semiconductor memory. Semiconductor memory includes, for example, mask ROM, programmable ROM (PROM), erasable PROM (EPROM), flash ROM, and random access memory (RAM). The program may also be provided to the detection devices 10, 20, and 30 by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media may provide the program to the detection devices 10, 20, and 30 via wired communication paths such as electrical wires and optical fibers, or wireless communication paths.

[0070] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0071] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes: (Supplementary Note 1) A detection device comprising: a first detection unit that detects a single item image of a single item in a captured image of an item shelf; a first identification unit that identifies a plurality of shelf level image areas in the captured image, each corresponding to a plurality of shelf levels of the item shelf; a determination unit that determines the orientation of an item placed on the shelf level corresponding to each shelf level image area based on the single item image detected in each shelf level image area; a determination unit that determines a model to be used when each shelf level image area is set as a processing target image area from a plurality of models, each model being a model for detecting an item group image area and having different detection characteristics from one another, based on the orientation of the item placed; and a second detection unit that detects an item group image area corresponding to an image of the item group in the processing target image area by applying the usage model to the processing target image area. (Supplementary Note 2) The plurality of models include a first model that identifies a first item group image area based on an item main occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image area that corresponds to an image of the item group arranged at the front in the target image, and the determination unit determines the second model as the usage model for shelf image areas corresponding to shelf levels where the posture on which the item is placed is determined to be an upright posture, and determines the first model as the usage model for shelf image areas corresponding to shelf levels where the posture on which the item is placed is determined to be a lying posture. This is a detection device as described in Supplementary Note 1.(Supplementary Note 3) The detection device according to Supplementary Note 1 further comprises: a second identification unit that identifies the length of each shelf level image area corresponding to the height direction of the item shelf as the height of each shelf level image area; and an average value calculation unit that calculates an average value of the length corresponding to the height direction of at least one single item image detected by the first detection unit in each shelf level image area, wherein the determination unit determines that the posture of the item placed on the shelf level corresponding to the target shelf level image area is upright if the average value for the target shelf level image area is equal to or greater than a value obtained by multiplying the height of the target shelf level image area by a predetermined rate, and determines that the posture of the item placed on the shelf level corresponding to the target shelf level image area is lying on its side if the average value for the target shelf level image area is less than the value obtained by multiplying the height of the shelf level corresponding to the target shelf level image area by the predetermined rate. (Supplementary Note 4) The detection device according to Supplementary Note 1, wherein the determination unit determines the orientation of the item placed on the shelf corresponding to each shelf level image area based on the ratio of the length to the width of the image of a single item detected in each shelf level image area. (Supplementary Note 5) The detection device according to Supplementary Note 1, wherein the first identification unit identifies a line corresponding to a surface of the item in the image of the single item that is in contact with a shelf board of the item shelf, and identifies the multiple shelf level image areas by dividing the captured image along the identified line. (Supplementary Note 6) The detection device according to Supplementary Note 2, further comprising an integration unit that obtains an integrated image by integrating the first item group image area obtained by applying the first model to the image area to be processed and the second item group image area obtained by applying the second model to the image area to be processed. (Supplementary Note 7) The detection device according to Supplementary Note 6, further comprising a space identification unit that identifies empty space on each shelf level where no item is placed, based on the integrated image.(Supplementary Note 8) A detection method including: detecting a single item image of a single item in a captured image of an item shelf; identifying a plurality of shelf level image areas in the captured image, respectively corresponding to a plurality of shelf levels of the item shelf; determining the posture in which the item is placed on the shelf level corresponding to each shelf level image area, based on the single item image detected in each shelf level image area; determining a model to be used when each shelf level image area is set as a processing target image area, from a plurality of models, each model being a model for detecting an item group image area and having different detection characteristics from one another, based on the posture in which the item is placed; and detecting an item group image area corresponding to an image of the item group in the processing target image area by applying the model to the processing target image area. (Supplementary Note 9) The detection method described in Supplementary Note 8, wherein the plurality of models include a first model that identifies a first item group image area based on an item main occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image area that corresponds to an image of the item group arranged at the front in the target image, and the determining includes: determining the second model as the usage model for a shelf image area corresponding to a shelf on which the orientation on which the item is placed is determined to be an upright orientation; and determining the first model as the usage model for a shelf image area corresponding to a shelf on which the orientation on which the item is placed is determined to be a lying orientation.(Supplementary Note 10) A non-transitory computer-readable medium storing a program that causes a detection device to execute processes including: detecting a single item image of a single item in a captured image of an item shelf; identifying a plurality of shelf level image areas in the captured image, each corresponding to a plurality of shelf levels of the item shelf; determining the posture in which the item is placed on the shelf level corresponding to each shelf level image area, based on the single item image detected in each shelf level image area; determining a usage model when each shelf level image area is set as a processing target image area, from a plurality of models, each model being a model for detecting an item group image area and having different detection characteristics from one another, based on the posture in which the item is placed; and detecting an item group image area that corresponds to an image of the item group in the processing target image area by applying the usage model to the processing target image area. (Supplementary Note 11) The non-transitory computer-readable medium described in Supplementary Note 10, wherein the plurality of models include a first model that identifies a first item group image area based on an item primary occupied space that is primarily occupied by the item group in the target image, and a second model that identifies a second item group image area corresponding to an image of the item group arranged at the front in the target image, and the determining includes: determining the second model as the usage model for shelf image areas corresponding to shelf levels on which the items are determined to be placed in an upright position; and determining the first model as the usage model for shelf image areas corresponding to shelf levels on which the items are determined to be placed in a horizontally laid position.

[0072] REFERENCE SIGNS LIST 10 Detection device 11 Detection unit (first detection unit) 12 Identification unit (first identification unit) 13 Determination unit 14 Detection unit (second detection unit) 15 Determination unit 20 Detection device 21 Detection unit (first detection unit) 22 Identification unit (first identification unit) 23 Determination unit 24 Detection unit (second detection unit) 25 Determination unit 26 Identification unit (second identification unit) 27 Average value calculation unit 30 Detection device 31 Integration unit 32 Space identification unit

Claims

1. a first detection unit that detects an image of a single item in a captured image of the item shelf; a first identification unit that identifies a plurality of shelf level image areas in the captured image that correspond to a plurality of shelf levels of the item shelf; a determination unit that determines the orientation of an item placed on a shelf corresponding to each shelf image area based on an image of the item alone detected in each shelf image area; a determination unit that determines a model to be used when each shelf image area is set as a processing target image area from among a plurality of models, each model being a model for detecting an article group image area and having different detection characteristics, based on the orientation in which the article is placed; a second detection unit that detects an item group image area corresponding to an image of an item group in the processing target image area by applying the usage model to the processing target image area; A detection device comprising:

2. The plurality of models include a first model that identifies a first item group image area based on an item main occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image area that corresponds to an image of the item group arranged in the front in the target image, The determination unit determining the second model as the usage model for a shelf level image area corresponding to a shelf level where the posture on which the article is placed is determined to be an upright posture; For a shelf image area corresponding to a shelf where it has been determined that the orientation on which the item is placed is a horizontal orientation, the first model is determined as the usage model. The detection device according to claim 1 .

3. a second specifying unit that specifies a length of each shelf level image area corresponding to a height direction of the article shelf as a height of each shelf level image area; an average value calculation unit that calculates an average value of lengths corresponding to the height direction of at least one single item image detected by the first detection unit in each shelf image area; Further comprising: The determination unit If the average value for the target shelf level image area is equal to or greater than a value obtained by multiplying the height of the target shelf level image area by a predetermined ratio, it is determined that the position in which the item is placed on the shelf corresponding to the target shelf level image area is an upright position; If the average value corresponding to the target shelf level image area is less than a value obtained by multiplying the height of the shelf level corresponding to the target shelf level image area by the predetermined ratio, it is determined that the position at which the item is placed on the shelf level corresponding to the target shelf level image area is a position where the item is laid sideways. The detection device according to claim 1 .

4. the determination unit determines the orientation of the item placed on the shelf corresponding to each shelf image area based on the ratio of the length to the width of the image of the item alone detected in each shelf image area. The detection device according to claim 1 .

5. the first identification unit identifies a line corresponding to a surface of the item in the image of the single item that is in contact with a shelf board of the item shelf, and identifies the plurality of shelf level image areas by dividing the photographed image along the identified line; The detection device according to claim 1 .

6. The detection device of claim 2, further comprising an integration unit that integrates the first item group image area obtained by applying the first model to the processing target image area and the second item group image area obtained by applying the second model to the processing target image area to obtain an integrated image.

7. a space identifying unit that identifies an empty space on each shelf where no article is placed based on the integrated image; The detection device according to claim 6.

8. Detecting an image of a single item in a photographed image of the item shelf; Identifying a plurality of shelf level image areas in the captured image, each of which corresponds to a plurality of shelf levels of the item shelf; determining the posture of the item placed on the shelf corresponding to each shelf image area based on the image of the item alone detected in each shelf image area; determining a model to be used when each shelf image area is set as a processing target image area from among a plurality of models each of which is a model for detecting an article group image area and has different detection characteristics based on the posture in which the article is placed; detecting an item group image area corresponding to an image of an item group in the processing target image area by applying the usage model to the processing target image area; A detection method comprising:

9. The plurality of models include a first model that identifies a first item group image area based on an item main occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image area that corresponds to an image of the item group arranged in the front in the target image, The determining step comprises: determining the second model as the usage model for a shelf level image area corresponding to the shelf level on which the posture on which the item is placed is determined to be an upright posture; determining the first model as the usage model for a shelf image area corresponding to a shelf where it has been determined that the orientation on which the item is placed is a horizontal orientation; Including, The detection method according to claim 8.

10. Detecting an image of a single item in a photographed image of the item shelf; Identifying a plurality of shelf level image areas in the captured image, each of which corresponds to a plurality of shelf levels of the item shelf; determining the posture of the item placed on the shelf corresponding to each shelf image area based on the image of the item alone detected in each shelf image area; determining a model to be used when each shelf image area is set as a processing target image area from among a plurality of models each of which is a model for detecting an article group image area and has different detection characteristics based on the posture in which the article is placed; detecting an item group image area corresponding to an image of an item group in the processing target image area by applying the usage model to the processing target image area; A program that causes a detection device to execute a process including the steps of: