Detection device, detection method, and program
The detection device improves image region accuracy by employing multiple models tailored to item orientations, addressing the limitations of single detection models in identifying grouped articles on shelves.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- NEC CORP
- Filing Date
- 2023-03-28
- Publication Date
- 2026-05-19
AI Technical Summary
Existing detection models have limitations in accurately identifying image regions of grouped articles due to varying detection capabilities for different orientations and positions of items on shelves.
A detection device and method that utilizes multiple models with distinct detection characteristics to identify shelf image regions, determining the appropriate model based on the orientation of items, and applies these models to improve detection accuracy of item group image regions.
Enhances the detection accuracy of item group image regions by compensating for the weaknesses of individual models, allowing for precise identification of both foreground and background items, even in complex shelf arrangements.
Smart Images
Figure 0007861913000001 
Figure 0007861913000002 
Figure 0007861913000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to a detection device, a detection method, and a non-temporary computer-readable medium.
Background Art
[0002] A technique has been proposed for detecting a region where a group of articles such as products continuously exists in an image of the group of articles using one learned discrimination model (for example, Patent Document 1).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] The present inventor has found that the detection accuracy of an image region may not be sufficiently obtained with one discrimination model. That is, usually, a discrimination model has a detection target with which it is good at and a detection target with which it is not good at. Therefore, the present inventor has found that the detection accuracy of an image region can be improved by applying a first model and a second model having different characteristics to an image of a group of articles.
[0005] One object of the present disclosure is to provide a detection device, a detection method, and a non-temporary computer-readable medium capable of improving the detection accuracy of an image region. It should be noted that this object is only one of the multiple objects that the multiple embodiments disclosed in this specification seek to achieve. Other objects or problems and novel features will be clarified from the description of this specification or the attached drawings.
Means for Solving the Problems
[0006] In one aspect, the detection device is A first detection unit detects individual item images in a captured image of a shelf, A first identification unit identifies multiple shelf image regions in the captured image that correspond to multiple shelf levels of the item shelf, A determination unit determines the orientation in which an item is placed on the shelf corresponding to each shelf image region, based on the individual item image detected in each shelf image region. Based on the orientation in which the items are placed, a determination unit determines which model to use when each shelf image region is designated as the image region to be processed, from among a plurality of models, each of which is a model for detecting the image region of a group of items and which has different detection characteristics from one another. A second detection unit detects an item group image region corresponding to an image of an item group in the item group in the item group in the item group by applying the usage model to the item group image region to be processed, It is equipped with.
[0007] In other embodiments, the detection method is: The process involves detecting individual item images within captured images of shelves, and In the aforementioned captured image, a plurality of shelf image regions corresponding to each of the plurality of shelf levels of the item shelf are identified, Based on the individual item images detected in each shelf image region, the orientation in which the item is placed on the shelf corresponding to each shelf image region is determined. Based on the orientation in which the items are placed, the model to be used when each shelf image region is the image region to be processed is determined from among multiple models, each of which is a model for detecting the image region of the group of items and has different detection characteristics from one another. By applying the aforementioned model to the image region to be processed, the object group image region corresponding to the image of the object group in the image region to be processed is detected. Includes.
[0008] In other embodiments, non-temporary computer-readable media are The process involves detecting individual item images within captured images of shelves, and In the aforementioned captured image, a plurality of shelf image regions corresponding to each of the plurality of shelf levels of the item shelf are identified, Based on the individual item images detected in each shelf image region, the orientation in which the item is placed on the shelf corresponding to each shelf image region is determined. Based on the orientation in which the items are placed, the model to be used when each shelf image region is the image region to be processed is determined from among multiple models, each of which is a model for detecting the image region of the group of items and has different detection characteristics from one another. By applying the aforementioned model to the image region to be processed, the object group image region corresponding to the image of the object group in the image region to be processed is detected. It stores a program that causes the detection device to execute a process that includes [specific details]. [Effects of the Invention]
[0009] This disclosure provides a detection device, a detection method, and a non-temporary computer-readable medium that can improve the detection accuracy of image regions. [Brief explanation of the drawing]
[0010] [Figure 1] This is a block diagram showing an example of a detection device in the first embodiment. [Figure 2] This flowchart shows an example of the processing operation of the detection device in the first embodiment. [Figure 3] This is a block diagram showing an example of a detection device in the second embodiment. [Figure 4A] This is a diagram showing an example of a shelf image. [Figure 4B] This figure shows an example of the second image region. [Figure 4C] This figure shows an example of the first image region. [Figure 5] This is a block diagram showing an example of a detection device in the third embodiment. [Figure 6] This figure shows an example of an integrated image. [Figure 7]This is a diagram showing an example of the hardware configuration of the detection device.
Embodiments for Carrying Out the Invention
[0011] Hereinafter, embodiments will be described with reference to the drawings. In the embodiments, the same or equivalent elements are denoted by the same reference numerals, and overlapping descriptions are omitted.
[0012] <First Embodiment>
[0013] <Example of the Configuration of the Detection Device> FIG. 1 is a block diagram showing an example of the detection device in the first embodiment. In FIG. 1, the detection device 10 includes a detection unit (first detection unit) 11, a specifying unit (first specifying unit) 12, a determination unit 13, a detection unit (second detection unit) 14, and a determination unit 15. This captured image is, for example, an image of an article shelf in which the article shelf is captured. Hereinafter, the description will be made on the premise that the captured image is an image of an article shelf. This article shelf has a plurality of shelf levels.
[0014] ]> The detection unit 11 detects an "image of a single article" for a single article in the captured image. For example, the detection unit 11 may detect the "image of a single article" using "AI (Artificial Intelligence) for detecting an individual".
[0015] The specifying unit 12 specifies a plurality of "shelf-level image regions" respectively corresponding to the plurality of shelf levels in the captured image. The shelf-level image region corresponding to one shelf level is, for example, an image region corresponding to the space between the shelf board of the one shelf level and the shelf board of the shelf level immediately above the one shelf level.
[0016] The determination unit 13 determines the posture in which an article is placed on each shelf level based on the image of a single article detected in each shelf-level image region. The postures in which an article is placed on a shelf level include, for example, a standing posture (hereinafter sometimes referred to as the "first posture") and a lying-on-side posture (hereinafter sometimes referred to as the "second posture").
[0017] [[ID=३३]] Based on the "position in which items are placed on the shelf" determined by the judgment unit 13, the determination unit 15 determines the "model to use" from among multiple models when each shelf image region is set as the "image region to be processed" by the detection unit 14. Each of the multiple models is a model for detecting the image region of a group of items. Furthermore, the multiple models have different detection characteristics from each other.
[0018] The detection unit 14 detects the "item group image area" corresponding to the image of the item group in the "image area to be processed" by applying the "usage model" to the "image area to be processed". The detection unit 14 designates each of the multiple shelf image areas identified by the identification unit 12 as the "image area to be processed".
[0019] <Example of detection device operation> An example of the processing operation of the detection device 10 having the above configuration will now be described. Figure 2 is a flowchart showing an example of the processing operation of the detection device in the first embodiment.
[0020] The detection unit 11 detects individual object images in the captured image (step S101).
[0021] The identification unit 12 identifies multiple shelf image regions in the captured image that correspond to multiple shelf levels (step S102).
[0022] The determination unit 13 determines the orientation in which an item is placed on the shelf corresponding to each shelf image region, based on the individual item image detected in each shelf image region (step S103).
[0023] Based on the determined "position in which the item is placed on the shelf," the determination unit 15 determines which model to use when each shelf image area is used as the image area to be processed, from among multiple models (step S104).
[0024] The detection unit 14 detects the image region of a group of items in the image region to be processed by applying the model used to the image region to be processed (step S105).
[0025] As described above, according to the first embodiment, the detection device 10 uses multiple models with different detection characteristics to detect the object group image region in the captured image. This allows one model to compensate for the detection of objects that another model is not good at detecting, thereby improving the detection accuracy of the object group image region.
[0026] Furthermore, in the detection device 10, the determination unit 15 determines the model to be used when each shelf image area is the image area to be processed, based on the orientation in which the items are placed on the shelf corresponding to each shelf image area, from among multiple models. Each of the multiple models is a model for detecting the item group image area. Also, the multiple models have different detection characteristics from each other. The detection unit 14 detects the item group image area in the image area to be processed by applying the model to the image area to be processed.
[0027] The configuration of this detection device 10 allows a model corresponding to the arrangement of items on each shelf to be applied to the shelf image area, thereby improving the detection accuracy of the item group image area.
[0028] <Second Embodiment> The second embodiment relates to an embodiment that further elaborates on the content of the first embodiment.
[0029] Figure 3 is a block diagram showing an example of a detection device in the second embodiment. In Figure 3, the detection device 20 includes a detection unit (first detection unit) 21, a specification unit (first specification unit) 22, a determination unit 23, a detection unit (second detection unit) 24, a decision unit 25, a specification unit (second specification unit) 26, and an average value calculation unit 27. The detection device 20 acquires a captured image, similar to the detection device 10 in the first embodiment. This captured image is, for example, an image of a shelving unit. The following explanation assumes that the captured image is an image of a shelving unit. This shelving unit has multiple shelves.
[0030] The detection unit (first detection unit) 21 detects "single item images" of individual items in the captured image, similar to the detection unit 11 in the first embodiment.
[0031] Figure 4A is a diagram showing an example of a shelf image. Figure 4A shows a shelf image of a shelf displaying bottled beverages. The shelf image in Figure 4A has four shelves. On the top three shelves, the bottled beverages are arranged upright. On the bottom shelf, the bottled beverages are arranged horizontally and stacked. The space above the bottles is narrow on the top three shelves, while the space above the bottles is wide on the bottom shelf. As a result, in the images of the top three shelves, most of the bottles at the back are hidden by the bottles at the front, while in the image of the bottom shelf, the bottles at the back are also visible.
[0032] For example, in Figure 4A, each image region enclosed by the rectangular frame BB corresponds to an "individual item image." This rectangular frame may also be a so-called bounding box.
[0033] Returning to the explanation of Figure 3, the identification unit 22 (first identification unit) identifies multiple "shelf tier image regions" in the captured image, each corresponding to a plurality of shelf tiers, similar to the identification unit 12 in the first embodiment. For example, the identification unit 22 may identify lines corresponding to the surface of an item in contact with the shelf board of the item shelf in an image of an item alone. The identification unit 22 may then identify multiple shelf tier image regions by dividing the captured image using the identified lines. That is, the identification unit 22 may identify an image region sandwiched between two adjacent lines as a "shelf tier image region".
[0034] Alternatively, for example, the identification unit 22 may directly identify the front image of the shelf board by pattern matching or the like. This front image of the shelf board can also be identified as lines corresponding to the surface of the PET bottle in contact with the shelf board. The identification unit 22 may then identify the image region enclosed by two adjacent lines as the "shelf tier image region". In Figure 4A, the image regions SA11, SA12, SA13, and SA14 enclosed by dashed lines are examples of shelf tier image regions.
[0035] The average value calculation unit 27 calculates the average length of at least one individual item image in each shelf image region, corresponding to the height direction of the item shelf. In other words, in the example in Figure 4A, the average values for the top three shelves tend to be high because the PET bottles are placed upright on these shelves. On the other hand, the average value for the bottom shelf tends to be low because the PET bottles are placed horizontally.
[0036] The identification unit (second identification unit) 26 identifies the length of each shelf image region corresponding to the height direction of the goods shelf as the height (α) of each shelf image region.
[0037] Similar to the determination unit 13 in the first embodiment, the determination unit 23 determines the orientation in which an item is placed on the shelf corresponding to each shelf image region, based on the individual item image detected in each shelf image region.
[0038] For example, the determination unit 23 calculates a "reference value" for each shelf image region by multiplying the height (α) of each shelf image region by a predetermined ratio (for example, 0.7). The determination unit 23 then compares the calculated "reference value" for each shelf image region with the average value of the length of the individual item image corresponding to each shelf image region, calculated by the average value calculation unit 27, to determine the orientation in which the item is placed on the shelf corresponding to each shelf image region. For example, if the average value for the target shelf image region is greater than or equal to the reference value for the target shelf image region, the determination unit 23 may determine that the item is placed in an upright position (i.e., first orientation) on the shelf corresponding to the target shelf image region. On the other hand, if the average value for the target shelf image region is less than the reference value for the target shelf image region, the determination unit 23 may determine that the item is placed in a lying-down position (i.e., second orientation) on the shelf corresponding to the target shelf image region.
[0039] Similar to the determination unit 15 in the first embodiment, the determination unit 25 determines the "model to use" from among multiple models, based on the "position in which the item is placed on the shelf" determined by the judgment unit 23, when each shelf image area is set as the "image area to be processed" by the detection unit 24.
[0040] For example, the determination unit 25 determines the "second model" as the "model to use" for shelf image regions corresponding to shelf rows where the orientation in which the item is placed is determined to be upright. On the other hand, the determination unit 25 determines the "first model" as the "model to use" for shelf image regions corresponding to shelf rows where the orientation in which the item is placed is determined to be lying on its side.
[0041] Here, we will explain the first and second models.
[0042] The "second model" is a model that identifies the image region corresponding to the image of the group of items placed in the foreground of the target image (hereinafter sometimes referred to as the "second image region" or "second item group image region"). Note that, hereinafter, the "image of the group of items" may be referred to as the "item group image."
[0043] For example, the second model may be a pre-trained model that has been trained using training data that includes the following images. - An image showing the entirety of one side of an object. - A shelf image in which the image region corresponding to the item group image of the item group arranged in the front row of each shelf is designated as the "item group image region".
[0044] In addition, for items arranged in the front row of a shelf, typically a large portion of one side of the item (for example, more than half of one side) is visible in the shelf image.
[0045] When training is performed using this type of training data, the second model can accurately detect image regions corresponding to groups of items located at the front of each shelf, but it may not be able to accurately detect image regions corresponding to groups of items located behind the front groups of items, where a large portion of one side is hidden.
[0046] Furthermore, the "first model" described above is a model that identifies an image region (hereinafter sometimes referred to as the "first image region" or "first group of items image region") based on the space primarily occupied by a group of items in the target image (hereinafter sometimes referred to as the "mainly occupied space of items").
[0047] For example, the first model may be a pre-trained model that has been trained using training data including the following images. - An image showing the entirety of one side of an object. - A shelf image in which the image region corresponding to the item group image of the item group arranged in the front row of each shelf is designated as the "item group image region". - A shelf image in which the image area corresponding to the "image equivalent to the true background (hereinafter sometimes referred to as the "true background image")" is designated as the "true background image area".
[0048] Here, the "true background image area" includes an image area corresponding to the image of the back panel, side panel, or shelf of the shelving unit that is not obscured by the shadows of the items in the shelving unit image.
[0049] Through training using such training data, the first model has the characteristic of being able to accurately detect image regions that are likely to be background images (for example, image regions corresponding to large empty spaces where no items are placed) and image regions corresponding to the above-mentioned spaces occupied by items. However, because the first model learns conflicting information such as "item group image regions" and "true background image regions," for regions that cannot be designated as either, there is a possibility that regions close to the "item group image region" will be detected as "item group image regions," and regions close to the "true background image region" will be detected as "true background image regions." Therefore, there is a possibility that an image region corresponding to a narrow empty space sandwiched between two "item group image regions" will be detected as part of the item group image region. This is because it is thought that in many cases, sufficient information to detect that an image is a background image cannot be obtained from an image corresponding to a narrow empty space.
[0050] Furthermore, unlike the training data for the first model, the training data for the second model described above does not include any shelf images with a specified "true background image region," or if it does, the number of such images is small.
[0051] The detection unit (second detection unit) 24, similar to the detection unit 14 in the first embodiment, detects the "item group image region" corresponding to the image of the item group in the "image region to be processed" by applying the "usage model" to the "image region to be processed".
[0052] For example, the detection unit 24 detects a "second image region" by applying the second model to the shelf image region corresponding to the shelf where it is determined that the item is placed in an upright position. The detection unit 24 also detects a "first image region" by applying the first model to the shelf image region corresponding to the shelf where it is determined that the item is placed in a lying-down position.
[0053] Figure 4B shows an example of the second image region. The result of applying the second model to the item shelf image in Figure 4A is shown in Figure 4B. In Figure 4B, the shaded area corresponds to the second image region. In reality, the second model is not applied to the shelf image region SA14 where the items are placed lying on their sides, but for reference, the result of applying the second model to the shelf image region SA14 is also shown in Figure 4B.
[0054] As can be seen in Figure 4B, the second model accurately detects the image region corresponding to the group of PET bottles in the images of the top three shelves. As a result, the second model can accurately detect even image regions corresponding to narrow empty spaces, such as those sandwiched between two image regions of items.
[0055] On the other hand, as can be seen in Figure 4B, in the image of the bottom shelf, the image region of the item group corresponding to the group of PET bottles located at the front can be detected, but the image region of the item group corresponding to the group of PET bottles located at the back cannot be detected.
[0056] Figure 4C shows an example of the first image region. In Figure 4C, the shaded area in shelf image region SA14 corresponds to the first image region. In reality, the first model is not applied to shelf image regions SA11, SA12, and SA13, where the items are placed in an upright position. However, for reference, Figure 4C also shows the results when the second model is applied to shelf image regions SA11, SA12, and SA13.
[0057] As can be seen in Figure 4C, Model 1 accurately detects not only the item group image region corresponding to the group of PET bottles located at the front, but also the item group image region corresponding to the group of PET bottles located at the back, within the shelf image region SA14 (i.e., the image of the bottom shelf). However, as can be seen in Figures 4B and 4C, Model 1 may mistakenly detect the image region SP1, which corresponds to a narrow empty space sandwiched between two item group image regions, as part of the item group image region.
[0058] As described above, according to the second embodiment, the determination unit 25 of the detection device 20 determines the "second model" as the "used model" for shelf image regions corresponding to shelf rows where the orientation in which the items are placed is determined to be upright. The determination unit 25 also determines the "first model" as the "used model" for shelf image regions corresponding to shelf rows where the orientation in which the items are placed is determined to be lying on their sides. The second model is a model that identifies the second image region corresponding to the image of the group of items placed on the front side of the target image. The first model is a model that identifies the first image region based on the main space occupied by the items in the target image.
[0059] The configuration of this detection device 20 makes it possible to improve the detection accuracy of the item group image region. That is, for example, in the shelf image region corresponding to a shelf where it is determined that the orientation in which the items are placed is horizontal, there is a high probability that items located further back are also captured in the image. When the second model is applied to such a shelf image region, there is a possibility that the image region corresponding to items located further back may not be detected accurately. On the other hand, when the first model is applied to such a shelf image region, there is a high probability that the image region corresponding to items located further back will be detected accurately. In contrast, in the shelf image region where it is determined that the orientation in which the items are placed is upright, there is a high probability that items located further back will hardly be captured in the image. When the second model is applied to such a shelf image region, there is a high probability that the image region corresponding to the item group located at the front will be detected accurately. Therefore, since one model can compensate for the detection of targets that the other model is not good at detecting, the detection accuracy of the item group image region can be improved.
[0060] <Modified form of the second embodiment> In the above explanation, the determination unit 23 determined that if the average value for the target shelf image region is equal to or greater than the reference value for the target shelf image region, the orientation in which the item is placed on the shelf corresponding to the target shelf image region is upright (i.e., first orientation). Furthermore, it was explained that if the average value corresponding to the target shelf image region is less than the reference value for the target shelf image region, the determination unit 23 determined that the orientation in which the item is placed on the shelf corresponding to the target shelf image region is lying on its side (i.e., second orientation). However, this disclosure is not limited thereto.
[0061] For example, the determination unit 23 may determine the orientation in which an item is placed on the shelf corresponding to each shelf image region based on the ratio of the vertical and horizontal lengths of the individual item image detected in each shelf image region. For example, if the height direction of the item shelf is defined as vertical and the direction perpendicular to it as horizontal, the vertical length of the individual item image corresponding to a PET bottle in an upright position is longer than the horizontal length, so the value of vertical length / horizontal length is greater than 1. On the other hand, the vertical length of the individual item image corresponding to a PET bottle lying on its side is shorter than the horizontal length, so the value of vertical length / horizontal length is less than 1. Therefore, the orientation in which an item is placed on the shelf corresponding to each shelf image region can be determined based on the ratio of the vertical and horizontal lengths of the individual item image detected in each shelf image region.
[0062] <Third Embodiment> The third embodiment relates to the identification of vacant spaces.
[0063] Figure 5 is a block diagram showing an example of a detection device in the third embodiment. In Figure 5, the detection device 30 includes a detection unit (first detection unit) 11, a specification unit (first specification unit) 12, a determination unit 13, a detection unit (second detection unit) 14, a decision unit 15, an integration unit 31, and a space specification unit 32.
[0064] The integration unit 31 integrates the first image region in the processing target image region detected by the detection unit 14 and the second image region in the processing target image region detected by the detection unit 11 to obtain an "integrated image". For example, in the case of Figures 4B and 4C above, the second image regions in the shelf image regions SA11, SA12, and SA13 detected by the second model and the first image region in the shelf image region SA14 detected by the first model are integrated to form an "integrated image". Figure 6 shows an example of an integrated image.
[0065] The space identification unit 32 identifies empty spaces on each shelf based on the integrated image. For example, the space identification unit 32 may identify empty spaces by subtracting the integrated image from the shelf image area. In Figure 6, for example, the areas enclosed by rectangular frames (spaces SP1, SP2, SP3, SP4) correspond to empty spaces.
[0066] As described above, the detection device 30 in the third embodiment identifies empty spaces based on an integrated image obtained by integrating the image regions obtained by applying the first model and the second model, which are the shelf image regions for which each model excels, so that empty spaces can be identified with high accuracy.
[0067] Although this description assumes that the integration unit 31 and the space identification unit 32 are applied to the detection device 10 of the first embodiment, this disclosure is not limited thereto. The integration unit 31 and the space identification unit 32 may also be applied to the detection device 20 of the second embodiment.
[0068] <Other Embodiments> Figure 7 shows an example of the hardware configuration of a detection device. In Figure 7, the detection device 100 has a processor 101 and a memory 102. The processor 101 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 101 may include multiple processors. The memory 102 is composed of a combination of volatile memory and non-volatile memory. The memory 102 may include storage located away from the processor 101. In this case, the processor 101 may access the memory 102 via an I / O interface that is not shown.
[0069] The detection devices 10, 20, and 30 of the first to third embodiments may each have the hardware configuration shown in Figure 7. The detection units 11 and 21, the identification units 12 and 22, the determination units 13 and 23, the detection units 14 and 24, the determination units 15 and 25, the identification unit 26, the average value calculation unit 27, the integration unit 31, and the space identification unit 32 of the detection devices 10, 20, and 30 of the first to third embodiments may be realized by the processor 101 reading and executing a program stored in the memory 102. The program can be stored using various types of non-transitory computer-readable media and supplied to the detection devices 10, 20, and 30. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives) and magneto-optical recording media (e.g., magneto-optical disks). Furthermore, examples of non-transitory computer-readable media include CD-ROMs (Read Only Memory), CD-Rs, and CD-R / Ws. Furthermore, examples of non-transitory computer-readable media include semiconductor memory. Semiconductor memory includes, for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory). Programs may also be supplied to detection devices 10, 20, and 30 by various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can supply programs to detection devices 10, 20, and 30 via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.
[0070] Although the present invention has been described above with reference to embodiments, the present invention is not limited thereto. Various modifications to the structure and details of the present invention can be made that are understandable to those skilled in the art within the scope of the invention.
[0071] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) A first detection unit detects individual item images in a captured image of a shelf, A first identification unit identifies multiple shelf image regions in the captured image that correspond to multiple shelf levels of the item shelf, A determination unit determines the orientation in which an item is placed on the shelf corresponding to each shelf image region, based on the individual item image detected in each shelf image region. Based on the orientation in which the items are placed, a determination unit determines which model to use when each shelf image region is designated as the image region to be processed, from among a plurality of models, each of which is a model for detecting the image region of a group of items and which has different detection characteristics from one another. A second detection unit detects an item group image region corresponding to an image of an item group in the item group in the item group in the item group by applying the usage model to the item group image region to be processed, A detection device equipped with the following. (Note 2) The aforementioned plurality of models include a first model that identifies a first item group image region based on the item-dominant occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image region corresponding to the image of the item group arranged in the foreground in the target image. The aforementioned determination unit, For shelf image regions corresponding to shelf tiers where the orientation in which the item is placed is determined to be upright, the second model is determined to be the model used. For shelf image regions corresponding to shelf tiers where the orientation in which the item is placed is determined to be lying on its side, the first model is determined to be the model used. The detection device described in Appendix 1. (Note 3) A second identification unit identifies the length of each shelf image region corresponding to the height direction of the item shelf as the height of each shelf image region, An average value calculation unit calculates the average value of the length corresponding to the height direction of at least one individual item image detected by the first detection unit in each shelf image region, Furthermore, it is equipped with, The determination unit, If the average value for the target shelf image region is greater than or equal to the value obtained by multiplying the height of the target shelf image region by a predetermined percentage, then it is determined that the orientation in which the item is placed on the shelf corresponding to the target shelf image region is upright. If the average value corresponding to the target shelf image region is less than the value obtained by multiplying the height of the shelf corresponding to the target shelf image region by the predetermined percentage, it is determined that the orientation in which the item is placed on the shelf corresponding to the target shelf image region is lying on its side. The detection device described in Appendix 1. (Note 4) The determination unit determines the orientation in which the item is placed on the shelf corresponding to each shelf image region, based on the ratio of the length and width of the individual item image detected in each shelf image region. The detection device described in Appendix 1. (Note 5) The first identification unit identifies lines in the individual image of the item that are in contact with the shelf board of the item shelf, and identifies the plurality of shelf image regions by dividing the captured image by the identified lines. The detection device described in Appendix 1. (Note 6) The detection device according to Appendix 2, further comprising an integration unit that integrates the first item group image region obtained by applying the first model to the image region to be processed and the second item group image region obtained by applying the second model to the image region to be processed to obtain an integrated image. (Note 7) The system further includes a space identification unit that identifies empty spaces on each shelf where no items are placed, based on the aforementioned integrated image. The detection device described in Appendix 6. (Note 8) The process involves detecting individual item images within captured images of shelves, and In the aforementioned captured image, a plurality of shelf image regions corresponding to each of the plurality of shelf levels of the item shelf are identified, Based on the individual item images detected in each shelf image region, the orientation in which the item is placed on the shelf corresponding to each shelf image region is determined. Based on the orientation in which the items are placed, the model to be used when each shelf image region is the image region to be processed is determined from among multiple models, each of which is a model for detecting the image region of the group of items and has different detection characteristics from one another. By applying the aforementioned model to the image region to be processed, the object group image region corresponding to the image of the object group in the image region to be processed is detected. A detection method that includes [this]. (Note 9) The aforementioned plurality of models include a first model that identifies a first item group image region based on the item-dominant occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image region corresponding to the image of the item group arranged in the foreground in the target image. The above decision is, For the shelf image region corresponding to the shelf where the item is placed is determined to be in an upright position, the second model is determined to be the model used. For shelf image regions corresponding to shelf tiers where the orientation in which the item is placed is determined to be lying on its side, the first model is determined to be the model used. including, The detection method described in Appendix 8. (Note 10) The process involves detecting individual item images within captured images of shelves, and In the aforementioned captured image, a plurality of shelf image regions corresponding to each of the plurality of shelf levels of the item shelf are identified, Based on the individual item images detected in each shelf image region, the orientation in which the item is placed on the shelf corresponding to each shelf image region is determined. Based on the orientation in which the items are placed, the model to be used when each shelf image region is the image region to be processed is determined from among multiple models, each of which is a model for detecting the image region of the group of items and has different detection characteristics from one another. By applying the aforementioned model to the image region to be processed, the object group image region corresponding to the image of the object group in the image region to be processed is detected. A non-temporary, computer-readable medium containing a program that causes a detection device to perform a process including [specific actions]. (Note 11) The aforementioned plurality of models include a first model that identifies a first item group image region based on the item-dominant occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image region corresponding to the image of the item group arranged in the foreground in the target image. The above decision is, For the shelf image region corresponding to the shelf where the item is placed is determined to be in an upright position, the second model is determined to be the model used. For shelf image regions corresponding to shelf tiers where the orientation in which the item is placed is determined to be lying on its side, the first model is determined to be the model used. including, Non-temporary computer-readable media as described in Appendix 10. [Explanation of symbols]
[0072] 10 Detection device 11. Detection Unit (First Detection Unit) 12 Specific part (1st specific part) 13 Judgment section 14. Detection Unit (Second Detection Unit) 15. Decision Section 20 Detection device 21 Detection Unit (First Detection Unit) 22 Specific Section (1st Specific Section) 23 Judgment section 24 Detection unit (second detection unit) 25 Decision Section 26 Specific Section (Second Specific Section) 27. Average value calculation section 30 Detection device 31. Integration Department 32 Space Identification Section
Claims
1. A first detection unit detects individual item images in a captured image of a shelf, A first identification unit identifies multiple shelf image regions in the captured image that correspond to multiple shelf levels of the item shelf, A determination unit determines the orientation in which an item is placed on the shelf corresponding to each shelf image region, based on the individual item image detected in each shelf image region. Based on the orientation in which the items are placed, a determination unit determines which model to use when each shelf image region is designated as the image region to be processed, from among a plurality of models, each of which is a model for detecting the image region of a group of items and which has different detection characteristics from one another. A second detection unit detects an item group image region corresponding to an image of an item group in the item group in the item group in the item group by applying the usage model to the item group image region to be processed, A detection device equipped with the following.
2. The plurality of models include a first model that identifies a first item group image region based on the item-dominant occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image region corresponding to the image of the item group arranged in the foreground in the target image. The aforementioned determination unit, For the shelf image region corresponding to the shelf where the item is determined to be placed in an upright position, the second model is determined to be the model used. For shelf image regions corresponding to shelf tiers where the orientation in which the item is placed is determined to be lying on its side, the first model is determined to be the model used. The detection device according to claim 1.
3. A second identification unit identifies the length of each shelf image region corresponding to the height direction of the item shelf as the height of each shelf image region, An average value calculation unit calculates the average value of the length corresponding to the height direction of at least one individual item image detected by the first detection unit in each shelf image region, Furthermore, it is equipped with, The determination unit, If the average value for the target shelf image region is greater than or equal to the value obtained by multiplying the height of the target shelf image region by a predetermined percentage, then it is determined that the orientation in which the item is placed on the shelf corresponding to the target shelf image region is upright. If the average value corresponding to the target shelf image region is less than the value obtained by multiplying the height of the shelf corresponding to the target shelf image region by the predetermined percentage, it is determined that the orientation in which the item is placed on the shelf corresponding to the target shelf image region is lying on its side. The detection device according to claim 1.
4. The determination unit determines the orientation in which the item is placed on the shelf corresponding to each shelf image region, based on the ratio of the length and width of the individual item image detected in each shelf image region. The detection device according to claim 1.
5. The first identifying unit identifies lines in the individual image of the article that are in contact with the shelf board of the article shelf, and identifies the plurality of shelf image regions by dividing the captured image by the identified lines. The detection device according to claim 1.
6. The detection device according to claim 2, further comprising an integration unit that integrates the first item group image region obtained by applying the first model to the image region to be processed and the second item group image region obtained by applying the second model to the image region to be processed to obtain an integrated image.
7. The system further includes a space identification unit that identifies empty spaces on each shelf where no items are placed, based on the aforementioned integrated image. The detection device according to claim 6.
8. The process involves detecting individual item images within captured images of shelves, and In the aforementioned captured image, a plurality of shelf image regions corresponding to each of the plurality of shelf levels of the item shelf are identified, Based on the individual item images detected in each shelf image region, the orientation in which the item is placed on the shelf corresponding to each shelf image region is determined. Based on the orientation in which the items are placed, the model to be used when each shelf image region is the image region to be processed is determined from among multiple models, each of which is a model for detecting the image region of the group of items and has different detection characteristics from one another. By applying the aforementioned model to the image region to be processed, the object group image region corresponding to the image of the object group in the image region to be processed is detected. A detection method that includes this.
9. The plurality of models include a first model that identifies a first item group image region based on the item-dominant occupied space that is mainly occupied by the item group in the target image, and a second model that identifies a second item group image region corresponding to the image of the item group arranged in the foreground in the target image. The above decision is, For the shelf image region corresponding to the shelf where the item is determined to be placed in an upright position, the second model is determined to be the model used. For shelf image regions corresponding to shelf tiers where the orientation in which the item is placed is determined to be lying on its side, the first model is determined to be the model used. including, The detection method according to claim 8.
10. The process involves detecting individual item images within captured images of shelves, and In the aforementioned captured image, a plurality of shelf image regions corresponding to each of the plurality of shelf levels of the item shelf are identified, Based on the individual item images detected in each shelf image region, the orientation in which the item is placed on the shelf corresponding to each shelf image region is determined. Based on the orientation in which the items are placed, the model to be used when each shelf image region is the image region to be processed is determined from among multiple models, each of which is a model for detecting the image region of the group of items and has different detection characteristics from one another. By applying the aforementioned model to the image region to be processed, the object group image region corresponding to the image of the object group in the image region to be processed is detected. A program that causes a detection device to perform a process that includes [a specific action].