Object detection method and object detection device
The use of non-rectangular detection frames aligned with the object's outline addresses image deformation issues, enhancing detection accuracy and speed in object recognition systems.
Patent Information
- Application Number
- JP2023579455
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-06-25
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2041-06-25
AI Technical Summary
Existing object detection methods using rectangular frames are prone to inaccuracies due to image deformation caused by varying photographing angles, leading to incomplete or incorrect detection of products on shelves.
The method employs a non-rectangular second detection frame set by comparing multiple non-rectangular shapes to the object's outline, ensuring the frame encompasses the entire object, improving detection accuracy.
Enhances object detection accuracy by preventing partial omission and false detections, especially for densely arranged products, by aligning the detection frame with the object's shape, thereby improving precision and speed.
Smart Images

Figure 0007591670000001 
Figure 0007591670000002 
Figure 0007591670000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an object detection method and an object detection device for detecting an object from an image. [Background technology]
[0002] With the advent of the smart retail era, information such as shelf location, number of products, and product placement locations has become important for retail companies. Also, understanding the trends of products on shelves is of great significance in determining sales activities and management policies. In order to understand the trends of products on shelves without manually visiting the shelves, it is necessary to detect and identify the products on the shelves and accurately understand the product display information on the shelves.
[0003] An example of a product detection method is disclosed in Patent Document 1. Patent Document 1 discloses a method of acquiring images of displayed products, detecting the products in the images, and classifying the products by cutting out the images of each product based on the spatial positional relationship of each product. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] China Patent Application Publication No. 110738123 Summary of the Invention [Problem to be solved by the invention]
[0005] In Patent Document 1, the image of each product is cut out using a rectangular frame such as a rectangle or a square. However, depending on the shooting angle when acquiring the product image, the outer shape of the product in the image may be deformed and may no longer be rectangular. In this case, the cut image may not include part of the product to be detected, or may include an image other than the product to be detected, reducing the accuracy of product detection.
[0006] The present invention has been made to solve the above-mentioned problems, and has an object to provide an object detection method and an object detection device that can improve the object detection accuracy. [Means for solving the problem]
[0007] The object detection method according to the present invention includes an image acquisition step of acquiring an image including an object, a first detection step of detecting an object in the image using a rectangular first detection frame, a detection frame setting step of setting a non-rectangular second detection frame corresponding to the detected object, and a second detection step of detecting the object using the second detection frame. the detection frame setting step includes a step of detecting an outline of an object, and a step of comparing a plurality of non-rectangular detection frames with the outline of the object, and setting a detection frame including the entire outline of the object as a second detection frame. nothing. The object detection device according to the present invention includes an image acquisition unit that acquires an image including an object, a first detection unit that detects an object in the image using a rectangular first detection frame, a detection frame setting unit that sets a non-rectangular second detection frame corresponding to the detected object, and a second detection unit that detects the object using the second detection frame. The detection frame setting unit detects the contour of the object, compares the plurality of non-rectangular detection frames with the contour of the object, and sets the detection frame that includes the entire contour of the object as a second detection frame. do. Effect of the Invention
[0008] According to the object detection method and object detection device of the present invention, the object detection accuracy can be improved by detecting an object using a non-rectangular second detection frame that corresponds to the detected object. [Brief description of the drawings]
[0009] [Figure 1] 1 is a schematic configuration diagram of an object information acquisition system according to a first embodiment. [Diagram 2] 1 is a control block diagram of an object information acquisition system according to a first embodiment. [Diagram 3] 4 is a flowchart of an object information acquisition process according to the first embodiment. [Figure 4] 1 is an example of a front image of a shelf captured by the image capture device 2. [Diagram 5] 4 is a flowchart of an object detection process according to the first embodiment. [Figure 6] 4 is an example of a detection result by a first detection unit according to the first embodiment. [Figure 7] 4 is an example of a detection result by a second detection unit according to the first embodiment. [Figure 8] 13 is a flowchart of an object information acquisition process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an object detection method and an object detection device according to an embodiment of the present invention will be described with reference to the drawings. In each drawing, the same or corresponding parts are denoted by the same reference numerals, and their description will be omitted or simplified as appropriate. Furthermore, the shape, size, arrangement, etc. of the configurations shown in each drawing may be changed as appropriate within the scope of the present invention.
[0011] Embodiment 1 FIG. 1 is a schematic diagram of an object information acquisition system 100 according to a first embodiment. The object information acquisition system 100 according to the present embodiment is used in a retail store such as a supermarket, and automatically detects and identifies objects P, which are products stored on a shelf S in the store, and acquires information on the objects P stored on the shelf S. The object information acquisition system 100 includes a processing device 1 and a photographing device 2. The processing device 1 is a PC equipped with a CPU and a memory, or a server on the cloud. The photographing device 2 is a camera installed on the ceiling or wall of the store and captures an image of the front of the shelf S. The processing device 1 and the photographing device 2 are connected to each other so as to be able to communicate with each other via wire or wirelessly. An image captured by the photographing device 2 is transmitted to the processing device 1.
[0012] 2 is a control block diagram of the object information acquisition system 100 according to the first embodiment. The processing device 1 has an object detection unit 10 that detects an object P from an image, an object identification unit 20 that identifies the detected object P, and a storage unit 30. The object detection unit 10 and the object identification unit 20 are functional units that are realized by a CPU executing a program. Alternatively, the object detection unit 10 and the object identification unit 20 may be realized by a dedicated processing circuit.
[0013] The object detection unit 10 has an image acquisition unit 11, a first detection unit 12, a detection frame setting unit 13, and a second detection unit 14. The image acquisition unit 11 acquires an image captured by the imaging device 2 and transmits it to the first detection unit 12. The first detection unit 12 detects an object P in the acquired image using an algorithm such as SSD (Single Shot Multibox Detector) using deep learning. The first detection unit 12 detects the object P using a rectangular first detection frame F1.
[0014] The detection frame setting unit 13 sets a non-rectangular second detection frame F2 corresponding to the object P detected by the first detection unit 12. The second detection unit 14 detects the object P using the second detection frame F2 set by the detection frame setting unit 13, and transmits the detection result to the object identification unit 20.
[0015] The object identification unit 20 identifies the object P included in the image detected by the second detection unit 14 of the object detection unit 10, based on the shelf information and product information. The object identification unit 20 identifies the type and product name of the object P by an algorithm using known machine learning.
[0016] The storage unit 30 is, for example, a volatile or non-volatile memory such as a RAM, a ROM, or a flash memory. The storage unit 30 stores programs executed by the object detection unit 10 and the object identification unit 20, various parameters used in the programs, shelf information, product information, detection frame data, detection history, and the like. The shelf information includes the location of each shelf S in the store, the category of products stored on each shelf S, the number of tiers and size of each shelf S, and the number of detection frames on each shelf S. The product information includes identification information such as the type and name of the product. The detection frame data is data on multiple non-rectangular detection frames that are candidates for the second detection frame F2 set by the detection frame setting unit 13.
[0017] FIG. 3 is a flowchart of the object information acquisition process according to the first embodiment. This process is periodically executed by the processing device 1. First, the system is initialized (S1). Here, initial values are set for each parameter of the object information acquisition process. The parameters include the number of detection frames, the maximum number of detection frames that can be detected per shelf level, the maximum number of shelf levels, and the type of detection frames. Then, an image of the front of the shelf S is captured by the imaging device 2 and acquired by the image acquisition unit 11 of the processing device 1 (S2).
[0018] Then, the object detection unit 10 executes an object detection process based on the acquired image (S3). As a result, a plurality of objects P contained in the image are detected. When an object P is detected by the object detection process, the object identification unit 20 executes an object identification process (S4). As a result, the detected object P is identified, and information on the object P stored on the shelf S can be acquired. The acquired information on the object P is transmitted to a management server or the like, and used for grasping sales data, product management, or the like.
[0019] Next, the object detection process of this embodiment will be described. Fig. 4 is an example of a front image of a shelf S photographed by the photographing device 2. In a general object detection algorithm, object detection is performed using a rectangular or square detection frame. However, when the photographing device 2 is installed above a ceiling or a wall and an image of the shelf S is photographed from above, the outer shape of the object P in the image is deformed from a rectangle, as shown in Fig. 4. Therefore, in the object detection process of this embodiment, the detection frame for detecting the object is set to a detection frame suitable for the deformation of the image, and then the image is detected.
[0020] Fig. 5 is a flowchart of the object detection process according to the first embodiment. In this process, first detection is performed by the first detection unit 12 based on the acquired image (S31). Here, an object P is detected using a rectangular first detection frame F1. Fig. 6 is an example of a detection result by the first detection unit 12 according to the first embodiment.
[0021] Next, the detection frame setting unit 13 applies the multiple non-rectangular detection frames stored in the storage unit 30 to each detected object P, and obtains the reliability of each detection frame (S32). The shapes of the multiple non-rectangular detection frames are parallelograms, trapezoids, circles, ellipses, etc. Also, multiple sizes of frames are included for each shape. Then, the center coordinates of the non-rectangular detection frame are matched with the center coordinates of the rectangular first detection frame F1 that includes the object P detected by the first detection, and the reliability of the non-rectangular detection frame is obtained. The reliability of each detection frame is obtained based on the size of the area of the object P within the detection frame (the size of the area of the common area between the object P and the non-rectangular detection frame) or the size of the area other than the object P within the detection frame (the size of the area of the non-common area between the object P and the non-rectangular detection frame) when multiple non-rectangular detection frames are applied to the object P. Specifically, the larger the area of the object P within the detection frame, the higher the reliability, and the smaller the area of the object P within the detection frame, the higher the reliability. In other words, the higher the ratio of the area of the object P to the area of the detection frame, the higher the reliability.
[0022] Next, the detection frame setting unit 13 filters the non-rectangular detection frames (S33). Here, among the non-rectangular detection frames, detection frames with a reliability lower than a preset threshold are excluded from the candidates. Then, the detection frame setting unit 13 acquires the center coordinates of the non-rectangular detection frames with a reliability equal to or higher than the threshold for each object P (S34). The detection frame setting unit 13 performs contour detection for each object P in the image using the acquired center coordinates, and acquires position information of the contour of the object P (S35). A known contour detection algorithm such as edge detection is used for the contour detection of the object P. Here, the contour detection of the object P is performed for each center coordinate of the non-rectangular detection frames. Note that, by filtering out detection frames with low reliability in step S33, it is possible to suppress the occurrence of an abnormality in which the center coordinates of the non-rectangular detection frame are located outside the object P and the contour of the object P cannot be detected.
[0023] The detection frame setting unit 13 compares the non-rectangular detection frame with the contour of the object P (S36). Then, the non-rectangular detection frame that includes the entire contour of the object P is set as the second detection frame F2 (S37). Steps S36 and S37 are performed for each object P, and the second detection frame F2 corresponding to each object P is set. The shape and position of the set second detection frame F2 are stored in the storage unit 30. Here, if there are multiple non-rectangular detection frames that include the entire contour of the object P, the detection frame with the highest reliability is set as the second detection frame F2. The reliability is calculated based on the ratio of the area of the object P to the area of the detection frame as described above. Also, if there are multiple detection frames with the highest reliability, any one of them is selected as the second detection frame F2. In this case, an overlap detection algorithm that calculates the overlapping area may be used, and if the ratio of the overlapping area between them is equal to or greater than a threshold, it may be determined that they are the same, and only one may be left. In this way, the optimal detection frame is selected for each object P.
[0024] Then, the second detection unit 14 performs a second detection (S38). In the second detection, the object P is detected using the set second detection frame F2. Fig. 7 shows an example of a detection result by the second detection unit 14 according to the first embodiment.
[0025] 7, by performing the object detection process of this embodiment, it is possible to detect object P in a second detection frame F2 that follows the outline of object P. As a result, even if object P in the image is deformed due to the influence of the shooting angle, it is possible to prevent a part of object P from going undetected and to prevent objects other than object P from being detected, thereby improving the detection accuracy of object P. In particular, for objects P that are densely arranged, such as products stored on a shelf S, using a detection frame that follows the outline of the object improves accuracy and detection speed.
[0026] In addition, in the object detection process, the following process may be performed to further improve the detection accuracy. As shown in FIG. 4, the deformation of the object P, that is, the inclination of the object P, differs depending on the position of the object P on the shelf S. For example, the inclination of the object P placed on the right side of the shelf S differs from that of the object P placed on the left side. Therefore, the detection frame setting unit 13 may cluster the multiple non-rectangular detection frames filtered in step S33 at the position of the shelf S of the object P, detect a detection frame having an inclination different from other detection frames in the same cluster as an error, and exclude it from the candidates. Alternatively, the detection frame setting unit 13 may estimate the inclination of the object P from the position information of the imaging device 2, detect a non-rectangular detection frame having an inclination different from the estimated inclination as an error, and exclude it from the candidates. Furthermore, the detection frame setting unit 13 may detect a non-rectangular detection frame larger than the size of the shelf as an error based on the shelf information, and exclude it from the candidates.
[0027] Embodiment 2 8 is a flowchart of an object information acquisition process according to embodiment 2. The configuration of object information acquisition system 100 in this embodiment is the same as that in embodiment 1.
[0028] In the object information acquisition process of this embodiment, initialization (S1) and image acquisition (S2) are performed as in the first embodiment. Then, the object detection unit 10 determines whether or not the current detection is a re-detection based on the detection history (S11). Re-detection refers to a case where object detection has been performed on the shelf S in the past and a second detection frame F2 for an object on the shelf S has been stored. If the current detection is not a re-detection, that is, if it is the first detection (S11: NO), then object detection process (S3) and object recognition process (S4) are performed as in the first embodiment.
[0029] On the other hand, if the current detection is a re-detection (S11: YES), a difference between the image captured by the photographing device 2 at the time of the previous detection and the image captured by the photographing device 2 this time is acquired (S12). Then, an object detection process (S3) and an object identification process (S4) are executed for the difference region. That is, in this embodiment, object P is detected and identified only for regions that have changed since the previous detection, and the information of object P in other regions is considered to be the same as the previous detection.
[0030] According to this embodiment, by performing object detection processing and object identification processing only on areas where a change has occurred, it is possible to reduce the processing load in the case of re-detection and improve the processing speed.
[0031] The above is a description of the embodiment, but the above embodiment can be modified and combined. For example, in the first embodiment, the detection frame setting unit 13 performs the process of steps S32 to S36 in FIG. 5 to set the second detection frame F2, but the present invention is not limited to this. For example, the detection frame setting unit 13 may estimate the inclination of the object P from the position information of the image capture device 2, and set a non-rectangular detection frame having the estimated inclination as the second detection frame F2. Alternatively, the detection frame setting unit 13 may set the detection frame with the highest reliability obtained in step S32 in FIG. 5 as the second detection frame F2. Alternatively, the detection frame setting unit 13 may detect the contour of the object P detected by the first detection unit 12, compare the contour of the object P with a plurality of non-rectangular detection frames stored in the storage unit 30, and set the detection frame including the contour of the object P as the second detection frame F2.
[0032] Furthermore, the object detection unit 10 may set the rectangular first detection frame F1 as one of the candidates for the second detection frame F2, and perform the processes of steps S32 to S36 in Fig. 5. If the first detection frame F1 includes the entire contour of the object P and has a higher reliability than a non-rectangular detection frame, the object detection unit 10 sets the first detection frame F1 as the second detection frame F2. Alternatively, the object detection unit 10 may set a rectangular detection frame of a different size from the first detection frame F1 as a candidate for the second detection frame F2, and perform the processes of steps S32 to S36 in Fig. 5.
[0033] Furthermore, while the above embodiment is directed to detecting a product object P from an image of a shelf S in a retail store, the present invention is not limited to this and can be applied to a method for detecting objects from an image containing multiple objects.
[0034] Furthermore, in the above embodiment, the processing device 1 is configured to have an object detection unit 10 and an object identification unit 20, but the object detection device having the object detection unit 10 and the object identification device having the object identification unit 20 may be configured as separate entities. [Explanation of symbols]
[0035] 1 processing device, 2 photographing device, 10 object detection unit, 11 image acquisition unit, 12 first detection unit, 13 detection frame setting unit, 14 second detection unit, 20 object identification unit, 30 memory unit, 100 object information acquisition system, F1 first detection frame, F2 second detection frame, P object, S shelf.
Claims
1. An image capturing step of capturing an image including the object; a first detection step of detecting the object in the image using a rectangular first detection frame; a detection frame setting step of setting a non-rectangular second detection frame corresponding to the detected object; a second detection step of detecting the object using the second detection frame, The detection frame setting step includes: detecting a contour of the object; comparing a plurality of non-rectangular detection frames with the contour of the object, and setting a detection frame that includes all of the contour of the object as the second detection frame.
2. An object detection method as described in claim 1, wherein, when there are multiple detection frames that include all of the contours of the object, the detection frame with the highest reliability is set as the second detection frame.
3. The detection frame setting step includes: applying the plurality of non-rectangular detection frames to the object detected in the first detection step; obtaining a reliability for each of the plurality of non-rectangular detection windows; The object detection method according to claim 1 , further comprising: excluding, from the candidates for the second detection window, any detection window whose reliability is less than a preset threshold value, from among the plurality of non-rectangular detection windows.
4. A detection method as described in claim 2 or 3, wherein the reliability is set higher as the ratio of the area of the object to the area of the detection frame becomes higher.
5. 4. The object detection method according to claim 3, wherein the detection frame setting step includes a step of clustering the non-rectangular detection frames applied to the object based on a position of the object, and excluding a detection frame having an inclination different from other detection frames in the same cluster from candidates for the second detection frame.
6. 4. The object detection method according to claim 3, wherein the detection frame setting step includes a step of estimating an inclination of the object from position information of an imaging device that captured the image, and excluding a non-rectangular detection frame having an inclination different from the estimated inclination from candidates for the second detection frame.
7. Further comprising a step of determining whether or not re-detection is performed, The object detection method according to any one of claims 1 to 6, wherein in the case of re-detection, the first detection step, the detection frame setting step, and the second detection step are performed for a difference area between a previous image and a current image.
8. The object detection method according to any one of claims 1 to 7, wherein the second detection frame is a parallelogram or a trapezoid.
9. an image acquisition unit for acquiring an image including an object; a first detection unit that detects the object in the image by using a rectangular first detection frame; a detection frame setting unit that sets a non-rectangular second detection frame corresponding to the detected object; a second detection unit that detects the object using the second detection frame, The detection frame setting unit is Detecting a contour of the object; An object detection device that compares a plurality of non-rectangular detection frames with the contour of the object, and sets a detection frame that includes the entire contour of the object as the second detection frame.
Citation Information
Patent Citations
Irregular-shaped object detection method based on trapezoidal convolution
CN110334752A
Method and device for identifying densely-displayed commodities
CN110738123A
Image detection apparatus, control program, and image detection method
JP2014191369A
Connected interactive content data creation, organization, distribution and analysis
US20200196015A1
