Illumination adjustment device, illumination adjustment method, and product recognition system

Abstract

To stabilize item recognition accuracy.SOLUTION: A lighting adjustment device according to one embodiment of the present disclosure includes a recognition unit that recognizes abnormal regions in a captured image, and an adjustment unit that adjusts the intensity of illumination light according to a shape and a type of the abnormal regions.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a lighting adjustment device, a lighting adjustment method, and a product recognition system. [Background technology]

[0002] In retail stores such as convenience stores and supermarkets, the introduction of so-called self-checkout terminals is increasing in order to shorten waiting times at the cash register. Some self-checkout terminals recognize products placed on a checkout counter (product placement section) by users such as customers by performing image processing on the captured image captured by an imaging means such as a camera.

[0003] When recognizing products through image processing in this way, fluctuations in the lighting around the self-checkout terminal have a significant impact on the accuracy of product recognition.

[0004] For example, Patent Document 1 discloses a technology for improving product recognition accuracy by evenly illuminating the entire surface of a specified area including an object so that when the shadow of an imaged object (product) appears in the captured image, the shadow of the object is eliminated or reduced. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-085068 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 may cause dynamic or localized abnormal areas in the captured image, such as whiteout caused by the reflection of lighting or unexpected reflection of the user's shadow, which may adversely affect the accuracy of product recognition.

[0007] Non-limiting embodiments of the present disclosure contribute to providing a lighting adjustment device, a lighting adjustment method, and a product recognition system that can stabilize product recognition accuracy. [Means for solving the problem]

[0008] An illumination adjustment device according to an embodiment of the present disclosure includes: A product area of ​​a product is recognized from the captured image, and the recognized product area is excluded from the difference between the captured image and the background image. In the captured image shadow A recognition unit that recognizes an area; When the shadow area is recognized, The intensity of the lighting , so that the intensity of the illumination light is stronger than the intensity of the illumination light irradiated to acquire the captured image. and an adjustment unit for adjusting the

[0009] A method for adjusting illumination according to an embodiment of the present disclosure includes: A product area of ​​a product is recognized from the captured image, and the recognized product area is excluded from the difference between the captured image and the background image. In the captured image shadow Recognize the area, When the shadow area is recognized, The intensity of the lighting , so that the intensity of the illumination light is stronger than the intensity of the illumination light irradiated to acquire the captured image. Adjust.

[0010] A commodity recognition system according to an embodiment of the present disclosure includes an illumination unit that irradiates an illumination light onto a predetermined area on which a commodity is placed, and an acquisition unit that captures and acquires an image of the predetermined area irradiated with the illumination light; A product area of ​​a product is recognized from the captured image, and the recognized product area is excluded from the difference between the captured image and the background image. In the captured image, shadow Recognize the area, When the shadow area is recognized, The intensity of the lighting , so that the intensity of the illumination light is stronger than the intensity of the illumination light irradiated to acquire the captured image. and a lighting adjustment device that adjusts the illumination intensity of the specified area, and the acquisition unit re-images and re-acquires an image of the specified area to which the illumination unit has irradiated or not irradiated the illumination light at the adjusted intensity, and the lighting adjustment device recognizes the product in the re-imaged and re-acquired image.

[0011] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0012] According to an embodiment of the present disclosure, in a captured image of a predetermined area where a commodity is placed, shadow areaSince the illumination light can be adjusted according to the situation, it is possible to stabilize the accuracy of product recognition.

[0013] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings. Such advantages and / or benefits may be provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of them may be provided to obtain one or more identical features. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an example of a product recognition system according to a first embodiment of the present disclosure. [Diagram 2] FIG. 1 is a functional block diagram illustrating an example of a product recognition system according to a first embodiment of the present disclosure. [Diagram 3] FIG. 13 is a diagram showing an example of a captured image obtained by capturing an image of an object (product) placed on the placement surface of the product placement unit using default irradiation light; [Figure 4] FIG. 4 is a diagram for explaining recognition of a product area in the captured image shown in FIG. 3; [Diagram 5] FIG. 4 is a diagram for explaining recognition of a shadow area in the captured image shown in FIG. 3 and calculation of shadow intensity; [Figure 6] FIG. 4 is a diagram for explaining recognition of a whiteout region in the captured image shown in FIG. 3; [Figure 7] 1 is a flowchart showing an example of an operation of the product recognition system according to the first embodiment of the present disclosure. [Figure 8] FIG. 4 is a diagram for explaining the output contents (brightness of the placement surface) of the illumination projection unit after the irradiation light is adjusted, using the captured image shown in FIG. [Figure 9] FIG. 13 is a diagram showing an example of a captured image obtained by capturing an image of an object (product) placed on the placement surface of the product placement section after the irradiation light is adjusted; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters or duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

[0016] It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0017] (Embodiment 1) <Product Recognition System Configuration> A product recognition system 10 according to a first embodiment of the present disclosure will be described with reference to FIGS.

[0018] FIG. 1 is a block diagram showing an example of a commodity recognition system 10 according to the first embodiment.

[0019] The product recognition system 10 includes an illumination device 101 , an imaging device 102 , an information processing device 103 , a display device 104 , and a product placement unit 105 .

[0020] The lighting device 101 may be, for example, a projector. The lighting device 101 irradiates (illumination) light onto a placement surface (a predetermined area) of a commodity placement section 105 on which an object such as a commodity 106 is placed, as indicated by an arrow A, under the control of an information processing device 103 (more specifically, a control device 111 described later).

[0021] The lighting device 101, which is a projector, can locally control the light irradiated onto the mounting surface of the commodity mounting section 105. Specifically, the lighting device 101, which is a projector, recognizes in advance the distance (height) between itself and the mounting surface of the commodity mounting section 105, and controls the amount of light partially (locally), thereby controlling the intensity (also called the strength) and the shape or pattern (distribution of the brightness of the light on the mounting surface of the commodity mounting section 105) of the light irradiated by the lighting device 101.

[0022] Here, the intensity of the irradiated light emitted by the lighting device 101 may be a preset stepped light amount, for example, step 0 (light off (zero light amount)), step 1 (default intensity), step 2 stronger than step 1, step 3 stronger than step 2, step 4 stronger than step 3, and step 5 stronger than step 4. Note that the intensity of the irradiated light emitted by the lighting device 101 essentially means the stepped brightness of the light on the placement surface of the product placement section 105.

[0023] The imaging device 102 may be an imaging element (camera) such as a CCD (Charge-Coupled Device). Under the control of the information processing device 103, the imaging device 102 captures an image of an object placed on the placement surface of the product placement section 105, and generates and acquires a two-dimensional color or monochrome captured image including an image of the object.

[0024] The information processing device 103 may be, for example, a computer. The information processing device 103 has a control device 111 such as a CPU (Central Processing Unit), a storage device 112 such as a memory or a hard disk, and a communication device 113 such as a network interface card. Although not shown, the information processing device 103 may have an input / output device that a computer generally has. The information processing device 103 controls the operations of the lighting device 101, the imaging device 102, and the display device 104, which are connected to each other by wire or wirelessly via the communication device 113.

[0025] The display device 104 may be, for example, a touch panel or an LCD (Liquid Crystal Display). The display device 104 displays information (such as a checkout button, details of purchased items, and total checkout amount) to the user under the control of the information processing device 103. The display device 104 may have an input device and may accept operations from the user as necessary.

[0026] Products are placed on the product placement unit 105 so that the user can recognize the products to be purchased. A material with a simple color scheme, such as a white plate, is used for the placement surface of the product placement unit 105 so that the products can be easily distinguished from the placement surface. The sides and top of the product placement unit 105 may be enclosed, so that the light from the lighting device 101 may be irradiated onto the placement surface of the product placement unit 105 from above while reducing the influence of ambient light.

[0027] The information processing device 103 (particularly, the illumination adjustment unit 204 described below) is an example of an illumination adjustment device according to the present disclosure.

[0028] In addition, there are no particular restrictions on the positional relationship between the lighting device 101 and the imaging device 102, as long as the shadow of a user standing in front of the product placement section 105 is not reliably reflected in the captured image acquired by the imaging device 102 using the light emitted by the lighting device 101.

[0029] In the first embodiment, a self-checkout terminal may be realized by integrating one information processing device 103, one lighting device 101, one imaging device 102, and one display device 104. Alternatively, one information processing device 103 may be configured to control the operations of a plurality of sets of lighting devices 101, imaging devices 102, and display devices 104.

[0030] FIG. 2 is a functional block diagram showing an example of the commodity recognition system 10 according to the first embodiment.

[0031] The commodity recognition system 10 includes a lighting projection unit 201, an image acquisition unit 202, an image storage unit 203, a lighting adjustment unit 204, a commodity recognition unit 205, a commodity recognition model storage unit 206, and a learning unit 207.

[0032] The illumination projection unit 201 corresponds to, for example, the illumination device 101 shown in FIG. 1. The illumination projection unit 201 irradiates illumination light onto the placement surface of the commodity placement unit 105. The illumination projection unit 201 first irradiates illumination light with the default intensity of the above-mentioned stage 1. Then, as described below, the illumination projection unit 201 irradiates illumination light with an adjusted light intensity according to the shape and type of the abnormal area described below, or does not irradiate illumination light, as necessary. In the first embodiment, it is assumed that the illumination projection unit 201 irradiates light according to the distance to the placement surface of the commodity placement unit 105, that is, irradiates light so as to have a desired brightness on the placement surface of the commodity placement unit 105.

[0033] The image acquisition unit 202 corresponds to, for example, the imaging device 102 shown in Fig. 1. The image acquisition unit 202 captures an image of an object or the like placed on the placement surface of the product placement unit 105 to generate and acquire a captured image. Then, the image acquisition unit 202 stores the acquired captured image in the image storage unit 203.

[0034] The illumination projection unit 201 and the image acquisition unit 202 share the coordinate system of the placement surface of the product placement unit 105, and therefore the position coordinate information of the captured image.

[0035] The illumination adjustment unit 204 determines and adjusts the illumination light (e.g., its intensity and shape) of the illumination projection unit 201 according to the shape and type of the abnormal area based on at least one of the shadow area and the blown-out highlight area (shadow area and blown-out highlight area are collectively called "abnormal area"; that is, types of abnormal area include shadow area and blown-out highlight area) in the captured image acquired by the image acquisition unit 202. Here, the "shadow area" refers to a dark area in the captured image that is caused as a result of an object, a person, etc. blocking the progress of light, and the "blown-out highlight area" refers to an area in the captured image where blown-out highlights have occurred, that is, an area where the gradation of a bright part has been lost and turned white. Note that blown-out highlights occur when, for example, the light receiving element is unable to recognize the gradation of brightness when the brightness exceeds the upper limit of the dynamic range of the light receiving element.

[0036] The illumination adjustment unit 204 has an area recognition unit 211, a commodity detection model storage unit 212, a shadow intensity calculation unit 213, a shadow intensity-light intensity correspondence relationship storage unit 214, a blown-out highlight detection unit 215, and a blown-out highlight detection model storage unit 216.

[0037] The area recognition unit 211 recognizes the product area in the captured image stored in the image storage unit 203 on a pixel basis, based on the product detection model that the learning unit 207 has learned and stored in the product detection model storage unit 212, and on edge detection. The edge detection is used to detect the outline of the product and recognize the product area on a pixel basis. The area recognition unit 211 recognizes the product area by combining the product detection model and edge detection, in order to more accurately determine whether the detected edge is the edge of the product or the edge of noise (e.g., the user's hand).

[0038] The area recognition unit 211 also recognizes a shadow area in the captured image stored in the image storage unit 203 based on the recognized product area and the background difference method. Specifically, in order to recognize the shadow area, the area recognition unit 211 first calculates the difference in brightness between the captured image (background image) before the product stored in the storage device 112 is placed and the captured image after the product is placed, for example, to detect the shadow area and the product area. Next, the area recognition unit 211 recognizes the shadow area by excluding the product area recognized by the above-mentioned product detection from the detected shadow area and product area. Then, the area recognition unit 211 outputs the position coordinate information of the recognized shadow area and the difference value of the shadow area between the captured images before and after the product is placed to the shadow intensity calculation unit 213. Note that in this embodiment, it is assumed to recognize a shadow area that temporarily covers a part of the product when the product is placed, such as the shadow of a person who is about to make a payment. Therefore, in this embodiment, an image captured in a state where there is no object that causes a shadow (in the above example, the person who is about to make a payment) is used as the background image. In addition, by using an image taken in the real environment before the product is placed on it, a background image can be obtained that reflects the brightness according to the ambient light in the real environment, etc., so that the shadow area and product area can be accurately detected.

[0039] The shadow intensity calculation unit 213 calculates the shadow intensity of the shadow region based on the difference value of the shadow region between the captured images before and after the product is placed, which is input from the area recognition unit 211. The shadow intensity is a value indicating the darkness of the shadow. Specifically, the shadow intensity calculation unit 213 calculates the average value of the absolute value of the difference value for each pixel of the shadow region between the background image and the captured image after the product is placed, as the shadow intensity of the shadow region.

[0040] Next, based on the calculated shadow intensity, the shadow intensity calculation unit 213 determines the intensity of light to be emitted by the illumination projection unit 201 in accordance with the shape of the shadow region. Specifically, the shadow intensity calculation unit 213 refers to the shadow intensity-light intensity correspondence relationship data stored in the shadow intensity-light intensity correspondence relationship storage unit 214, and determines the intensity of light to be emitted by the illumination projection unit 201 in accordance with the shape of the shadow region.

[0041] Here, the shadow intensity-light intensity correspondence data is data indicating a correspondence relationship between the shadow intensity of a shadow region and the intensity of light irradiated by the illumination projection unit 201 according to the shape of the shadow region. For example, a shadow intensity of 50 or more and less than 60 may be associated with the light intensity of the above-mentioned stage 2, a shadow intensity of 60 or more and less than 70 may be associated with the light intensity of the above-mentioned stage 3, a shadow intensity of 70 or more and less than 80 may be associated with the light intensity of the above-mentioned stage 4, and a shadow intensity of 80 or more may be associated with the light intensity of the above-mentioned stage 5. Note that this correspondence relationship between the shadow intensity and the light intensity may be determined by confirming a correspondence relationship that has a small effect on the recognition accuracy through an experiment. Furthermore, a correspondence in a finer unit may be performed, such as a one-to-one correspondence between the shadow intensity and the light intensity.

[0042] Then, the shadow intensity calculation unit 213 outputs the position coordinate information of the shadow area input from the area recognition unit 211 and the determined light intensity to the illumination projection unit 201. In other words, the shadow intensity calculation unit 213 adjusts the illumination projection unit 201 to irradiate light with the determined intensity onto the portion of the mounting surface of the commodity mounting unit 105 that corresponds to the shadow area.

[0043] In this way, when a shadow occurs in a captured image, the light irradiated to the portion of the mounting surface of the commodity mounting unit 105 corresponding to the shadow area is made stronger (brighter) than the default intensity, thereby reducing the shadow intensity of the shadow area. As a result, it is possible to stabilize the commodity recognition accuracy.

[0044] The blown-out highlight detection unit 215 recognizes blown-out highlight areas in the captured image stored in the image storage unit 203, based on the blown-out highlight detection model that the learning unit 207 has learned and that is stored in the blown-out highlight detection model storage unit 216. Next, the blown-out highlight detection unit 215 determines not to irradiate light onto the portions of the placement surface of the product placement unit 105 that correspond to the blown-out highlight areas, i.e., to set the light intensity to the above-mentioned level 0 (off).

[0045] Then, the blown-out highlight detection unit 215 outputs the position coordinate information of the blown-out highlight area and the determined light intensity (level 0) to the lighting projection unit 201. In other words, the blown-out highlight detection unit 215 adjusts the lighting projection unit 201 so as not to irradiate light onto the portion of the mounting surface of the product mounting unit 105 that corresponds to the blown-out highlight area.

[0046] Note that it is desirable for the recognition of blown-out highlight areas by the blown-out highlight detection unit 215 to be performed after the irradiation light has been adjusted based on the shadow intensity by the shadow intensity calculation unit 213. This is because there is a risk of new blown-out highlights occurring as a result of the irradiation light adjustment based on the shadow intensity by the shadow intensity calculation unit 213, i.e., as a result of the irradiation light becoming brighter.

[0047] In this way, when whiteout occurs in a captured image, the reflection that causes whiteout is suppressed by turning off the light irradiated to the portion of the mounting surface of the commodity mounting unit 105 that corresponds to the whiteout area. As a result, it is possible to stabilize the commodity recognition accuracy.

[0048] The commodity recognition unit 205 recognizes commodities in the captured images stored in the image storage unit 203 based on the commodity recognition model that the learning unit 207 has learned and stored in the commodity recognition model storage unit 206 .

[0049] The learning unit 207 attaches the label "product" to the images generated and acquired by the image acquisition unit 202 capturing images of products placed on the placement surface of the product placement unit 105 in advance at various angles, and stores the images as training data in the product detection model storage unit 212. The learning unit 207 also generates (trains) a product detection model using the training data, and stores the generated product detection model in the product detection model storage unit 212. The product detection model is used to detect that an object placed on the placement surface of the product placement unit 105 is a product.

[0050] The learning unit 207 labels blown-out highlight images included in images generated and acquired by the image acquisition unit 202 by capturing images of the products placed on the placement surface of the product placement unit 105 at various angles, and stores the images as teacher data in the blown-out highlight detection model storage unit 216. The learning unit 207 also uses the teacher data to generate (train) a blown-out highlight detection model, and stores the generated blown-out highlight detection model in the blown-out highlight detection model storage unit 216. The blown-out highlight detection model is used to detect the presence of blown-out highlights in a captured image.

[0051] The learning unit 207 labels the images generated and acquired by the image acquisition unit 202 in advance, which are images of products placed on the placement surface of the product placement unit 105 at various angles, with product identification information such as product names and product codes, and stores the images as training data in the product recognition model storage unit 206. The learning unit 207 also generates (trains) a product recognition model using the training data, and stores the generated product recognition model in the product recognition model storage unit 206. The product recognition model is used to classify the type of product that an object placed on the placement surface of the product placement unit 105 is. The product images used to generate the product recognition model may be the same as the product images used to generate the product detection model.

[0052] The illumination projection unit 201 is an example of an illumination unit according to the present disclosure. The image acquisition unit 202 is an example of an acquisition unit according to the present disclosure. The area recognition unit 211 and the blown-out highlight detection unit 215 are an example of a recognition unit according to the present disclosure. The shadow intensity calculation unit 213 and the blown-out highlight detection unit 215 are an example of an adjustment unit according to the present disclosure.

[0053] The area recognition unit 211, the shadow intensity calculation unit 213, the whiteout detection unit 215, the product recognition unit 205 and the learning unit 207 may be software modules realized, for example, by the control device 111 of the information processing device 103 executing a program stored in the memory device 112 of the information processing device 103 shown in FIG. 1.

[0054] Such a program may exist in a server external to the information processing device 103, or may be executed by the control device 111 via a network through the communication device 113 of the information processing device 103. Furthermore, such a program may be stored (recorded) in a storage (recording) medium such as a CD-ROM or a DVD, in addition to a storage device such as a memory or a hard disk.

[0055] The image memory unit 203, the product detection model memory unit 212, the shadow intensity-light intensity correspondence relationship memory unit 214, the blown-out highlight detection model memory unit 216 and the product recognition model memory unit 206 may be, for example, memory areas formed in the memory device 112 of the information processing device 103 by the control device 111 of the information processing device 103 executing the above-mentioned programs.

[0056] 3 is a diagram showing an example of a captured image 300 obtained by using default irradiation light to capture an image of an object (product) placed on the placement surface of the product placement section 105. Here, the default irradiation light may be light that is irradiated onto the placement surface of the product placement section 105 at the intensity of the above-mentioned stage 1.

[0057] In the captured image 300, there are products 301-305, whiteout 306, and a shadow 307. Whiteout 306 is caused by reflection due to glare of lighting. Shadow 307 is caused by the shadow of a user. In this way, whiteout and shadows may occur dynamically or locally in the captured image during the process of recognizing products by the product recognition system.

[0058] FIG. 4 is a diagram for explaining recognition of a product area in the captured image 300 shown in FIG.

[0059] The area recognition unit 211 recognizes the product areas 301' to 305' in the captured image 300 based on the product detection model stored in the product detection model storage unit 212 and edge detection. Since a material with a simple color scheme such as a white plate is used for the mounting surface of the product mounting unit 105, the background in the captured image 300 is a simple background, and product detection is a single-class recognition of whether or not it is a product, which is a simple task compared to product classification. Therefore, with regard to product detection, recognition that is robust against lighting variations is possible. Furthermore, the area recognition unit 211 detects the outline of the product by edge detection, recognizes the product area on a pixel basis, and acquires position coordinate information of the product area.

[0060] FIG. 5 is a diagram for explaining recognition of a shadow area in the captured image 300 shown in FIG. 3 and calculation of the shadow intensity.

[0061] The area recognition unit 211 calculates the difference in brightness between the captured image (background image) before the product is placed and the captured image after the product is placed, and detects the shadow area 307' and product areas 301' to 304' (and possibly 305').The area recognition unit 211 then recognizes the shadow area 307' by excluding the product areas 301' to 304' recognized by the above-mentioned product detection from the detected shadow area 307' and product areas 301' to 304'.

[0062] The shadow intensity calculation unit 213 calculates the average of the absolute values ​​of the difference values ​​for each pixel between the shadow region 307' in the captured image and the background image as the shadow intensity of the shadow region 307'. Then, the shadow intensity calculation unit 213 refers to the shadow intensity-light intensity correspondence relationship data stored in the shadow intensity-light intensity correspondence relationship storage unit 214, and determines the intensity of the light to be irradiated by the illumination projection unit 201 in accordance with the shape of the shadow region 307'.

[0063] FIG. 6 is a diagram for explaining how to recognize a whiteout region in the captured image 300 shown in FIG.

[0064] The blown-out highlight detection unit 215 recognizes a blown-out highlight area 306' in the captured image 300 based on the blown-out highlight detection model stored in the blown-out highlight detection model storage unit 216. The blown-out highlight area is recognized as a rectangular area as shown in the figure. This is because the learning unit 207 uses an image of a rectangular area as training data when training the blown-out highlight detection model. If the blown-out highlight detection model is trained using an image of another shape, such as a circle, as training data, the blown-out highlight area is recognized as an area that matches that shape. When the blown-out highlight detection unit 215 recognizes the blown-out highlight area 306', it determines not to irradiate light in accordance with the shape of the blown-out highlight area 306' (the intensity of the above-mentioned stage 0).

[0065] <Product Recognition System Operation> Next, the operation of the product recognition system 10 will be described.

[0066] Fig. 7 is a flowchart showing an example of the operation of the product recognition system 10 according to the first embodiment. The operation shown in Fig. 7 includes the lighting adjustment method according to the present disclosure, and may be executed at the timing when product recognition is performed at the time of product payment by a user. The operation shown in Fig. 7 may be executed, for example, when the user presses a payment button displayed on the display device 104, which is a touch panel.

[0067] In step ST001, the image acquisition unit 202 captures an image of the placement surface on which the commodity is placed of the commodity placement unit 105. In step ST001, the illumination projection unit 201 irradiates the placement surface of the commodity placement unit 105 with, for example, the default irradiation light of the above-mentioned stage 1.

[0068] In step ST002, the area recognition unit 211 recognizes the product area and the shadow area in the captured image acquired in step ST001.

[0069] In step ST003, the shadow intensity calculation unit 213 calculates the shadow intensity of the shadow region recognized in step ST002.

[0070] In step ST004, the shadow intensity calculation unit 213 determines the intensity of light to be emitted by the illumination projection unit 201 in accordance with the shape of the shadow region recognized in step ST002, based on the shadow intensity calculated in step ST003. That is, the shadow intensity calculation unit 213 adjusts the illumination projection unit 201 so that it emits light with the intensity determined in accordance with the shape of the shadow region recognized in step ST002.

[0071] In step ST005, the image acquisition unit 202 captures an image of the placement surface on which the product is placed of the product placement unit 105. In step ST005, the illumination projection unit 201 irradiates light with the intensity determined in step ST004 in accordance with the shape of the shadow area, and irradiates light with the intensity irradiated in step ST001 to areas other than the shadow area.

[0072] In step ST006, the whiteout detection unit 215 recognizes whiteout areas in the captured image acquired in step ST005.

[0073] In step ST007, the whiteout detection unit 215 determines the intensity of the light emitted by the illumination projection unit 201 to be the above-mentioned level 0 (off (zero)) in accordance with the shape of the whiteout area recognized in step ST006.

[0074] In step ST008, the image acquisition unit 202 captures an image of the placement surface on which the product is placed of the product placement unit 105, and acquires a captured image. In step ST008, the illumination projection unit 201 irradiates light with the intensity determined in step ST004 according to the shape of the shadow region, does not irradiate the blown-out white region with light, and irradiates the regions other than the shadow region and the blown-out white region with light with the intensity irradiated in step ST001.

[0075] In step ST009, the commodity recognition unit 205 recognizes the commodity in the captured image acquired in step ST008.

[0076] If a shadow region is not recognized in step ST002, steps ST003 to ST005 are not executed, and the captured image acquired in step ST001 is used in the detection of blown out highlights in step ST006.

[0077] If no whiteout area is recognized in step ST006, steps ST007 to ST008 are not executed.

[0078] If no shadow area is recognized in step ST002 and no blown-out area is recognized in step ST006, in step ST009, the product recognition unit 205 recognizes the product in the captured image acquired in step ST001 instead of the captured image acquired in step ST008.

[0079] If a shadow area is recognized in step ST002 and a blown-out area is not recognized in step ST006, in step ST009, the product recognition unit 205 recognizes the product in the captured image acquired in step ST005 instead of the captured image acquired in step ST008.

[0080] FIG. 8 is a diagram for explaining the output contents (brightness of the placement surface) of the illumination projection section 201 after the irradiation light adjustment in steps ST004 and ST007 in FIG. 7, using the captured image 300 shown in FIG.

[0081] Areas in the captured image 300 other than the blown-out highlight area 306' and the shadow area 307' are irradiated with light of, for example, the intensity of the above-mentioned level 1, and these areas are shown in gray for convenience. On the other hand, the blown-out highlight area 306' in the captured image 300 is not irradiated with light corresponding to, for example, the intensity of the above-mentioned level 0, and the blown-out highlight area 306' is shown in black, which is darker than gray, for convenience. On the other hand, the shadow area 307' in the captured image 300 is irradiated with light of, for example, the intensity of the above-mentioned level 2, and the shadow area 307' is shown in white, which is brighter than gray, for convenience.

[0082] FIG. 9 is a diagram showing an example of an image 300' reacquired by imaging an object (product) placed on the placement surface of the product placement section 105 in step ST009 of FIG. 7 after the irradiation light adjustment in steps ST004 and ST007 of FIG. 7.

[0083] In the captured image 300 ′, the original blown-out highlights 306 are suppressed compared to the captured image 300 , while the original shadows 307 are brighter compared to the captured image 300 .

[0084] <Advantages of the First Embodiment> As described above, the illumination adjustment device according to the first embodiment of the present disclosure includes a recognition unit that recognizes an abnormal area in a captured image, and an adjustment unit that adjusts the intensity of illumination light according to the shape and type of the abnormal area. With this configuration, it is possible to reduce the influence of the abnormal area on the product recognition accuracy by using, for example, a projector as an illumination unit that irradiates illumination light and performing illumination adjustment to, for example, brighten a shadow area, which is a type of abnormal area, and darken a blown-out highlight area, which is a type of abnormal area, and irradiate the illumination light. As a result, it is possible to generate a precise captured image that can deal with the abnormal area, and to stabilize the product recognition accuracy. Furthermore, it is possible to improve usability, such as reducing erroneous settlements. In addition, it is not necessary to adjust the lighting environment when installing the product recognition system 10.

[0085] <Modification> In the first embodiment, an example has been described in which the lighting projection unit 201 irradiates light in accordance with the distance to the placement surface of the commodity placement unit 105, but the present disclosure is not limited to this example. For example, a three-dimensional camera may be used as the imaging device 102 (light projection unit 201), or a distance sensor may be further provided to estimate the height of a tall commodity, and the shape of the irradiated light for such a commodity may be corrected three-dimensionally. Specifically, this is as follows.

[0086] When the product has a certain height, the distance between the top surface of the product and the lighting device 101 becomes short, and the image of the part corresponding to the shadow area illuminated with the adjusted illumination light or the blown-out area not illuminated with the illumination light may become smaller than predicted.

[0087] Therefore, the three-dimensional camera or distance sensor measures the distance between the top surface of the product and the lighting device 101, and outputs the measured distance to the lighting adjustment unit 204 (more specifically, one or both of the shadow intensity calculation unit 213 and the whiteout detection unit 215). The lighting adjustment unit 204 may then use the measured distance to correct the lighting so that the shadow area / whiteout area is accurately covered at the height of the top surface of each product. If there is product height information, the position and size of the product in the three-dimensional coordinate system can be grasped, so that the lighting adjustment unit 204 can correct the accurate irradiation position (range) of the irradiation light after the irradiation light adjustment in steps ST004 and ST007 of FIG. 7 even for tall products. Note that when the product height information is used, if the height of the product is equal to or greater than a predetermined threshold, the lighting adjustment unit 204 may correct the irradiation light as described above.

[0088] On the other hand, if there are no extremely expensive products, the lighting adjustment unit 204 can deal with shadow areas / blown out highlight areas for most products by setting the areas to be brightened (if a shadow area is recognized) / areas to be darkened (if a blown out highlight area is recognized) to be wider.

[0089] In the first embodiment, an example has been described in which the shadow intensity calculation unit 213 calculates the average of the absolute values ​​of the difference values ​​for each pixel in the shadow region from the background image as the shadow intensity of the shadow region. However, the present disclosure is not limited to this example.

[0090] For example, if the shadow area is larger than a predetermined block, the shadow intensity calculation unit 213 may divide the shadow area into a predetermined smaller block and calculate the average value of the absolute difference value for each pixel for each divided block. Then, the shadow intensity calculation unit 213 may determine the light intensity for each divided block. Note that the shape of the block is, for example, a rectangle, but may be another shape such as a triangle.

[0091] Furthermore, when there are multiple shadow regions separated from each other, the shadow intensity calculation unit 213 may calculate the shadow intensity for each shadow region and determine the light intensity for each shadow region. Here, multiple shadow regions separated from each other occur, for example, when multiple people are standing in front of the checkout counter at the same time, or when there is an object other than people near the checkout counter. In this case, too, when each shadow region is larger than a predetermined block, the shadow intensity may be calculated for each further divided block, and the light intensity may be determined.

[0092] Furthermore, when there is unevenness in the shadow intensity of a shadow region, the shadow intensity calculation section 213 may change the light intensity in accordance with the unevenness.

[0093] Similarly, if there is unevenness in a blown-out highlight area, the blown-out highlight detection unit 215 may change the light intensity in accordance with the unevenness. In this case, for example, one or more stages may be set between the above-mentioned stage 0 and stage 1.

[0094] In the first embodiment, an example has been described in which the illumination adjustment section 204 deals with both shadow regions and whiteout regions, but the present disclosure is not limited to this example.

[0095] For example, the illumination adjustment unit 204 may deal with only one of the shadow region and the blown-out highlight region. That is, the blown-out highlight detection unit 215 and the blown-out highlight detection model storage unit 216 may not exist, and only the region recognition unit 211, the product detection model storage unit 212, the shadow intensity calculation unit 213, and the shadow intensity-light intensity correspondence relationship storage unit 214 may exist. Alternatively, the region recognition unit 211, the product detection model storage unit 212, the shadow intensity calculation unit 213, and the shadow intensity-light intensity correspondence relationship storage unit 214 may not exist, and only the blown-out highlight detection unit 215 and the blown-out highlight detection model storage unit 216 may exist.

[0096] In the first embodiment, an example has been described in which, when at least one of a shadow region and a whiteout region is recognized, the at least one of the shadow region and the whiteout region is dealt with. However, for example, if at least one of the shadow region and the whiteout region is sufficiently small (for example, if it is less than a predetermined area), the illumination adjustment unit 204 may not recognize it as at least one of the shadow region and the whiteout region and may not deal with it. In other words, if at least one of the shadow region and the whiteout region is equal to or larger than a predetermined area, the illumination adjustment unit 204 may recognize it as at least one of the shadow region and the whiteout region and deal with it as described above. This is because if at least one of the shadow region and the whiteout region is sufficiently small, it is considered that the impact on the product recognition accuracy is small.

[0097] In the first embodiment, the adjustment of the light intensity has been described as an example, but the color of the light may be further adjusted. For example, when a light source of a special color is arranged around the placement table, the color may be mixed with the original color of the product, and the product may appear to be a different color, resulting in a decrease in recognition accuracy. In such a case, the illumination adjustment unit 204 can improve the recognition accuracy by instructing the illumination device, which is a projector, to irradiate light of a color that cancels out the color. The color that may affect the recognition accuracy can be specified, for example, from the difference between the color of the placement table in an ideal environment recorded in advance (white, for example) and the color of the placement table in the captured image. In addition, the color that cancels out the product color can be determined by calculating a color that may affect the recognition accuracy and becomes white when combined with the color that becomes white.

[0098] In the first embodiment, the color of the placement table of the commodity placement unit 105 is, as an example, a material with a simple color scheme, but the color or pattern of the placement surface may be variable. Such a placement surface can be realized, for example, by configuring the placement surface with a liquid crystal display or the like, or by using the placement surface as a screen to project an image from behind the placement surface with another projector or the like. In this case, the commodity recognition system 10 can obtain an image corresponding to the placement surface before the commodity is placed thereon by obtaining data of the image used for the placement surface. Then, by using this color or pattern as a background image, the illumination light can be adjusted according to the abnormal area as in the first embodiment.

[0099] In the first embodiment, the commodity recognition system 10 first irradiates illumination light with a default intensity. However, if a separately installed illuminance sensor or the like detects that the placement platform of the commodity placement unit 105 is sufficiently bright, this process may be omitted. In this case, since it is not possible to control the illumination light to be darker than the current state, the process related to the overexposed area may be omitted.

[0100] In the first embodiment, it was decided that the light intensity for the blown-out areas would be set to level 0 (off), but if there are multiple levels between the default and level 0, light that is less than the default and stronger than level 0 (off) may be used. When blown-out areas and shadow areas are mixed, it is possible to control the shadow areas so that they do not become too dark at level 0 (off). In addition, it may be decided whether the light intensity is set to level 0 (off) or to a higher intensity depending on the ratio of the shadow area included in the blown-out area, etc. Since the occurrence of blown-out is affected by the reflectance and material of the subject, in an area where blown-out has occurred with light of the default intensity, blown-out may occur unless the light is turned off (however, if the light is weak, the area of ​​blown-out may be small). Therefore, if the main purpose is to eliminate blown-out, such as when the blown-out area does not include a shadow area or the ratio of the shadow area is small, it is useful to set the light intensity for the blown-out area to level 0 (off) regardless of the number of levels between the default light intensity and level 0.

[0101] In the first embodiment, the background image may be an image of the placement table captured in advance in an ideal environment. Shadows that appear on the placement table include temporary shadows such as the shadow of a person and permanent shadows caused by surrounding objects (such as shelves and pillars). Therefore, if an image captured before a product is placed on the placement table is used as the background image, it is possible to obtain a background image without temporary shadows, but it is not possible to eliminate the permanent shadows. Therefore, in a method of identifying a shadow area based on a difference from a background image, there is a risk that a portion corresponding to a permanent shadow will not be detected as a shadow area. On the other hand, if an image of the placement table captured in advance in an ideal environment is used as the background image, an image without a permanent shadow can be used as the background image, so that it is possible to recognize and deal with the permanent shadow as a shadow area as well.

[0102] In the first embodiment, an example of adjusting the light emitted by the lighting device, which is a projector, for the shadow area and the blown-out highlight area has been described, but the present disclosure is not limited to this example.

[0103] For example, instead of using a projector, a single-color lighting device may be used to adjust the brightness of the shadow and whiteout areas. In this case, a projector is not required, and it can be realized at low cost. In this case, the lighting device can be adjusted so that it becomes brighter as the shadow becomes darker (the shadow intensity becomes stronger), and the wider the whiteout area becomes, the darker it becomes.

[0104] In the above-described embodiments, the notation "... part" used for each component may be replaced with other notations such as "... circuit", "... assembly", "... device", "... unit", or "... module".

[0105] Although the embodiments have been described above with reference to the drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modified or altered examples within the scope of the claims. It is understood that such modified or altered examples also belong to the technical scope of the present disclosure. In addition, the components in the embodiments may be arbitrarily combined within the scope of the present disclosure.

[0106] The present disclosure can be realized in software, hardware, or software in conjunction with hardware.

[0107] Each functional block used in the description of the above embodiments may be realized, in part or in whole, as an LSI, which is an integrated circuit, and each process described in the above embodiments may be controlled, in part or in whole, by one LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of one chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be called an IC, a system LSI, a super LSI, or an ultra LSI.

[0108] The method of integration is not limited to LSI, and may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, after LSI manufacturing, a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may be used. The present disclosure may be realized as digital processing or analog processing.

[0109] Furthermore, if a new integrated circuit technology that can replace LSI appears due to the progress of semiconductor technology or a derivative technology, it is possible to integrate the functional blocks using that technology. The application of biotechnology is also a possibility. [Industrial Applicability]

[0110] An embodiment of the present disclosure is useful as a product recognition device. [Explanation of symbols]

[0111] 10 Product recognition system, 101 Lighting device, 102 Imaging device, 103 Information processing device, 104 Display device, 105 Product placement unit, 106 Product, 111 Control device, 112 Storage device, 113 Communication device, 201 Lighting projection unit, 202 Image acquisition unit, 203 Image storage unit, 204 Lighting adjustment unit, 205 Product recognition unit, 206 Product recognition model storage unit, 207 216....brightness detection model storage unit, 300....captured image, 301-305....product, 306....brightness, 307....shadow, 301'-305'....product region, 306'....brightness region, 307'....shadow region, 300'...captured image

Claims

1. A recognition unit that recognizes a product area of ​​a product from a captured image and recognizes a shadow area in the captured image by excluding the recognized product area from a difference between the captured image and a background image; an adjustment unit that adjusts the intensity of the illumination light when the shadow area is recognized so that the intensity of the illumination light is stronger than the intensity of the illumination light that was irradiated to acquire the captured image; A lighting adjustment device comprising:

2. The recognition unit, Recognizing a product region of the product based on a product detection model for detecting the product and edge detection; The illumination adjusting device according to claim 1 .

3. When the shadow area is recognized, the adjustment unit calculates a shadow intensity of the shadow area, and determines an intensity of the illumination light based on the shadow intensity of the shadow area. The illumination adjusting device according to claim 1 or 2.

4. The recognition unit recognizes an abnormal area including at least one of the shadow area and the blown-out white area, The adjustment unit adjusts the intensity of the illumination light according to the shape and type of the abnormal area. The illumination adjusting device according to claim 1 .

5. When the blown-out white area is recognized, the adjustment unit adjusts the intensity of the illumination light so that it is weaker than the intensity of the illumination light irradiated to acquire the captured image. The illumination adjusting device according to claim 4.

6. When the shadow region and the blown-out highlight region are recognized, the adjustment unit adjusts the intensity of the illumination light so that it is stronger than the intensity of the illumination light irradiated to obtain the captured image, and then adjusts the intensity of the illumination light so that it is weaker than the intensity of the illumination light irradiated to obtain the captured image. The illumination adjusting device according to claim 5 .

7. The recognition unit recognizes the abnormal region when the abnormal region has an area equal to or larger than a predetermined area. The illumination adjusting device according to any one of claims 4 to 6.

8. The adjustment unit is correcting an illumination range of the illumination light that is irradiated with the intensity according to the shape and type of the abnormal area based on the height of the product; The illumination adjusting device according to any one of claims 4 to 7.

9. A method for recognizing a product area of ​​a product from a captured image, and recognizing a shadow area in the captured image by excluding the recognized product area from a difference between the captured image and a background image; If the shadow area is recognized, the intensity of the illumination light is adjusted so that it is stronger than the intensity of the illumination light irradiated to acquire the captured image. How to adjust lighting.

10. an illumination unit that irradiates illumination light onto a predetermined area on which the product is placed; an acquisition unit that captures and acquires an image of the predetermined area irradiated with the illumination light; Recognizing a product area of ​​a product from the captured image, and excluding the recognized product area from a difference between the captured image and a background image, thereby recognizing a shadow area in the captured image; If the shadow area is recognized, the intensity of the illumination light is adjusted so that it is stronger than the intensity of the illumination light irradiated to acquire the captured image. A lighting control device; Equipped with the acquisition unit re-captures and re-acquires an image of the predetermined area to which the illumination unit irradiates or does not irradiate the illumination light with the adjusted intensity, The lighting adjustment device recognizes the product in the re-captured and re-acquired image. Product recognition system.

11. The illumination adjustment device recognizes an abnormal area including at least one of the shadow area and the blown-out highlight area, and adjusts the intensity of the illumination light according to the shape and type of the abnormal area; correcting an illumination range of the illumination light that is irradiated with the intensity according to the shape and type of the abnormal area based on the height of the product; The product recognition system according to claim 10.

Images