Image recognition device and program

Abstract

To provide an image recognition device and program capable of performing photographing on an appropriate photographing condition corresponding to a recognition object.SOLUTION: A self POS terminal (an image recognition device) includes: an image photographing part for photographing an image including a subject held over by an operator; a photographing condition setting part for setting a photographing condition by the image photographing part in any of a plurality of preset conditions; a photographing condition writing part for writing an identification ID (identification information) corresponding to the photographing condition set by the photographing condition setting part in the image photographed by the image photographing part; an image recognition part for reading the identification ID written in the image photographed by the image photographing part, and recognizing the subject included in the image by performing different image processing corresponding to a photographing condition corresponding to the identification ID; and a recognition result determination part (a determination part) for determining whether the image recognition part recognizes the subject. The photographing condition setting part changes the photographing condition and makes the image photographing part photograph the image again on condition that the recognition result determination part does not determine that the subject is recognized.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] FIELD An embodiment of the present invention relates to an image recognition device and a program. [Background technology]

[0002] A conventional technology for registering products uses general object recognition (object recognition), which recognizes the type of product, etc., by extracting feature amounts of the product from an image captured by an image recognition scanner, including the product's appearance, and comparing the extracted feature amounts with pre-prepared matching data. A product data processing device has been proposed that can perform both product recognition based on the results of object recognition and product recognition based on barcodes (for example, Patent Document 1).

[0003] In such conventional product data processing devices, since it is unknown what kind of recognition target will be presented to the scanner, images are taken under the same shooting conditions to recognize objects (products) and code symbols. However, in order to further improve recognition accuracy, it is desirable to be able to appropriately adjust the image shooting conditions (lighting brightness, gain of the imaging device, etc.) depending on the recognition target. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide an image recognition device and a program that are capable of capturing an image under appropriate capturing conditions according to the object to be recognized. [Means for solving the problem]

[0005] An image recognition device according to an embodiment includes an image capturing unit, a capturing condition setting unit, a capturing condition writing unit, an image recognition unit, and a determination unit. The image capturing unit captures an image including a subject held by an operator. The capturing condition setting unit sets the capturing conditions for the image capturing unit to one of a plurality of preset conditions. The capturing condition writing unit writes identification information corresponding to the capturing conditions set by the capturing condition setting unit into the image captured by the image capturing unit. The image recognition unit recognizes the subject included in the image by reading the identification information written into the image captured by the image capturing unit and performing different image processing according to the capturing conditions corresponding to the identification information. The determination unit determines whether the image recognition unit has recognized the subject. Furthermore, the capturing condition setting unit changes the capturing conditions and causes the image capturing unit to capture the image again, on condition that the determination unit does not determine that the subject has been recognized. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is an external view illustrating an example of a self-service POS terminal according to an embodiment. [Figure 2] FIG. 2 is a first diagram illustrating an example of a schematic configuration of a scanner included in a self-service POS terminal. [Figure 3] FIG. 3 is a second diagram illustrating an example of a schematic configuration of the scanner included in the self-service POS terminal. [Figure 4] FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the self-checkout POS terminal. [Figure 5] FIG. 5 is a diagram showing an example of the configuration of the imaging condition setting file. [Figure 6] FIG. 6 is a diagram showing an example of the structure of an image captured by a scanner. [Figure 7] FIG. 7 is a diagram showing an example of an image obtained by capturing a printed code symbol. [Figure 8] FIG. 8 is a diagram showing an example of an image obtained by capturing a code symbol displayed on a mobile terminal. [Figure 9] FIG. 9 is a diagram showing an example of an image of a natural object such as a vegetable or fruit. [Figure 10]FIG. 10 is a diagram showing an example of an image of a packaged product. [Figure 11] FIG. 11 is a diagram showing an example of a writing position of an identification ID indicating the photographing conditions of a photographed image. [Figure 12] FIG. 12 is a diagram showing an example of a writing format for the identification ID. [Figure 13] FIG. 13 is a functional block diagram illustrating an example of a functional configuration of the self-checkout POS terminal according to the embodiment. [Figure 14] FIG. 14 is a flowchart illustrating an example of a flow of processing performed by the self-checkout POS terminal according to the embodiment. [Figure 15] FIG. 15 is a flowchart illustrating an example of the flow of image recognition processing performed by the self-checkout POS terminal according to the embodiment. [Figure 16] FIG. 16 is a hardware block diagram illustrating an example of a hardware configuration of a self-checkout POS terminal according to a modified example of the embodiment. [Figure 17] FIG. 17 is a functional block diagram illustrating an example of a functional configuration of a self-checkout POS terminal according to a modified example of the embodiment. [Figure 18] FIG. 18 is a flowchart showing an example of the flow of image recognition processing performed by the self-checkout POS terminal according to the modified example of the embodiment. [Figure 19] FIG. 19 is a flowchart showing an example of the flow of image recognition processing performed by an image processing device in the modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] (Explanation of the general configuration of the self-service POS terminal) A self-checkout POS terminal 10 according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is an external view showing an example of a self-checkout POS terminal according to an embodiment.

[0008] The self-service POS terminal 10 is a cashless payment machine operated by a customer in a store such as a retail shop to register and pay for products that the customer wishes to purchase. The self-service POS terminal 10 is an example of an image recognition device in the present disclosure.

[0009] The self-service POS terminal 10 includes, on the top of the housing 11, various devices for registering products and making payments for registered products.

[0010] A product stand 12 is provided on the side of the housing 11. A customer places a shopping basket containing unregistered products on the product stand 12.

[0011] A product stand 13 is formed on the top surface of the housing 11. The customer places registered products in their own bag placed on the product stand 13.

[0012] The upper part of the housing 11 is provided with a display device 14, an operation device 15, a scanner 16, a payment terminal 17, a receipt printer 18, and a warning light 19.

[0013] The display device 14 displays various information related to product registration and payment to the customer. The various information related to product registration includes, for example, registered product information and the total price. The various information related to payment includes, for example, payment method options and payment details. The display device 14 is configured, for example, by a liquid crystal monitor or an organic electroluminescence (EL) monitor.

[0014] The operation device 15 acquires various operation information related to product registration and payment from the customer. The various operation information related to product registration is, for example, an instruction to start product registration, an instruction to complete registration, etc. The various operation information related to payment is, for example, an instruction to select a payment method, etc. The operation device 15 is configured, for example, by being stacked on the display device 14.

[0015] The scanner 16 has an internal imaging device 164 (see FIG. 2) and captures an image of a code symbol (e.g., a barcode) with a registered product code that is held up to a reading window 161 on the front. The scanner 16 then recognizes the product code registered in the code symbol shown in the captured image. The scanner 16 recognizes a printed code symbol that is attached or printed on a product, or a code symbol that is displayed on the screen of a mobile terminal carried by a customer.

[0016] The scanner 16 also captures an image of a product without a code symbol attached that is held over the reading window 161. The scanner 16 then recognizes the product shown in the captured image and outputs a product code corresponding to the recognized product. More specifically, the scanner 16 recognizes natural objects such as vegetables and fruits, as well as packaged products.

[0017] Payment terminal 17 performs various cashless payments. Payment terminal 17 includes a display device 171, a numeric keypad 172, and a card reader 173. Payment terminal 17 also includes a reader (not shown) that performs short-range wireless communication with an electronic money card or the like.

[0018] The display device 171 displays various information such as operation guidance when various cashless payments are made. The display device 171 is, for example, a liquid crystal monitor or an organic EL monitor. The display function of the display device 171 may be substituted by the display device 14 described above.

[0019] The ten keys 172 are operation keys for inputting various personal identification numbers related to cashless payment.

[0020] Card reader 173 reads information registered on various card media such as credit cards and electronic money cards when making a cashless payment. Card reader 173 reads information registered on a card media inserted through card insertion slot 174. Card reader 173 also reads information registered on the card media by performing short-range wireless communication with the card media held over a predetermined position on payment terminal 17. Card reader 173 may also have a function to read information registered on a card media scanned along a scanning groove (not shown) for reading a magnetic card.

[0021] Cashless payments include credit card payments, code payments, electronic money payments, etc.

[0022] Credit card payment is a payment method that uses a customer's credit card. When making a credit card payment, the customer has the payment terminal 17 read the personal information and debit account information registered on the customer's credit card to make the payment.

[0023] Code payment is a payment method in which payment is made using a payment app pre-installed on a customer's mobile device. When making a code payment, the customer displays a code symbol on their mobile device and makes the payment by having the displayed code symbol read by, for example, a scanner 16.

[0024] Electronic money payment is a payment method in which payment is made using electronic money loaded onto, for example, an IC card. When making an electronic money payment, a customer holds their electronic money card over payment terminal 17 to establish short-range wireless communication with payment terminal 17, which reads the balance loaded on the electronic money card and allows payment to be made within the remaining balance.

[0025] The receipt printer 18 issues a receipt on which the result of the payment is printed.

[0026] The warning light 19 is installed at the tip of a pole extending upward from the housing 11, and emits light in blue, red, or the like. The warning light 19 emits light in different colors and lighting patterns depending on the operating state of the self-service POS terminal 10. For example, if a customer does not know how to operate the self-service POS terminal 10 and presses the store clerk call button displayed on the display device 14, the warning light 19 will light up and call for help from a nearby store clerk.

[0027] (Scanner structure) The structure of the scanner 16 will be described with reference to Figures 2 and 3. Figure 2 is a first diagram showing an example of the schematic configuration of a scanner provided in a self-service POS terminal. Figure 3 is a first diagram showing an example of the schematic configuration of a scanner provided in a self-service POS terminal.

[0028] 2 is a cross-sectional side view of scanner 16. Scanner 16 includes an LED 163, an imaging device 164, and a control board 165 inside a housing 162. An opening 166 is provided in housing 162 at a position facing the imaging direction of imaging device 164, and a reading window 161 is fitted into opening 166. Reading window 161 is made of a transparent glass plate with low reflectance.

[0029] The LED (Light Emitting Diode) 163 is disposed with its light emitting surface facing the reading window 161. The LED 163 emits illumination light that illuminates an object when the imaging device 164 captures an image of the object held over the reading window 161. The detailed configuration of the LED 163 will be described later in detail (see FIG. 3).

[0030] The imaging device 164 has a lens 167 on its front surface. The lens 167 is made up of a convex lens or a concave lens, and forms an image of the code symbol or product that is held over the reading window 161 on the imaging device 164. The imaging device 164 is an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and has photoelectric elements that convert the brightness of the formed image into an electrical signal arranged in a two-dimensional matrix. In other words, the imaging device 164 and the lens 167 form a so-called camera.

[0031] The control board 165 instructs the start or stop of the imaging operation of the imaging device 164. During the imaging operation, the control board 165 amplifies the electrical signal output from the imaging device 164 and performs signal processing such as A / D conversion to generate a digital image (hereinafter simply referred to as an image).

[0032] The control board 165 also sets various imaging parameters for when the imaging device 164 captures an image. The imaging parameters include, for example, the brightness of the LED 163, the gain of the imaging device 164, the exposure time of the imaging device 164, and the like.

[0033] The brightness of the LED 163 is set, for example, by changing the duty ratio of the drive pulse. That is, when the duty ratio of the drive pulse is reduced, the ON time (illumination time) of the LED 163 is shortened, and therefore the LED 163 becomes dimmer. On the other hand, when the duty ratio of the drive pulse is increased, the ON time of the LED 163 is lengthened, and therefore the LED 163 becomes brighter.

[0034] The gain of the imaging device 164 is defined as the ratio of the output voltage to the amount of light incident on the imaging device 164. Increasing the gain of the imaging device 164 will result in a higher output voltage being output for a smaller amount of light, making it possible to capture high-contrast images even in dark environments, for example. The gain of the imaging device 164 can be set, for example, by changing the amplification factor of a video amplifier provided downstream of the imaging device 164.

[0035] The exposure time of the imaging device 164 is set by adjusting the time interval of a clock pulse that controls the imaging operation of the imaging device 164. Shortening the exposure time has the same effect as increasing the shutter speed, and can prevent overexposure, which is a loss of gradation in bright parts of the image. On the other hand, lengthening the exposure time has the same effect as decreasing the shutter speed, and can prevent underexposure, which is a loss of gradation in dark parts of the image. Note that if the lens 167 has an aperture adjustment function, the control board 165 may set the exposure time by adjusting the aperture of the lens 167.

[0036] 3 is a front view of the scanner 16 as viewed from outside the reading window 161. As shown in FIG. 3, the reading window 161 is provided in an opening 166 of the housing 162, and an LED 163 and a lens 167 are provided inside the reading window 161. An imaging device 164 (not shown) is provided behind the lens 167. As shown in FIG. 3, the lens 167 is disposed, for example, at approximately the center of the reading window 161 when the reading window 161 is viewed from the front. The imaging device 164 captures an image of a subject through the reading window 161 from inside the housing 162 through the lens 167 and outputs the captured image.

[0037] The LED 163 is an illumination device for illuminating an object held over the reading window 161. The LED 163 includes a plurality of light-emitting elements, and the brightness of each light-emitting element can be changed by a light-emitting control signal and a load current input from a control board 165 (see FIG. 2). FIG. 3 shows an example in which the LED 163 is a plurality of white LEDs, namely, LEDs 163a to LEDs 163h, arranged in a line along the upper side of the reading window 161. These LEDs 163 have the same specifications and emit light under the same set conditions.

[0038] The number of LEDs 163 is not limited to the example in Fig. 3. Furthermore, LEDs of a color other than white may be used as the light-emitting elements, or LEDs of multiple colors may be combined. Alternatively, light-emitting elements other than LEDs may be used.

[0039] A diffusion filter 168 that diffuses the light emitted by the LED 163 is provided in front of the light-emitting surface of the LED 163. The diffusion filter 168 is arranged on the emission path of the light from the LED 163 and diffuses the light emitted by the LED 163. The diffusion filter 168 reduces uneven brightness of the light emitted by the LED 163, and the light is emitted as uniform illumination light.

[0040] (Hardware configuration of self-service POS terminal) The hardware configuration of the self-checkout POS terminal 10 will be described with reference to Fig. 4. Fig. 4 is a hardware block diagram showing an example of the hardware configuration of the self-checkout POS terminal.

[0041] The self-service POS terminal 10 has a configuration in which a control unit 21, a storage unit 22, a peripheral device controller 24, and a communication controller 25 are interconnected via an internal bus 23.

[0042] The control unit 21 controls the overall operation of the self-service POS terminal 10. The control unit 21 includes a CPU (Central Processing Unit) 211, a ROM (Read Only Memory) 212, and a RAM (Random Access Memory) 213. The CPU 211 is connected to the ROM 212 and the RAM 213 via internal buses such as an address bus and a data bus. The CPU 211 loads various programs stored in the ROM 212 and the storage unit 22 into the RAM 213. The CPU 211 controls the operation of the self-service POS terminal 10 by operating in accordance with the various programs loaded into the RAM 213. In other words, the control unit 21 has the configuration of a general computer.

[0043] The storage unit 22 is a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage unit 22 may also be a non-volatile memory such as a flash memory that retains stored information even when the power is turned off. The storage unit 22 stores a control program 221, a product master 222, a product recognition dictionary 223, a photography condition setting file 224, a product registration file 225, and a payment file 226.

[0044] The control program 221 is a program that controls the overall operation of the self-checkout POS terminal 10. The control program 221 may be provided in a state stored in the storage unit 22, or may be provided by being recorded in an installable or executable file on a computer-readable non-transitory recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD. The control program 221 may also be provided by being stored on a computer connected to a network and downloaded via the network. Furthermore, the control program 221 may be provided or distributed via a network such as the Internet.

[0045] The product master 222 is a master file that stores product information such as product names and prices of products for sale. The product master 222 stores product names, unit prices, additional information, and the like in association with product codes. Note that the contents of the product master are updated as needed, so the self-service POS terminal 10 constantly updates the contents of the product master 222 to the latest state by, for example, acquiring the latest product master from a store server (not shown) via the communication controller 25.

[0046] The product recognition dictionary 223 is a dictionary for recognizing products shown in images captured by the scanner 16. The product recognition dictionary 223 is written, for example, in a neural network, and calculates an evaluation score (recognition score) indicating the likelihood of a specific product in response to input of a plurality of image features extracted from an image captured by the scanner 16.

[0047] The photographing condition setting file 224 is a file that stores setting conditions for various photographing parameters when the scanner 16 photographs an image. The photographing condition setting file 224 stores photographing parameters according to the object to be photographed by the scanner 16. The object to be photographed by the scanner 16 includes, for example, a printed code symbol (e.g., a barcode), a code symbol (e.g., a barcode) displayed on the screen of a mobile terminal, a product without a code symbol such as a vegetable or fruit, a packaged product, etc. In this embodiment, the scanner 16 sets the brightness of the LED 163, the gain of the imaging device 164, and the exposure time of the imaging device 164 to states according to the object to be photographed. The configuration of the photographing condition setting file 224 will be described in detail later (see FIG. 5).

[0048] The product registration file 225 is a file that stores registered product information (for example, product name, quantity, unit price, etc.) in association with a transaction code that identifies one transaction.

[0049] The settlement file 226 is a file that stores the settlement details (for example, the settlement date and time, the settlement method, the settlement amount, etc.) of the transaction in association with the transaction code.

[0050] The control unit 21 is connected to the display device 14, operation device 15, scanner 16, payment terminal 17, receipt printer 18, and warning light 19, which input and output various information, via the peripheral device controller 24. The outline of each input and output device has been described above, so a repeated description will be omitted. More specifically, the scanner 16 has a configuration in which the control board 165 is connected to the LED 163 and the imaging device 164, as described in FIG. 2.

[0051] The control unit 21 also communicates with a store server (not shown) via the communication controller 25. The self-service POS terminal 10 acquires the latest product master 222 from the store server. The self-service POS terminal 10 also periodically outputs, for example, a product registration file 225 and a settlement file 226 indicating transaction history to the store server.

[0052] (Configuration of shooting condition setting file) The structure of the imaging condition setting file 224 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the structure of the imaging condition setting file.

[0053] The photographing condition setting file 224 stores an illumination brightness 32, a gain 33, and an exposure time 34 in association with an identification ID 31 that uniquely identifies the photographing condition. The identification ID 31 is an example of identification information in the present disclosure.

[0054] The illumination brightness 32 defines the lighting conditions of the LED 163, for example, in five stages from 0 (off) to 4 (brightest). More specifically, the set values ​​0 to 4 indicating the brightness of the LED 163 are each associated with a predetermined duty ratio of the drive pulse. Note that brightness 0 of the LED 163 is associated with a duty ratio of 0, i.e., LED off.

[0055] The gain 33 defines the gain of the imaging device 164 in five stages, for example, from 0 (minimum gain) to 4 (maximum gain). More specifically, the setting values ​​0 to 4 indicating the gain 33 are each associated with a predetermined gain of the imaging device 164.

[0056] The exposure time 34 specifies the exposure time of the imaging device 164, for example, in 20 stages from 1 (minimum exposure time) to 20 (maximum exposure time). More specifically, the setting values ​​1 to 20 indicating the exposure time 34 are each associated with a predetermined time interval of a clock pulse that controls the imaging operation of the imaging device 164.

[0057] In this embodiment, four types of IDs are assigned to the identification ID 31.

[0058] The identification ID=1 corresponds to the photographing parameters suitable for photographing a printed code symbol.

[0059] The photographing condition setting file 224 stores setting values ​​1 to 4 as the lighting brightness 32 suitable for photographing the printed code symbol, setting values ​​0 to 2 as the gain 33, and setting values ​​1 to 10 as the exposure time 34, associated with identification ID=1.

[0060] In this way, there is a range of setting values ​​for each imaging parameter even for the same identification ID = 1. The scanner 16 appropriately sets a combination of setting values ​​for each imaging parameter corresponding to the identification ID = 1.

[0061] The identification ID=2 corresponds to the photographing parameters suitable for photographing the code symbol displayed on the mobile terminal.

[0062] The photographing condition setting file 224 is associated with the identification ID=2 and stores a setting value of 0 as the lighting brightness 32 suitable for photographing a code symbol displayed on a mobile terminal, a setting value of 2 to 4 as the gain 33, and a setting value of 4 to 20 as the exposure time 34. In particular, since a code symbol displayed on a mobile terminal is generally displayed against a bright background illuminated by a backlight, it is desirable to photograph it with the LED 163 turned off.

[0063] Identification ID=3 corresponds to photographing parameters suitable for photographing products that cannot be assigned a code symbol, such as vegetables and fruits.

[0064] The shooting condition setting file 224 stores setting values ​​1 to 2 as the lighting brightness 32 suitable for photographing products such as vegetables and fruits, associated with identification ID = 3, setting value 0 as gain 33, and setting values ​​1 to 4 as exposure time 34.

[0065] The identification ID=4 corresponds to the photographing parameters suitable for photographing packaged goods.

[0066] The photographing condition setting file 224 stores setting values ​​1 to 2 as the lighting brightness 32 suitable for photographing packaged goods, setting value 0 as the gain 33, and setting values ​​1 to 2 as the exposure time 34, associated with identification ID=4.

[0067] The self-service POS terminal 10 recognizes the subject in the image by processing the image captured by the imaging device 164 using an image processing method corresponding to the identification ID 31 that indicates the capture conditions of the image. For example, if the captured image was captured under the capture conditions of identification ID=1, the self-service POS terminal 10 performs image processing on the image to recognize the printed code symbol. Note that the identification ID that indicates the capture conditions of the image is assigned for convenience, so there is no guarantee that the printed code symbol will necessarily appear in an image captured under the capture conditions of identification ID=1. In that case, the self-service POS terminal 10 may fail to recognize the printed code symbol. In that case, the self-service POS terminal 10 changes the capture conditions, i.e., the identification ID 31, and captures the same subject again.

[0068] 5, depending on the shooting parameter settings, settings corresponding to multiple identification IDs 31 may be made simultaneously. For example, if the lighting brightness 32 is set to "1," the gain 33 to "0," and the exposure time 34 to "2," the identification IDs 31 are set to "1, 3, 4." In this way, when multiple identification IDs 31 are matched, the scanner 16 of this embodiment executes all of the multiple image processes corresponding to all of the matched identification IDs 31.

[0069] (Data structure of the image captured by the scanner) The format of the image I captured by the scanner 16 will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the structure of an image captured by the scanner.

[0070] The image I captured by the scanner 16 is a so-called digital image. As shown in Fig. 6, the image I is composed of a plurality of pixels P, m pixels in the horizontal direction (x direction) and n pixels in the vertical direction (y direction). The specific number of pixels is set depending on the imaging device 164 used.

[0071] Each pixel P constituting the image I stores a value corresponding to the magnitude of the electrical signal output by the imaging device 164, quantized to, for example, 8 bits. That is, each pixel P of the image I stores a value (pixel value) from 0 to 255. Generally, the smaller the pixel value, the darker the image, and the larger the pixel value, the brighter the image. As a result, the image I forms a two-dimensional image having a grayscale distribution according to the subject being photographed.

[0072] More specifically, as shown in Fig. 6, image I is made up of m pixels P aligned horizontally (x direction) and n pixels P aligned vertically (y direction). Each pixel P is associated with a coordinate value that identifies the pixel. That is, pixel P at the upper left corner of image I has coordinates (0,0), and pixel P at the lower right corner of image I has coordinates (m-1,n-1).

[0073] Each pixel P stores the pixel value described above, so the pixel P in the upper left corner of image I stores the pixel value P(0,0), and the pixel P in the lower right corner of image I stores the pixel value P(m-1,n-1).

[0074] (Example of an image captured by a scanner) Examples of images captured by the scanner 16 will be described with reference to Figs. 7 to 10. Fig. 7 is a diagram showing an example of an image obtained by capturing a printed code symbol. Fig. 8 is a diagram showing an example of an image obtained by capturing a code symbol displayed on a mobile terminal. Fig. 9 is a diagram showing an example of an image obtained by capturing a natural object such as a vegetable or fruit. Fig. 10 is a diagram showing an example of an image obtained by capturing a packaged product.

[0075] FIG. 7 shows an image Ia obtained by capturing a barcode 42 printed on a product 41. A customer holding the product 41 holds the barcode 42 printed on the product 41 over the reading window 161 (see FIG. 1) of the scanner 16. The imaging device 164 built into the scanner 16 repeatedly captures images, and each time an image is captured, an image recognition process (described later) is performed (see FIG. 15). If the barcode 42 appears in the captured image, the barcode 42 is recognized by this image recognition process. The barcode 42 is an example of a first code symbol in the present disclosure.

[0076] FIG. 8 shows an image Ib obtained by capturing a barcode 44 displayed on the mobile terminal 43. If the product is heavy and it is difficult to hold the barcode 44 over the reading window 161, the customer captures the barcode 44 of the product using the camera of the customer's mobile terminal 43. The customer then displays the captured barcode 44 on the screen of the mobile terminal 43 and holds it over the reading window 161 of the scanner 16. The imaging device 164 built into the scanner 16 repeatedly captures images, and performs image recognition processing (described later) each time an image is captured (see FIG. 15). If the barcode 44 appears in the captured image, the barcode 44 is recognized by this image recognition processing. The barcode 44 is an example of a second code symbol in the present disclosure. Another example of a customer displaying a code symbol such as a barcode 44 on the mobile terminal 43 and having the scanner 16 read it is when the customer displays their own membership registration information, point card, coupons, etc. at the store on the mobile terminal 43 and has the scanner 16 read it.

[0077] Fig. 9 shows an image Ic of a product 45. A customer holding a product 45 such as a vegetable or fruit holds the product 45 over the reading window 161 of the scanner 16. The imaging device 164 built into the scanner 16 repeatedly captures images, and each time an image is captured, an image recognition process (described later) is performed (see Fig. 15). If the product 45 appears in the captured image, the product 45 is recognized by this image recognition process.

[0078] FIG. 10 shows an image Id of a packaged product 46. A customer holding the packaged product 46 holds it over the reading window 161 of the scanner 16. The imaging device 164 built into the scanner 16 repeatedly captures images, and performs image recognition processing (described later) each time an image is captured (see FIG. 15). If the packaged product 46 appears in the captured image, the packaged product 46 is recognized by this image recognition processing.

[0079] (How to write the identification ID) A method for writing an identification ID 31 indicating the photographing conditions to an image captured by the scanner 16 will be described with reference to Figures 11 and 12. Figure 11 is a diagram showing an example of a writing position of an identification ID indicating the photographing conditions in a captured image. Figure 12 is a diagram showing an example of a writing format of the identification ID.

[0080] The scanner 16 outputs data of the captured image I to the control unit 21 (see FIG. 4) in the format shown in Fig. 11. Specifically, the data of the image I includes header information followed by pixel values ​​P(x, y) stored in the pixels P that form the image I, which are output serially from left to right and from top to bottom.

[0081] The header information may include information indicating that the following data is data for image I, and also information indicating the size of image I.

[0082] Then, in the area where the pixel value P(m-1, n-1) in the lower right corner of image I is stored, the identification ID 31 is written instead of the pixel value P(m-1, n-1).

[0083] In this way, by writing the identification ID 31, the pixel value of pixel P in the lower right corner of image I is lost, but since only one pixel in the corner of image I is lost, there is no effect on subsequent recognition processing.

[0084] Then, in the area where the pixel value P(m-1, n-1) is originally stored, the identification ID 31 is written in the format shown in FIG.

[0085] Specifically, the area in which the pixel value P(m-1, n-1) is stored originally has a capacity of 8 bits, but information indicating whether the image I corresponds to the identification ID = 1, 2, 3, or 4 described in Figure 5 is written in the lower 4 bits or the upper 4 bits of the 8-bit area.

[0086] For example, bit 4, which is the most significant bit of the 4-bit area in which the identification ID 31 is written, stores information indicating whether the shooting parameters of image I correspond to identification ID=1. Specifically, if the shooting parameters of image I correspond to identification ID=1, "1" is written to bit 4. On the other hand, if the shooting parameters of image I do not correspond to identification ID=1, "0" is written to bit 4.

[0087] Similarly, bit 3, which is the second most significant bit of the 4-bit area in which identification ID 31 is written, stores information indicating whether the shooting parameters of image I correspond to identification ID=2. Specifically, if the shooting parameters of image I correspond to identification ID=2, "1" is written to bit 3. On the other hand, if the shooting parameters of image I do not correspond to identification ID=2, "0" is written to bit 3.

[0088] Furthermore, bit 2, which is the third most significant bit of the 4-bit area in which the identification ID 31 is written, stores information indicating whether the shooting parameters of image I correspond to identification ID=3. Specifically, if the shooting parameters of image I correspond to identification ID=3, "1" is written to bit 2. On the other hand, if the shooting parameters of image I do not correspond to identification ID=3, "0" is written to bit 2.

[0089] Then, information indicating whether the shooting parameters of image I correspond to identification ID=4 is stored in bit 1, which is the least significant bit of the 4-bit area in which identification ID 31 is written. Specifically, if the shooting parameters of image I correspond to identification ID=4, "1" is written in bit 1. On the other hand, if the shooting parameters of image I do not correspond to identification ID=4, "0" is written in bit 1.

[0090] Therefore, for example, in the shooting condition setting file 224 shown in Figure 5, if the lighting brightness 32 is set to "1", the gain 33 to "0", and the exposure time 34 to "2", the identification ID 31 will be set to "1, 3, 4", and therefore "1011" will be written in the 4-bit area where the identification ID 31 is written.

[0091] (Functional configuration of self-service POS terminal) The functional configuration of the self-checkout POS terminal 10 of the embodiment will be described with reference to Fig. 13. Fig. 13 is a functional block diagram showing an example of the functional configuration of the self-checkout POS terminal of the embodiment.

[0092] The control unit 21 of the self-checkout POS terminal 10 deploys the control program 221 in the RAM 213 and runs it to realize, as functional units, an image capturing unit 51, an image capturing condition setting unit 52, an image capturing condition writing unit 53, an image recognition unit 54, a recognition result determination unit 55, a product registration unit 56, and a payment processing unit 57, all of which are shown in Fig. 13. Note that some or all of these functional units may be realized by dedicated hardware.

[0093] The image capturing unit 51 captures an image including a subject held over the reading window 161 of the scanner 16. Specifically, the image capturing unit 51 repeatedly captures images after receiving an instruction to start product registration. The image capturing unit 51 may capture an image that does not include the subject product, but such an image is discarded in the subsequent image recognition process (see FIG. 15). In this way, after capturing an image, the image capturing unit 51 captures another image under capturing conditions according to the results of the image recognition process in which the image recognition unit 54, described later, recognizes the subject included in the image I. Note that the image capturing unit 51 does not necessarily need to determine that the subject product has been held over it.

[0094] The photographing condition setting unit 52 sets the photographing conditions for the image photographing unit 51 to one of a plurality of preset conditions. Specifically, the photographing condition setting unit 52 sets a combination of the illumination brightness 32, gain 33, and exposure time 34 defined in the photographing condition setting file 224 (see FIG. 5) to one of the identification IDs = 1, 2, 3, and 4. Note that, although a plurality of combinations of the illumination brightness 32, gain 33, and exposure time 34 corresponding to the same identification ID can be set, the combinations may be set as appropriate.

[0095] Furthermore, the photographing condition setting unit 52 selects all of the identification IDs 31, i.e., all of the objects to be read, that correspond to the combination of the set illumination brightness 32, gain 33, and exposure time 34. Specifically, the photographing condition setting unit 52 selects all of the identification IDs 31 that correspond to the combination of the set illumination brightness 32, gain 33, and exposure time 34 by referring to the registered contents of the photographing condition setting file 224.

[0096] The photographing condition writing unit 53 writes the identification ID 31 corresponding to the photographing conditions set by the photographing condition setting unit 52 to the image I photographed by the image photographing unit 51. The method for writing the identification ID 31 is as described above, and the identification ID 31 is written at a predetermined position in the image I photographed by the image photographing unit 51.

[0097] When the photographing conditions are applicable to a plurality of different types of subjects, the photographing condition writing unit 53 writes a plurality of pieces of identification information corresponding to the subjects into the image I photographed by the image photographing unit 51.

[0098] The image recognition unit 54 reads the identification ID 31 written in the image I captured by the image capturing unit 51, and determines which of the identification IDs 31 the capturing conditions of the image I correspond to.

[0099] The image recognition unit 54 reads the identification ID 31 written in the image I captured by the image capturing unit 51, and recognizes the subject contained in the image I by performing different image recognition processing according to the capturing conditions corresponding to the identification ID 31.

[0100] More specifically, the image recognition unit 54 performs image processing to read a printed code symbol (for example, the barcode 42 in FIG. 7) when an identification ID=1 corresponding to the photographing parameters suitable for photographing the code symbol is written in the image I. Specifically, the image recognition unit 54 performs image processing provided in a known barcode reader.

[0101] Furthermore, the image recognition unit 54 performs image processing to read a code symbol displayed on the mobile terminal (for example, the barcode 44 in FIG. 8) when an identification ID=2 corresponding to the photographing parameters suitable for photographing the code symbol is written in the image I. Specifically, the image recognition unit 54 performs image processing provided in a known barcode reader.

[0102] In addition, when an identification ID=3 corresponding to the photographing parameters suitable for photographing a product such as a vegetable or fruit (for example, product 45 in Figure 9) is written in image I, the image recognition unit 54 performs image processing to recognize natural objects such as vegetables and fruits.

[0103] Furthermore, when an identification ID=4 corresponding to the photographing parameters suitable for photographing a packaged product (for example, packaged product 46 in FIG. 10) is written in image I, the image recognition unit 54 performs image processing to recognize the packaged product.

[0104] If a plurality of identification IDs 31 are written in the image I, the image recognition unit 54 performs a plurality of different image processes corresponding to the identification IDs 31.

[0105] The image recognition unit 54 also calculates a recognition score that indicates the reliability of the recognition result. The recognition score takes a larger value as the degree of match between the feature extracted from the image I and the feature registered in the commodity recognition dictionary 223 increases, and the larger the recognition score, the more reliable the recognition result.

[0106] The recognition result determination unit 55 determines whether the image recognition unit 54 has recognized the subject appearing in the image I. The recognition result determination unit 55 is an example of a determination unit in the present disclosure. Specifically, the recognition result determination unit 55 determines that the recognition has been successful when the recognition score calculated by the image recognition unit 54 is equal to or greater than a predetermined threshold.

[0107] The product registration unit 56 performs a product registration process to register the product code corresponding to the product recognized by the image recognition unit 54 in the product registration file 225. The product registration unit 56 also determines whether the customer has instructed the start of product registration. Furthermore, the product registration unit 56 determines whether the customer has instructed the end of product registration.

[0108] The payment processing unit 57 receives instructions on the payment method from the customer. The payment processing unit 57 then performs payment processing for the product registered in the product registration file 225 using the specified payment method. The payment processing unit 57 then generates a payment file 226 that stores the results of the payment processing.

[0109] (Processing flow of self-service POS terminal) The flow of processing performed by the self-service POS terminal 10 of the embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the flow of processing performed by the self-service POS terminal of the embodiment.

[0110] The product registration unit 56 determines whether or not there is an instruction from the customer to start product registration (step S11). If it is determined that there is an instruction from the customer to start product registration (step S11: Yes), the process proceeds to step S12. On the other hand, if it is not determined that there is an instruction from the customer to start product registration (step S11: No), step S11 is repeated. Whether or not there is an instruction to start product registration is determined by, for example, pressing an operator such as "Start product registration" displayed on the display device 14.

[0111] In step S11, if it is determined that the customer has instructed to start product registration, the image recognition unit 54 performs image recognition processing (step S12) to recognize subjects (code symbols such as barcodes, vegetables and fruits, packaged products) contained in the image captured by the scanner 16. The detailed flow of the image recognition processing will be described later (see FIG. 15).

[0112] The product registration unit 56 performs a product registration process to register the product recognized by the image recognition unit 54 in the product registration file 225 (step S13).

[0113] The product registration unit 56 determines whether the customer has instructed that product registration has been completed (step S14). If it is determined that the customer has instructed that product registration has been completed (step S14: Yes), the process proceeds to step S15. On the other hand, if it is not determined that the customer has instructed that product registration has been completed (step S14: No), the process returns to step S12.

[0114] If it is determined in step S14 that the customer has instructed that the product registration has been completed, the payment processing unit 57 performs payment processing for the product registered in the product registration file 225 using the payment method instructed by the customer (step S15). After that, the self-checkout POS terminal 10 ends the processing.

[0115] (Image recognition process performed by self-service POS terminals) The flow of the image recognition process performed by the self-checkout POS terminal 10 of the embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the flow of the image recognition process performed by the self-checkout POS terminal of the embodiment.

[0116] The photographing condition setting unit 52 sets photographing conditions when the image photographing unit 51 photographs (step S21). Specifically, the photographing condition setting unit 52 sets a combination of illumination brightness 32, gain 33, and exposure time 34 defined in the photographing condition setting file 224 (see FIG. 5) to one of the identification IDs = 1, 2, 3, and 4.

[0117] Next, the photographing condition setting unit 52 sets the LED 163 to be illuminated at the illumination brightness 32 corresponding to the identification ID 31 set in step S21. The photographing condition setting unit 52 also sets the imaging device 164 to operate at the gain 33 and exposure time 34 corresponding to the identification ID 31 set in step S21 (step S22).

[0118] Furthermore, the photographing condition setting unit 52 selects all the identification IDs 31, ie, all the objects to be read, that correspond to the combination of the illumination brightness 32, gain 33, and exposure time 34 set in step S21 (step S23).

[0119] The image capturing unit 51 captures an image I using the imaging device 164 (step S24).

[0120] The photographing condition writing unit 53 writes the photographing conditions set by the photographing condition setting unit 52 at a predetermined position of the image I (step S25).

[0121] The image recognition unit 54 determines whether the image I is an image of a shooting condition corresponding to the printed code symbol (step S26). If it is determined that the image I is an image of a shooting condition corresponding to the printed code symbol (step S26: Yes), the process proceeds to step S27. On the other hand, if it is not determined that the image I is an image of a shooting condition corresponding to the printed code symbol (step S26: No), the process proceeds to step S29. Note that the image recognition unit 54 determines that the image I is an image of a shooting condition corresponding to the printed code symbol when an identification ID=1 corresponding to the shooting parameters suitable for photographing the printed code symbol is written in the pixel in the lower right corner of the image I.

[0122] If it is determined in step S26 that image I is an image captured under conditions corresponding to the printed code symbol, the image recognition unit 54 performs image processing on image I to recognize the printed code symbol (step S27).

[0123] The recognition result determination unit 55 determines whether the image recognition unit 54 has succeeded in the recognition (step S28). If it is determined that the image recognition unit 54 has succeeded in the recognition (step S28: Yes), the process proceeds to step S38. On the other hand, if it is determined that the image recognition unit 54 has not succeeded in the recognition (step S28: No), the process proceeds to step S29. Note that the recognition result determination unit 55 determines that the recognition has succeeded if the recognition score calculated by the image recognition unit 54 is equal to or greater than a predetermined threshold.

[0124] If it is determined in step S28 that the image recognition unit 54 has not succeeded in the recognition, or if it is determined in step S26 that the image I is not an image of a photographing condition corresponding to the printed code symbol, the image recognition unit 54 determines whether the image I is an image of a photographing condition corresponding to the code symbol displayed on the mobile terminal (step S29). If it is determined that the image I is an image of a photographing condition corresponding to the code symbol displayed on the mobile terminal (step S29: Yes), the process proceeds to step S30. On the other hand, if it is not determined that the image I is an image of a photographing condition corresponding to the code symbol displayed on the mobile terminal (step S29: No), the process proceeds to step S32. Note that if an identification ID=2 corresponding to the photographing parameters suitable for photographing the code symbol displayed on the mobile terminal is written in the pixel in the lower right corner of the image I, the image recognition unit 54 determines that the image I is an image of a photographing condition corresponding to the code symbol displayed on the mobile terminal.

[0125] In step S29, if it is determined that image I is an image captured under conditions corresponding to the code symbol displayed on the mobile terminal, the image recognition unit 54 performs image processing on image I to recognize the code symbol displayed on the mobile terminal (step S30).

[0126] The recognition result determination unit 55 determines whether the image recognition unit 54 has succeeded in recognition (step S31). If it is determined that the image recognition unit 54 has succeeded in recognition (step S31: Yes), the process proceeds to step S38. On the other hand, if it is determined that the image recognition unit 54 has not succeeded in recognition (step S31: No), the process proceeds to step S32. Note that the recognition result determination unit 55 determines that the recognition has succeeded if the recognition score calculated by the image recognition unit 54 is equal to or greater than a predetermined threshold.

[0127] If it is determined in step S31 that the image recognition unit 54 has not succeeded in the recognition, or if it is determined in step S29 that the image I is not an image taken under shooting conditions corresponding to the code symbol displayed on the mobile terminal, the image recognition unit 54 determines whether the image I is an image taken under shooting conditions corresponding to a natural object, such as a vegetable or fruit (step S32). If it is determined that the image I is an image taken under shooting conditions corresponding to a natural object (step S32: Yes), the process proceeds to step S33. On the other hand, if it is not determined that the image I is an image taken under shooting conditions corresponding to a natural object (step S32: No), the process proceeds to step S35. Note that if an identification ID=3 corresponding to shooting parameters suitable for photographing a natural object, such as a vegetable or fruit, is written in the pixel in the lower right corner of the image I, the image recognition unit 54 determines that the image I is an image taken under shooting conditions corresponding to a natural object, such as a vegetable or fruit.

[0128] If it is determined in step S32 that image I is an image captured under conditions corresponding to natural objects, image recognition unit 54 performs image processing on image I to recognize natural objects such as vegetables and fruits (step S33).

[0129] The recognition result determination unit 55 determines whether the image recognition unit 54 has succeeded in the recognition (step S34). If it is determined that the image recognition unit 54 has succeeded in the recognition (step S34: Yes), the process proceeds to step S38. On the other hand, if it is determined that the image recognition unit 54 has not succeeded in the recognition (step S34: No), the process proceeds to step S35. Note that the recognition result determination unit 55 determines that the recognition has succeeded if the recognition score calculated by the image recognition unit 54 is equal to or greater than a predetermined threshold.

[0130] If it is determined in step S34 that the image recognition unit 54 has not succeeded in the recognition, the image recognition unit 54 determines whether image I is an image taken under shooting conditions that correspond to a packaged product (step S35). If it is determined that image I is an image taken under shooting conditions that correspond to a packaged product (step S35: Yes), the process proceeds to step S36. On the other hand, if it is not determined that image I is an image taken under shooting conditions that correspond to a packaged product (step S35: No), the process returns to step S21. Thereafter, the settings of the shooting conditions are changed, and the series of processes described above are repeated. Note that if an identification ID=4 corresponding to shooting parameters suitable for photographing a packaged product is written in the pixel in the lower right corner of image I, the image recognition unit 54 determines that image I is an image taken under shooting conditions that correspond to a packaged product.

[0131] If it is not determined that image I is an image of a packaged product under photography conditions, the process returns to step S21 and changes the photography condition settings in a predetermined order. The photography condition settings may be changed in the order of, for example, ID=1, ID=2, ID=3, and ID=4, or the combination of lighting brightness 32, gain 33, and exposure time 34 corresponding to the same ID may be changed.

[0132] If it is determined in step S35 that image I is an image captured under conditions corresponding to the packaged product, the image recognition unit 54 performs image processing on image I to recognize the packaged product (step S36).

[0133] The recognition result determination unit 55 determines whether the image recognition unit 54 has succeeded in the recognition (step S37). If it is determined that the image recognition unit 54 has succeeded in the recognition (step S37: Yes), the process proceeds to step S38. On the other hand, if it is determined that the image recognition unit 54 has not succeeded in the recognition (step S37: No), the process returns to step S21. Note that the recognition result determination unit 55 determines that the recognition has succeeded if the recognition score calculated by the image recognition unit 54 is equal to or greater than a predetermined threshold. Thereafter, the self-checkout POS terminal 10 changes the settings of the photographing conditions and repeats the series of processes described above.

[0134] If it is determined in steps S28, S31, S34, and S37 that the image recognition unit 54 has succeeded in the recognition, the image recognition unit 54 outputs the product code of the recognized product (step S38). After that, the process returns to the main routine of FIG.

[0135] When performing image recognition of natural objects such as vegetables and fruits or packaged goods, a procedure may be performed to improve recognition accuracy by taking multiple images of the same subject and accumulating the recognition scores of the recognition results for each image. Although not shown in the flowchart of Fig. 15, when recognizing images taken under shooting conditions corresponding to natural objects or packaged goods, after successful recognition, it is possible to take images repeatedly under the same shooting conditions.

[0136] 15, the customer may have the scanner 16 read a code symbol of a point card, membership card, coupon, or the like displayed on the mobile terminal at any time during product registration. The image recognition unit 54 recognizes the membership information, point information, coupon information, and the like presented by the customer, provided that it has successfully recognized the code symbol displayed on the mobile terminal. The self-service POS terminal 10 then manages the customer's purchase history by associating the read membership information with the product registration information. Furthermore, when performing payment processing, the self-service POS terminal 10 refers to the read point information, coupon information, and the like to calculate the payment amount and update the point information.

[0137] (Modification of the embodiment) Next, a description will be given of a self-service POS terminal 90, which is a modified example of the self-service POS terminal 10 described as an embodiment. The self-service POS terminal 90 recognizes natural objects such as vegetables and fruits and packaged goods, i.e., general object recognition, using an image processing device 96 connected to the self-service POS terminal.

[0138] In general, generic object recognition requires a large amount of calculation, so it is desirable to perform it using high-performance image processing equipment equipped with a dedicated GPU (Graphics Processing Unit).

[0139] (Hardware configuration of self-service POS terminal) The hardware configuration of the self-checkout POS terminal 90 will be described with reference to Fig. 16. Fig. 16 is a hardware block diagram showing an example of the hardware configuration of the self-checkout POS terminal according to a modified example of the embodiment.

[0140] The self-service POS terminal 90 has a configuration in which a control unit 91, a storage unit 92, a peripheral device controller 94, and a communication controller 95 are connected to one another via an internal bus 93. The configuration of the control unit 91 and the configuration of the peripheral devices connected to the peripheral device controller 94 are the same as those of the self-service POS terminal 10 described in the embodiment, and therefore will not be described again.

[0141] The storage unit 92 stores a control program 921 , a product master 222 , a photography condition setting file 224 , a product registration file 225 , and a payment file 226 .

[0142] The product master 222, the photography condition setting file 224, the product registration file 225, and the payment file 226 are the same as the files stored in the self-checkout POS terminal 10, and therefore will not be described again.

[0143] The control program 921 is a program that controls the overall operation of the self-checkout POS terminal 90. The control program 921 is a computer program that causes the control unit 91 to realize functions (see FIG. 17) that will be described later.

[0144] The control unit 91 communicates with a store server (not shown) via the communication controller 95. The self-service POS terminal 90 acquires the latest product master 222 from the store server.

[0145] The control unit 91 also communicates with an image processing device 96 via a communication controller 95. The image processing device 96 is a dedicated device for recognizing images photographed under conditions corresponding to natural objects such as vegetables and fruits, or images photographed under conditions corresponding to packaged goods.

[0146] The image processing device 96 stores the product master 222 and the product recognition dictionary 223, both of which are described above.

[0147] When the captured image I is an image captured under shooting conditions corresponding to natural objects such as vegetables and fruits, or an image captured under shooting conditions corresponding to packaged goods, the control unit 91 transmits the captured image I to the image processing device 96 via the communication controller 95.

[0148] The image processing device 96 receives the image I. Then, it performs general object recognition on the received image I to recognize natural objects such as vegetables and fruits or packaged products that appear in the image I.

[0149] Furthermore, the image processing device 96 transmits the product code of the recognized product and the recognition score of the recognition result to the control unit 91 via the communication controller 95.

[0150] (Functional configuration of self-service POS terminal) The functional configuration of the self-checkout POS terminal 90 according to the modified example of the embodiment will be described with reference to Fig. 17. Fig. 17 is a functional block diagram showing an example of the functional configuration of the self-checkout POS terminal according to the modified example of the embodiment.

[0151] The control unit 91 of the self-service POS terminal 90 deploys and operates the control program 921 in the RAM 913, thereby realizing, as functional units, an image capturing unit 101, an image capturing condition setting unit 102, an image capturing condition writing unit 103, an image transmitting unit 104, an image recognizing unit 105, a recognition result receiving unit 106, a recognition result determining unit 107 (determination unit), a product registration unit 108, and a payment processing unit 109, all of which are shown in Fig. 17. Note that some or all of these functional units may be realized by dedicated hardware.

[0152] Of the functional parts of the control unit 91, the functions of the image capturing unit 101, the capturing condition setting unit 102, the capturing condition writing unit 103, the recognition result determination unit 107, the product registration unit 108, and the payment processing unit 109 are the same as the functions of the functional parts of the same name realized by the control unit 21 of the self-POS terminal 10 described above, so a repeated explanation will be omitted.

[0153] The image transmitting unit 104 transmits the image I to the image processing device 96 when the image I captured by the image capturing unit 101 is an image captured under conditions corresponding to natural objects such as vegetables and fruits, or a captured image under conditions corresponding to packaged goods.

[0154] The image recognition unit 105 reads the identification ID 31 written in the image I captured by the image capturing unit 101, and determines which of the identification IDs 31 the capturing conditions of the image I correspond to.

[0155] In addition, the image recognition unit 105 reads the identification ID 31 written in the image I captured by the image capturing unit 51, and performs image processing to recognize the code symbol if the identification ID 31 corresponds to the capturing parameters suitable for capturing a printed code symbol or the capturing parameters suitable for capturing a code symbol displayed on a mobile terminal.

[0156] Furthermore, the image recognition unit 105 calculates a recognition score that indicates the likelihood of the recognition result of the code symbol.

[0157] The recognition result receiving unit 106 receives from the image processing device 96 the product ID recognized by the image processing device 96 and a recognition score indicating the accuracy of the recognition result.

[0158] 17. The image processing device 96 also implements, as functional units, an image receiving unit 111, an image recognition unit 112, and a recognition result transmission unit 113. Note that some or all of these functional units may be implemented by dedicated hardware.

[0159] The image receiving unit 111 receives the image I transmitted by the image transmitting unit 104 .

[0160] The image recognition unit 112 performs image processing to recognize natural objects such as vegetables and fruits when an identification ID=3 corresponding to photographing parameters suitable for photographing a product such as vegetables or fruits (for example, product 45 in Figure 9) is written in the image I received by the image receiving unit 111.

[0161] Furthermore, when an identification ID=4 corresponding to the photographing parameters suitable for photographing a packaged product (for example, packaged product 46 in FIG. 10) is written in the image I received by the image receiving unit 111, the image recognition unit 112 performs image processing to recognize the packaged product.

[0162] Furthermore, the image recognition unit 112 calculates a recognition score that indicates the likelihood of the recognition result.

[0163] The recognition result transmission unit 113 transmits the product ID recognized by the image recognition unit 112 and the recognition score to the recognition result reception unit 106.

[0164] (Processing flow of self-service POS terminal) The flow of processing performed by the self-service POS terminal 90 is the same as the flow of processing performed by the self-service POS terminal 10 described above (see FIG. 14), and therefore will not be described again. However, since only the content of the image recognition processing performed in step S12 in FIG. 14 is different, the content of the image recognition processing will be described later (see FIGS. 18 and 19).

[0165] (Image recognition process performed by self-service POS terminals) The flow of image recognition processing performed by the self-service POS terminal 90 according to the modified embodiment will be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a flowchart showing an example of the flow of image recognition processing performed by the self-service POS terminal according to the modified embodiment. Fig. 19 is a flowchart showing an example of the flow of image recognition processing performed by the image processing device according to the modified embodiment.

[0166] First, the flow of the image recognition process performed by the self-checkout POS terminal 90 will be described with reference to FIG.

[0167] The photographing condition setting unit 102 sets photographing conditions when the image photographing unit 101 photographs (step S41). Specifically, the photographing condition setting unit 102 sets a combination of illumination brightness 32, gain 33, and exposure time 34 defined in the photographing condition setting file 224 (see FIG. 5) to one of identification IDs = 1, 2, 3, and 4.

[0168] Next, the photographing condition setting unit 102 sets the LED 163 to be illuminated at the illumination brightness 32 corresponding to the identification ID 31 set in step S41. The photographing condition setting unit 52 also sets the imaging device 164 to operate at the gain 33 and exposure time 34 corresponding to the identification ID 31 set in step S41 (step S42).

[0169] Furthermore, the photographing condition setting unit 102 selects all the identification IDs 31, ie, all the objects to be read, that correspond to the combination of the illumination brightness 32, gain 33, and exposure time 34 set in step S41 (step S43).

[0170] The image capturing unit 101 captures an image I using the imaging device 164 (step S44).

[0171] The photographing condition writing unit 103 writes the photographing conditions set by the photographing condition setting unit 102 at a predetermined position in the image I (step S45).

[0172] The image recognition unit 105 determines whether the image I is an image of a shooting condition corresponding to the printed code symbol (step S46). If it is determined that the image I is an image of a shooting condition corresponding to the printed code symbol (step S46: Yes), the process proceeds to step S47. On the other hand, if it is not determined that the image I is an image of a shooting condition corresponding to the printed code symbol (step S46: No), the process proceeds to step S49. Note that the image recognition unit 105 determines that the image I is an image of a shooting condition corresponding to the printed code symbol when an identification ID=1 corresponding to the shooting parameters suitable for photographing the printed code symbol is written in the pixel in the lower right corner of the image I.

[0173] If it is determined in step S46 that image I is an image captured under conditions corresponding to the printed code symbol, image recognition unit 105 performs image processing on image I to recognize the printed code symbol (step S47).

[0174] The recognition result determination unit 107 determines whether the image recognition unit 105 has succeeded in the recognition (step S48). If it is determined that the image recognition unit 105 has succeeded in the recognition (step S48: Yes), the process proceeds to step S55. On the other hand, if it is determined that the image recognition unit 105 has not succeeded in the recognition (step S48: No), the process proceeds to step S49. Note that the recognition result determination unit 107 determines that the recognition has succeeded if the recognition score calculated by the image recognition unit 105 is equal to or greater than a predetermined threshold.

[0175] If it is determined in step S48 that the image recognition unit 54 has not succeeded in the recognition, or if it is determined in step S46 that the image I is not an image of a photographing condition corresponding to the printed code symbol, the image recognition unit 105 determines whether the image I is an image of a photographing condition corresponding to the code symbol displayed on the mobile terminal (step S49). If it is determined that the image I is an image of a photographing condition corresponding to the code symbol displayed on the mobile terminal (step S49: Yes), the process proceeds to step S50. On the other hand, if it is not determined that the image I is an image of a photographing condition corresponding to the code symbol displayed on the mobile terminal (step S49: No), the process proceeds to step S52. Note that if an identification ID=2 corresponding to the photographing parameters suitable for photographing the code symbol displayed on the mobile terminal is written in the pixel in the lower right corner of the image I, the image recognition unit 105 determines that the image I is an image of a photographing condition corresponding to the code symbol displayed on the mobile terminal.

[0176] In step S49, if it is determined that image I is an image captured under conditions corresponding to the code symbol displayed on the mobile terminal, image recognition unit 105 performs image processing on image I to recognize the code symbol displayed on the mobile terminal (step S50).

[0177] The recognition result determination unit 107 determines whether the image recognition unit 105 has succeeded in the recognition (step S51). If it is determined that the image recognition unit 105 has succeeded in the recognition (step S51: Yes), the process proceeds to step S55. On the other hand, if it is determined that the image recognition unit 105 has not succeeded in the recognition (step S51: No), the process proceeds to step S52. Note that the recognition result determination unit 107 determines that the recognition has succeeded if the recognition score calculated by the image recognition unit 105 is equal to or greater than a predetermined threshold.

[0178] If it is determined in step S51 that the image recognition unit 105 has not succeeded in the recognition, or if it is determined in step S49 that the image I is not an image captured under conditions corresponding to the code symbol displayed on the mobile terminal, the image transmission unit 104 transmits the image I to the image processing device 96 (step S52). Thereafter, the image processing device 96 performs object recognition on the image I, and the flow of the processing performed at that time will be described later (see FIG. 19).

[0179] The recognition result receiving unit 106 receives, from the image processing device 96, the product ID recognized by the image processing device 96 and a recognition score indicating the accuracy of the recognition result (step S53).

[0180] The recognition result determination unit 107 determines whether the image processing device 96 has succeeded in the recognition (step S54). If it is determined that the image processing device 96 has succeeded in the recognition (step S54: Yes), the process proceeds to step S55. On the other hand, if it is determined that the image processing device 96 has not succeeded in the recognition (step S54: No), the process returns to step S41. Note that the recognition result determination unit 107 determines that the recognition has succeeded if the recognition score calculated by the image processing device 96 is equal to or greater than a predetermined threshold. Thereafter, the self-service POS terminal 90 changes the settings of the photographing conditions and repeats the series of processes described above.

[0181] If it is determined in step S54 that the image processing device 96 has not succeeded in the recognition, the process returns to step S41, and the settings of the photographing conditions are changed in a predetermined order. The order in which the settings of the photographing conditions are changed may be, for example, ID=1, ID=2, ID=3, and ID=4, or the combination of the illumination brightness 32, gain 33, and exposure time 34 corresponding to the same ID may be changed.

[0182] If it is determined in steps S48, S51, and S54 that the recognition is successful, the image recognition unit 105 outputs the product code of the recognized product (step S55), and then returns to the main routine (see FIG. 14) described above.

[0183] Next, the flow of the image recognition process performed by the image processing device 96 will be described with reference to FIG.

[0184] The image receiving unit 111 determines whether image I has been received (step S61). If it is determined that image I has been received (step S61: Yes), the process proceeds to step S62. On the other hand, if it is not determined that image I has been received (step S61: No), step S61 is repeated.

[0185] If it is determined in step S61 that image I has been received, image recognition unit 112 determines whether image I is an image taken under conditions corresponding to a natural object such as a vegetable or fruit (step S62). If it is determined that image I is an image taken under conditions corresponding to a natural object (step S62: Yes), the process proceeds to step S63. On the other hand, if it is not determined that image I is an image taken under conditions corresponding to a natural object (step S62: No), the process proceeds to step S66.

[0186] If it is determined in step S62 that image I is an image captured under conditions corresponding to natural objects, image recognition unit 112 performs image processing on image I to recognize natural objects such as vegetables and fruits (step S63).

[0187] The image recognition unit 112 determines whether the recognition of the natural object has been successful (step S64). If it is determined that the recognition of the natural object has been successful (step S64: Yes), the process proceeds to step S65. On the other hand, if it is determined that the recognition of the natural object has not been successful (step S64: No), the process proceeds to step S66. Note that the image recognition unit 112 determines that the recognition has been successful if the recognition score calculated as a result of object recognition is equal to or greater than a predetermined threshold value.

[0188] If it is determined in step S64 that the natural object has not been successfully recognized, the image recognition unit 112 performs image processing for recognizing packaged goods on the image I (step S66). Then, the process proceeds to step S65.

[0189] If it is determined in step S64 that the natural object has been successfully recognized, or following step S66, the recognition result transmission unit 113 transmits the product ID and recognition score recognized by the image recognition unit 112 to the recognition result reception unit 106 (step S65). Thereafter, the image processing device 96 ends the process.

[0190] (Effects of the embodiment) As described above, the self-service POS terminal 10 (image recognition device) of the embodiment includes an image capturing unit 51 that captures an image I including a subject held up by an operator, a capturing condition setting unit 52 that sets the capturing conditions for the image capturing unit 51 to one of a plurality of preset conditions, a capturing condition writing unit 53 that writes an identification ID 31 (identification information) corresponding to the capturing conditions set by the capturing condition setting unit 52 into the image captured by the image capturing unit 51, an image recognition unit 54 that recognizes the subject included in the image I by reading the identification ID 31 written in the image captured by the image capturing unit 51 and performing different image processing according to the capturing conditions corresponding to the identification ID 31, and a recognition result determination unit 55 (determination unit) that determines whether the image recognition unit 54 has recognized the subject. If the recognition result determination unit 55 does not determine that the subject has been recognized, the capturing condition setting unit 52 changes the capturing conditions and causes the image capturing unit 51 to capture an image again. Therefore, the subject to be recognized can be captured under appropriate capturing conditions according to the object to be recognized.

[0191] Furthermore, in the self-service POS terminal 10 (image recognition device) of the embodiment, the photographing conditions include at least one of the brightness 32 of the lighting that illuminates the subject, the gain 33 of the imaging device 164 that captures the image I, and the exposure time 34 of the imaging device 164. Therefore, the photographing conditions can be set according to the type of subject.

[0192] In the self-service POS terminal 10 (image recognition device) of the embodiment, the subject includes at least one of the barcode 42 (first code symbol) attached to the product 41, the appearance of the product 45 or packaged product 46, and the barcode 44 (second code symbol) displayed on the mobile terminal 43 carried by the operator. Therefore, appropriate photographing conditions can be set according to the variations of the product to be registered.

[0193] Furthermore, in the self-service POS terminal 10 (image recognition device) of the embodiment, the photographing condition writing unit 53 writes the identification ID 31 (identification information) at a predetermined position of the image I captured by the image photographing unit 51. Therefore, the identification ID 31 indicating the photographing conditions of the image I can be written without affecting the state of the subject appearing in the image I.

[0194] Furthermore, in the self-service POS terminal 10 (image recognition device) of the embodiment, when the photographing conditions are applicable to a plurality of different types of subjects, the photographing condition writing unit 53 writes a plurality of identification IDs 31 (identification information) corresponding to the subjects to the image I photographed by the image photographing unit 51. Therefore, even if the set photographing conditions are suitable for photographing a plurality of different subjects, it is possible to write all of the identification IDs 31 indicating the photographing conditions of the image I.

[0195] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as set forth in the claims. [Explanation of symbols]

[0196] 10 Self-service POS terminal (image recognition device) 11. Housing 12,13 Product stand 14 Display Devices 15 Control Devices 16 Scanner 17 Payment terminal 18 Receipt Printer 19 Warning light 31 Identification ID (identification information) 32 Lighting brightness 33 Gain 34 Exposure time 41 items 42 Barcode (first code symbol) 43 Mobile devices 44 Barcode (secondary code symbol) 45 items 46 Packaged Products 51 Image capture unit 52 Shooting condition setting section 53 Shooting condition writing section 54 Image Recognition Unit 55 Recognition result judgment section (judgment section) 56 Product Registration Department 57 Payment processing section 96 Image Processing Device 101 Image capture unit 102 Shooting condition setting section 103 Shooting condition writing section 104 Image transmission unit 105 Image Recognition Unit 106 Recognition result receiving unit 107 Recognition result judgment section (judgment section) 108 Product Registration Department 109 Payment Processing Unit 111 Image receiving unit 112 Image Recognition Unit 113 Recognition result transmission unit 161 Reading window 162 Case 163 LED 164 Imaging Device 165 Control Board 166 Opening 167 Lens 168 Diffusion Filter 171 Display Devices 172 10-key 173 Card Reader 174 Card slot 221 Control Program 222 Product Master 223 Product Recognition Dictionary 224 Shooting condition setting file 225 Product Registration File 226 Payment File I, Ia, Ib, Ic, Id Images P pixel [Prior art documents] [Patent documents]

[0197] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-89084

Claims

1. an image capturing unit that captures an image including a subject held over by an operator; an imaging condition setting unit that sets the imaging conditions of the image capturing unit to one of a plurality of preset conditions; an imaging condition writing unit that writes identification information corresponding to the imaging conditions set by the imaging condition setting unit into the image captured by the image capturing unit; an image recognition unit that recognizes the subject included in the image by reading the identification information written in the image captured by the image capturing unit and performing different image processing according to the capturing conditions corresponding to the identification information; a determination unit that determines whether the image recognition unit has recognized the subject, On the condition that the determination unit does not determine that the subject has been recognized, the photographing condition setting unit changes the photographing conditions and causes the image photographing unit to photograph an image again. Image recognition device.

2. The imaging conditions are: the brightness of the illumination illuminating the subject, the gain of an imaging device capturing the image, and at least one of the exposure time of the imaging device; The image recognition device according to claim 1 .

3. the subject includes at least one of a first code symbol affixed to a product, an appearance of the product, and a second code symbol displayed on a mobile terminal carried by the operator; The image recognition device according to claim 1 .

4. the photographing condition writing unit writes the identification information at a predetermined position of the image photographed by the image photographing unit; The image recognition device according to claim 1 .

5. the photographing condition writing unit writes, when the photographing conditions are applicable to a plurality of different types of subjects, a plurality of pieces of identification information corresponding to the subjects into the image photographed by the image photographing unit; The image recognition device according to claim 4.

6. The computer that controls the image recognition device an image capturing unit that captures an image including a subject held over by an operator; an imaging condition setting unit that sets the imaging conditions of the image capturing unit to one of a plurality of preset conditions; an imaging condition writing unit that writes identification information corresponding to the imaging conditions set by the imaging condition setting unit into the image captured by the image capturing unit; an image recognition unit that recognizes the subject included in the image by reading the identification information written in the image captured by the image capturing unit and performing different image processing according to the capturing conditions corresponding to the identification information; The image recognition unit is operated as a determination unit that determines whether the image recognition unit has recognized the subject, On the condition that the determination unit does not determine that the subject has been recognized, the photographing condition setting unit changes the photographing conditions and causes the image photographing unit to photograph an image again. program.

Images