Terminal device and image processing method

Abstract

To provide a technique capable of quickly recognizing an object.SOLUTION: A POS terminal device comprises: imaging means for imaging an object to generate an image; display means for displaying the image; determination means for determining whether at least a part of the object is present in the image; comparison means for comparing the image with a commodity image database when at least a part of the object is present in the image; and control means for executing control to display guidance of changing the image displayed by the display means when it is not possible to identify which commodity the object is, as a result of the comparison.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an apparatus that uses an object identification technology, or to a POS (Point Of Sales) terminal device that uses an object identification technology. [Background technology]

[0002] Patent Document 1 discloses a barcode scanner technology. First, an image determination unit of the barcode scanner determines whether or not there is a candidate image for a barcode in the frame of a captured image. Next, if a decode processing unit detects that the barcode candidate image is partially missing, a guide image is displayed on a display to guide a captured image display unit so that the barcode candidate image can be captured as a barcode. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-231436 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, as in Patent Document 1, if partial loss of a barcode candidate image cannot be detected, a guide image for guidance cannot be displayed, which poses a problem in that the object cannot be quickly identified.

[0005] In order to solve the above-mentioned problems, an object of the present invention is to provide a technique capable of quickly recognizing an object. [Means for solving the problem]

[0006] A POS terminal device according to one embodiment of the present invention comprises an imaging means for imaging an object to generate an image, a display means for displaying the image, a determination means for determining whether or not at least a part of the object is present in the image, a comparison means for comparing the image with a product image database if at least a part of the object is present in the image, and a control means for controlling the display means to display a message instructing the user to change the image displayed thereon if it is not possible to identify which product the object is as a result of the comparison.

[0007] An information processing device according to one aspect of the present invention comprises a determination means for determining whether or not at least part of an object is present in a captured image, a comparison means for comparing the image with a product image database if at least part of the object is present in the image, and a control means for controlling the display means to display a message prompting the image displayed on the display means to change if it is not possible to identify which product the object is as a result of the comparison.

[0008] In one form of the image processing method of the present invention, a computer determines whether or not at least part of an object is present in a captured image, and if at least part of the object is present in the image, compares the image with a product image database, and if the result of the comparison indicates that it is not possible to identify which product the object is, controls the display means to display a message that prompts the image displayed on the display means to change.

[0009] A program that is one form of the present invention causes a computer to function as a determination means for determining whether or not at least part of an object is present in a captured image, a comparison means for comparing the image with a product image database if at least part of the object is present in the image, and a control means for controlling the display means to display a message that prompts the image being displayed on the display means to change if it is not possible to identify which product the object is as a result of the comparison. Effect of the Invention

[0010] According to the present invention, an object can be quickly recognized. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing an overview of a POS terminal device according to a first embodiment of the present invention. [Diagram 2] 1 is a side view showing an external appearance of a POS terminal device according to a first embodiment. [Diagram 3] 1 is a block diagram showing a configuration of a POS terminal device according to a first embodiment. [Figure 4] 5 is a flowchart showing the operation of the POS terminal device and the information processing device according to the first embodiment. [Figure 5A] 4 is a diagram showing the positional relationship between an object and an imageable area according to the first embodiment. FIG. [Figure 5B] FIG. 13 is a diagram showing guidance displayed on a display unit. [Figure 5C] FIG. 13 is a diagram showing guidance displayed on a display unit. [Figure 6A] 4 is a diagram showing the positional relationship between an object and an imageable area according to the first embodiment. FIG. [Figure 6B] FIG. 13 is a diagram showing guidance displayed on a display unit. [Figure 7] FIG. 11 is a block diagram showing a configuration of a POS terminal device according to a second embodiment. [Figure 8] 10 is a flowchart showing the operation of a POS terminal device and an information processing device according to a second embodiment. [Figure 9] FIG. 2 is a diagram illustrating a hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (First embodiment) An overview of a first embodiment of the present invention will be described. Fig. 1 is a diagram showing an overview of a POS terminal device 1 which is an example of the first embodiment of the present invention. As shown in Fig. 1, the POS terminal device 1 includes an imaging unit 10, a display unit 20, and an information processing device 50. The information processing device 50 includes a determination unit 30 and a control unit 40.

[0013] The imaging unit 10 captures an object and generates an image. The display unit 20 displays the image generated by the imaging unit 10. The determination unit 30 of the information processing device 50 determines whether or not at least a part of the object is present in the image generated by the imaging unit 10. When a part of the object is present in the image generated by the imaging unit 10, the control unit 40 of the information processing device 50 displays a guidance display on the display unit 20 to guide the object in the image in a predetermined direction. For example, the guidance display of the control unit 40 may be a guidance display that guides the image of the object to be displayed entirely on the display screen of the display unit 20, or may be a guidance display that guides the object in a direction in which the image of the object is in focus. Note that the control unit 40 may control the display unit 20 to display an image of the object captured by the imaging unit 10 while displaying the guidance display on the display unit 20.

[0014] When at least a part of an object is present in an image, the POS terminal device 1 according to the embodiment of the present invention displays a guidance display for guiding the object in the image in a predetermined direction on the display unit 20. This allows the object to be quickly recognized.

[0015] The above is an example of a configuration in which the POS terminal device 1 includes the imaging unit 10, the display unit 20, and the information processing device 50, but is not limited to this. For example, the POS terminal device 1 may include the imaging unit 10 and the display unit 20, the information processing device 50 installed outside the POS terminal device 1 includes the determination unit 30 and the control unit 40, and the POS terminal device 1 and the information processing device 50 may be connected by wire or wirelessly.

[0016] A specific example of the first embodiment will be described in more detail below with reference to the drawings. FIG. 2 is a side view showing the appearance of a POS terminal device 100 which is a specific example of the first embodiment. FIG. 3 is a block diagram showing the configuration of the POS terminal device 100 according to the first embodiment. The POS terminal device 100 includes a store clerk display unit 110, a customer display unit 112, an information processing device 120, and a commodity reading device 140. The store clerk display unit 110 in FIG. 2 displays information for the store clerk, and the customer display unit 112 displays information for the customer.

[0017] The clerk display unit 110 and the customer display unit 112 may be a touch panel display or an LCD (Liquid Crystal Display). The clerk display unit 110 and the customer display unit 112 may be equipped with an input device such as a keyboard. The clerk display unit 110 displays information required by the clerk and accepts operations by the clerk under the control of the information processing device 120.

[0018] The customer display unit 112 displays information required for the customer under the control of the information processing device 120. It may also accept operations by the customer as necessary.

[0019] The information processing device 120 controls the operations of the store clerk display unit 110, the customer display unit 112, and the commodity reading device 140. The information processing device 120 also performs necessary processing in response to operations accepted by the store clerk display unit 110. The information processing device 120 also performs necessary processing such as image processing in response to image information read by the commodity reading device 140.

[0020] The commodity reading device 140 has a housing 142 and an imaging unit 130. A light-transmitting commodity reading surface 144 is provided in a part of the housing 142. The commodity reading surface 144 is provided on the face of the housing 142 facing the store clerk for the store clerk's work, and an object is directed toward the commodity reading surface 144 when an image of the object is to be captured. The imaging unit 130 is provided inside the housing 142. When the store clerk directs an object received from a customer toward the commodity reading surface 144, the imaging unit 130 reads the image of the object. As a result, the POS terminal device 100 performs a recognition process of the object.

[0021] The range in which the imaging unit 130 can capture an object (hereinafter referred to as the imaging range) depends on the optical characteristics such as the angle of view and focus of the lens of the imaging unit 130. The imaging range A is composed of the angle of view range that is reflected in the imaging unit 130 through the lens due to the angle of view, etc., and the focus range in which a clear image is obtained when the focus is adjusted. For this reason, the imaging range in the POS terminal device 100 in FIG. 2 is shown as the imaging range A surrounded by a dashed line. In FIG. 2, the vertical direction of the angle of view range is shown by a dashed line spreading from the imaging unit 130 through the commodity reading surface 144. In FIG. 2, the horizontal direction of the angle of view range is not shown. Note that the vertical direction and the horizontal direction start from the position of the imaging unit 130. The focus range is the range in the depth direction from the imaging unit 130 to the commodity reading surface 144. In this case, the horizontal direction of the angle of view range not shown in FIG. 2 is the direction perpendicular to the vertical direction and the depth direction.

[0022] The imaging unit 130 will be described in detail below. The imaging unit 130 can take at least the following three forms. In the first case, the imaging unit 130 includes a two-dimensional image imaging unit that captures a two-dimensional image, a distance sensor that measures the distance to the commodity, and a distance image generation unit. The two-dimensional image imaging unit captures an image of an object facing the commodity reading surface 144, and generates a two-dimensional color image or a two-dimensional monochrome image including an image of the object.

[0023] The distance sensor measures the distance from the distance sensor to the position of an object facing the commodity reading surface 144, for example, using a TOF (Time Of Flight) method. That is, the distance sensor irradiates a light beam such as infrared light, and measures the distance from the time it takes for the irradiated light beam to travel to and from the object. The distance image generating unit measures the distance at each position of the object, and generates a distance image (three-dimensional image) by superimposing two-dimensional images. In the first case, the imaging unit 130 can capture an object whose distance to the commodity is within a predetermined range (for example, 15 cm to 30 cm).

[0024] In the second case, the imaging unit 130 includes one two-dimensional image imaging unit that captures a two-dimensional image. In the second case, the imaging unit 130 can obtain an image of the object by calculating the difference between a background image captured in advance by the imaging unit 130 and an image including the object.

[0025] In the third case, the imaging unit 130 includes a plurality of two-dimensional image imaging units that capture two-dimensional images, and a distance image generating unit that can generate a distance image (three-dimensional image) based on the difference in field of view between the plurality of imaging units.

[0026] The determination unit 122 determines whether or not at least a part of an object is present in the image. This process can be realized, for example, by executing a program under the control of the control unit 124. Specifically, this process is realized by executing a program stored in a storage unit (not shown).

[0027] Fig. 4 is a flowchart showing the operations of the POS terminal 100 and the information processing device 120. In Fig. 4, steps S100 to S300 are a flowchart showing the operations of the POS terminal 100, and steps S200 to S300, which are a part of the flowchart, are a flowchart showing the operations of the information processing device 120.

[0028] The imaging unit 130 of the POS terminal device 100 captures an image of an object to generate an image (S100). Next, the determination unit 122 of the information processing device 120 determines whether or not at least a part of the object is present in the image (S200). Specifically, if the image is a distance image including distance information, it is determined whether or not at least a part of the object is present within a predetermined range in the up-down direction, the left-right direction, and the depth direction, which is the imageable area A. Also, if the image is a normal two-dimensional image not including distance information, it is determined whether or not at least a part of the object is present within a predetermined range in the up-down direction and the left-right direction, which is the imageable area A.

[0029] Next, if at least a part of the object is not present in the image (NO in S200), the process returns to step S100, and the imaging section 130 captures an image of the object again.

[0030] Next, if at least a part of the object is present in the image (YES in S200), the control unit 124 displays a guidance display on the store clerk display unit 110 to guide the object in the image in a predetermined direction (S300). As an example, the control unit 124 performs control to display a guidance display that guides the object so that the entire object enters the imageable area A. By displaying a guidance display that guides the object so that the entire object enters, the store clerk moves the object according to the guidance display. As a result, the entire object is captured by the imaging unit 130, so that image matching with the database of product images can be performed quickly.

[0031] 5A is a diagram showing an example of the positional relationship between the object 131 and the imageable area A of the imaging unit 130 of the POS terminal device 100 of FIG. 2. The imageable area A in FIG. 5A is an area set in the direction from the imaging unit 130 to the object side. FIG. 5A shows a state in which a part of the object 131 is included in the imageable area A of the imaging unit 130. In FIG. 5A, the object is shown as a circle, and specific examples are fresh food such as tomatoes or apples, or packaged sweets. Note that the object may have a shape other than a circle. In FIG. 5A, a part of the object 131 is present on the upper right side of the imageable area A when viewed from the imaging unit 130 side.

[0032] Fig. 5B is a diagram showing an example of guidance display 111 displayed on store clerk display unit 110 of Fig. 3 or 4. When at least a part of object 131 is present in an image captured by imaging unit 130 (Fig. 4), control unit 124 (Fig. 4) processes the image to recognize in which part of the image the part of object 131 is present.

[0033] The image captured by the imaging unit 130 is composed of pixels separated in the vertical and horizontal directions. The control unit 124 then recognizes in which pixel the object is located. Furthermore, the control unit 124 determines the direction of guidance so that the entire object 131 is included in the imageable area A. The control unit 124 determines the direction of guidance by calculating the direction from the pixel position where the object is captured to the pixel position that is approximately at the center of the entire image. At this time, the position of the image of the object captured by the imaging unit 130 and the position of the object as seen by the store clerk from the working position are symmetrical. Taking this into consideration, the control unit 124 displays a guidance display 111 corresponding to the guiding direction on the store clerk display unit 110.

[0034] Also, as shown in FIG. 5C, the control unit 124 may display the captured image 114 of the object together with the guidance display 111 on the clerk display unit 110. Even when the captured image 114 of the object is displayed on the clerk display unit 110, the position of the image of the object captured by the imaging unit 130 and the position of the object as seen by the clerk from the working position are symmetrical. For this reason, the control unit 124 displays an image that is inverted left and right with respect to the center of the image on the clerk display unit 110. Since the clerk can confirm the position of the object and the guidance display on the clerk display unit 110, the clerk can move the object so that the entire object can be captured. As a result, the entire object is captured by the imaging unit 130, and image matching with the database of product images can be performed quickly.

[0035] 5A to 5C, the control unit 124 may change the color displayed on the store clerk display unit 110 in accordance with the movement of the object 131. For example, green may be displayed when the entire object is within the imageable area A, yellow may be displayed when more than half of the object is within the imageable area A, and red may be displayed when less than half of the object is within the imageable area A.

[0036] 6A is a diagram showing the positional relationship between an object and an imageable area A according to the first embodiment. An imaging unit 130 installed in a commodity reading device 140, which is a part of a POS terminal device, captures an image of an object 131 via a commodity reading surface 144. As shown in FIG. 6A, a part of the object 131 is within the imageable area A, and is located closer to the imaging unit 130 than the imageable area A.

[0037] In this case, the control unit 124 (FIG. 4) determines that the object is located closer to the imaging unit 130 than the imageable area A, based on the fact that the imaging unit 130 can acquire the distance to the object in the first and second cases described above. The control unit 124 recognizes which part of the image in the depth direction is located based on the acquired distance information. Next, the control unit 124 displays a guidance display on the store clerk display unit 110 to guide the object in the image in a predetermined direction. As a desirable example, the guidance display is displayed on the store clerk display unit 110 so that the entire object is within the imageable area A.

[0038] Fig. 6B is a diagram showing a guidance display displayed on the store clerk display unit 110. As shown in Fig. 6A, when it is necessary to move the object in the depth direction relative to the imaging unit 130, the control unit 124 displays the text "Please move the product away from the product reading surface" as the guidance display 113. As another guidance display, the control unit 124 displays red when the distance between the object 131 and the product reading surface 144 is too close, and displays green when the distance between the object and the product reading surface 144 is too far.

[0039] The above describes a configuration in which the POS terminal device 100 includes the imaging unit 130, the clerk display unit 110, the determination unit 122, and the control unit 124, but the present embodiment is not limited to the above example. For example, the POS terminal device 100 may include the imaging unit 130 and the clerk display unit 110, the information processing device 120 installed outside the POS terminal device 100 may include the determination unit 122 and the control unit 124, and the POS terminal device 100 and the information processing device 120 may be connected by wire or wirelessly.

[0040] Second Embodiment Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that the information processing device 120 includes a storage unit 126 and a collation unit 128. Note that components substantially similar to those in the first embodiment are given the same reference numerals and will not be described.

[0041] Fig. 7 is a block diagram showing the configuration of a POS terminal device 100 according to a second embodiment. In addition to the components shown in Fig. 3, the POS terminal device 100 of the second embodiment further includes a storage unit 126 and a matching unit 128. The storage unit 126 stores a database of product images. The database of product images contains information on the shape and color of each product image, which are characteristics of the product image. The matching unit 128 matches the image captured by the imaging unit 130 with the characteristics of the product image in the database of product images to identify which product the captured image is.

[0042] The determination unit 122 and the matching unit 128 can be realized, for example, by executing a program under the control of the control unit 124. Specifically, they are realized by executing a program stored in the storage unit 126. In the above description, the database of product images is stored in the storage unit 126, but this is not limited to this. The database of product images may be saved in a storage device (not shown) installed outside the POS terminal device 100. In this case, the matching unit 128 acquires the characteristics of the product images from the storage device and compares them with the image captured by the imaging unit 130.

[0043] Fig. 8 is a flowchart showing the operation of the POS terminal 100 or the information processing device 120. In Fig. 8, steps S100 to S300 in Fig. 4 are a flowchart showing the operation of the POS terminal 100, and steps S200 to S230, which are a part of the flowchart, are a flowchart showing the operation of the information processing device 120.

[0044] The imaging unit 130 of the POS terminal device 100 captures an image of an object to generate an image (S100). Next, the determination unit 122 of the information processing device 120 determines whether or not at least a part of the object is present in the image (S200). Specifically, if the image is a distance image including distance information, it is determined whether or not at least a part of the object is present within a predetermined range in the up-down direction, the left-right direction, and the depth direction, which is the imageable area A. Also, if the image is a normal two-dimensional image not including distance information, it is determined whether or not at least a part of the object is present within a predetermined range in the up-down direction and the left-right direction, which is the imageable area A.

[0045] Next, if at least a part of the object is not present in the image (NO in S200), the process returns to S100, and the imaging section 130 captures an image of the object again.

[0046] The determination unit 122 may not perform the process of NO in S200. This is because there is a possibility that the product can be identified by image matching in the next step.

[0047] Next, if at least a part of an object is present in the image (YES in S200), the matching unit 128 matches the captured image with the product image database stored in the storage unit 126. If the matching unit 128 identifies which product the captured image is, the control unit 124 proceeds to the payment process with the identified product (S300). If the matching unit 128 is unable to identify which product the captured image is as a result of the matching, the control unit 124 displays a prompting message on the store clerk display unit 110 (S230). The prompting message is the same as the display shown in FIG. 5B or C and FIG. 6B.

[0048] The above describes a case where the POS terminal 1 includes the imaging unit 130, the clerk display unit 110, the determination unit 122, the collation unit 128, and the control unit 124, but the present invention is not limited to this. For example, the POS terminal 100 may include the imaging unit 130 and the clerk display unit 110, and the information processing device 120 installed outside the POS terminal 100 may include the determination unit 122, the control unit 124, and the collation unit 128, and the POS terminal 100 and the information processing device 120 may be connected by wire or wirelessly.

[0049] At least a part of the information processing device 50, 120 described above may be realized by executing a program (software program, computer program) in the CPU 910 of the computer 900 shown in FIG. 9. Specifically, the information processing device 50, 120 may be realized by executing a program that constitutes the determination unit 30 and the control unit 40 in FIG. 1, the determination unit 122 and the control unit 124 in FIG. 3 and FIG. 7, and the collation unit 128 in FIG. 7. These constituent elements may be realized by the CPU (Central Processing Unit) 910 reading a program from a ROM (Read Only Memory) 930 or a hard disk drive 940, and executing the read program using the CPU 910 and the RAM (Random Access Memory) 920, for example, as shown in the flowcharts in FIG. 4 and FIG. 8. In such a case, the present invention described using the above-mentioned embodiment as an example can be considered to be configured by a code representing the computer program or a computer-readable storage medium in which the code representing the computer program is stored. The computer-readable storage medium is, for example, a hard disk drive 940, a removable magnetic disk medium, an optical disk medium, or a memory card, not shown. The determination unit 30 and control unit 40 in FIG. 1, the determination unit 122 and control unit 124 in FIGS. 3 and 7, and the collation unit 128 in FIG. 7 may be dedicated hardware implemented as an integrated circuit.

[0050] The present invention has been described above by taking the above-mentioned embodiment as an exemplary example. However, the present invention is not limited to the above-mentioned embodiment. That is, the present invention can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present invention.

[0051] This application claims priority based on Japanese Patent Application No. 2014-065932, filed March 27, 2014, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0052] 1 POS terminal 10. Imaging unit 20 Display section 30 Judgment section 40 Control section 50 Information processing device 100 POS terminals 110 Display section for clerks 111 Guidance display 112 Customer display 113 Guidance display 114 Images of Objects 120 Information processing device 122 Judgment section 124 Control Unit 128 Matching section 130 Imaging unit 131 Object 140 Product reader 142 Case 144 Product reading surface 900 Computers 910 CPU 920 RAM 930 ROM 940 Hard Disk Drive 950 Communication Interface

Claims

1. An imaging means for imaging a commodity and generating an image; A comparison means for comparing the image with a product image database; a control means for causing a display means to output a display for re-collation according to a position of the product when the product cannot be identified as a result of the collation; the display is a display that guides the user to change a vertical or horizontal position of the product in the image or a distance of the product from the imaging means and perform re-matching.

2. An imaging means for imaging a commodity and generating an image; A comparison means for comparing the image with a product image database; a control means for controlling a display means to output a display for re-collation including information regarding the position of the product that could not be identified when the product cannot be identified as a result of the collation; the display is a display that guides the user to change a vertical or horizontal position of the product in the image or a distance of the product from the imaging means and perform re-matching.

3. The terminal device according to claim 1 , wherein the display guides the user to place the entire product within an image capture area of ​​the imaging means.

4. The terminal device according to claim 1 , wherein the display guides the user in a depth direction included in a focal range of the imaging means in an imageable area.

5. The terminal device according to claim 1 , wherein the display is a display that guides the user in an up-down direction, a left-right direction, or a combination of these directions in an imageable area.

6. The terminal device according to claim 1 , wherein the indication is a change in color according to the top, bottom, left, right, or right position of the product in the image, or according to the distance from the imaging means.

7. The computer A product image is captured and the generated image is compared with a product image database; If the product cannot be identified as a result of the comparison, a display for re-comparison is output on a display means in accordance with the position of the product; The image processing method, wherein the display is a display that guides the user to change the vertical, horizontal position of the product in the image, or the distance of the product from an imaging means and perform the match again.

8. The computer A product image is captured and the generated image is compared with a product image database; If the product cannot be identified as a result of the comparison, a display for re-comparison including information regarding the position of the product that could not be identified is output to a display means; The image processing method, wherein the display is a display that guides the user to change the vertical, horizontal position of the product in the image, or the distance of the product from an imaging means and perform the match again.

Images