Electronic devices, methods for controlling electronic devices, programs

Abstract

To provide a mechanism that can more easily manage objects by using a result of detection of the objects from an image.SOLUTION: An electronic apparatus has: control means that controls to perform processing of detecting a plurality of objects from a photographed image; and display control means that, when at least one of the plurality of objects is detected in the detection processing, controls to display, of pieces of code information corresponding to the plurality of objects, code information corresponding to the object detected in the detection processing.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a technique for managing an object by detecting the object from an image.

Background Art

[0002] A system that detects an object (for example, a product) from an image and performs an automatic process is known.

[0003] In Patent Document 1, it is proposed that image information of products stored in a shopping cart by a purchaser is acquired and notified to a product settlement device, and the product settlement device recognizes the products by image recognition and accepts settlement of the total price of each product stored in the shopping cart.

Prior Art Document

Patent Document

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] When introducing a separate system from an existing system (for example, a cash register) like the product settlement device of Patent Document 1, system construction becomes complicated. For example, in a store, if there are products settled using a cash register and products automatically settled without using a cash register using image recognition mixed together, it is necessary to introduce a system for automatic settlement separately from the cash register. Then, costs such as system introduction costs and installation space increase. Also, the settlement methods become plural, and management of product sales and the like becomes complicated. This is the same not only for product management in stores but also for component management in factories and article management in logistics.

[0006] Therefore, an object of the present invention is to provide a mechanism that can manage an object more Buy using the detection result of the object from the image. Efficient processing efficiently. [Means for solving the problem]

[0007] In view of the above problems, the electronic device of the present invention is A control means that controls the detection process for multiple objects from the captured image, When at least one of the multiple objects is detected in the detection process, a display control means controls the display of the code information corresponding to the object detected in the detection process from among the code information corresponding to each of the multiple objects. It is characterized by having the following features. [Effects of the Invention]

[0008] According to the present invention, using the object detection result from the image, Buy From the object Efficient processing can. [Brief explanation of the drawing]

[0009] [Figure 1] This is a system configuration diagram of this embodiment. [Figure 2] Block diagram showing an example of the hardware configuration of electronic device 100. [Figure 3] This is a flowchart of the product recognition application process. [Figure 4] This is a flowchart of the product registration process. [Figure 5] This is an example of how product recognition results are displayed in a product recognition application. [Figure 6] This is an example of how the product registration process will be displayed. [Figure 7] This is an example of how it is displayed during product registration processing. [Figure 8] This is an example of a registration information table. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0011] Figure 1 shows an example of a system configuration in an embodiment of the present invention.

[0012] Electronic device 100 is an electronic device such as a tablet terminal with a camera. Cash register 200 is a cash register system for accounting in a store, and barcode reader 201 is a barcode reader connected to cash register 200. Products 41-45 are products to be recognized in this system. In this embodiment, the products to be recognized are assumed to be products that do not have paper or stickers with barcodes directly attached (for example, prepared bread). Tray 30 is a tray on which products are placed. Stand 20 is a stand (platform) on which tray 30 is placed, such as a store counter. Administrator 10 is a person who operates this system (for example, a store employee). In this system, a customer (not shown) or worker 10 (for example, a store employee) who has placed products on a tray places tray 30 with products 41-45 on it onto stand 20, and the camera mounted on electronic device 100 photographs tray 30 and products 41-45. Subsequently, the electronic device 100 automatically recognizes the product type (item) of each of the products 41-45 from the captured images and displays the barcode corresponding to the recognized product on the display unit of the electronic device 100. The worker 10 reads this barcode with the barcode reader 201 and makes the payment (checkout) at the register 200. If it is a self-checkout, this operation can be performed by the customer. In this way, by simply introducing the electronic device 100 to the existing system (register 200), payment and product management using the existing system (register 200) becomes possible.

[0013] Figure 2 shows an example of the configuration of an electronic device 100, which is an example of a device to which the present invention can be applied. The electronic device 100 can be configured using a tablet terminal, smartphone, personal computer (hereinafter referred to as PC), etc.

[0014] In FIG. 2, a CPU 101, a memory 102, a non-volatile memory 103, an image processing unit 104, a display 105, an operation unit 106, a recording medium I / F 107, an external I / F 109, and a communication I / F 110 are connected to an internal bus 150. Each unit connected to the internal bus 150 is configured to be able to exchange data with each other via the internal bus 150. The memory 102 is composed of, for example, a RAM (a volatile memory using semiconductor elements, etc.). The CPU 101 controls each part of the electronic device 100 by using the memory 102 as a work memory according to a program stored in the non-volatile memory 103, for example. The non-volatile memory 103 stores image data, audio data, other data, various programs for the operation of the CPU 101, and the like. The non-volatile memory 103 is composed of, for example, a hard disk (HD) or a ROM, etc.

[0015] Based on the control of the CPU 101, the image processing unit 104 performs various image processes on images stored in the non-volatile memory 103 or the recording medium 108, video signals acquired via the external I / F 109, images acquired via the communication I / F 110, captured images, and the like. The image processes performed by the image processing unit 104 include A / D conversion processing, D / A conversion processing, encoding processing of image data, compression processing, decoding processing, enlargement / reduction processing (resizing), noise reduction processing, color conversion processing, and the like. The image processing unit 104 may be composed of dedicated circuit blocks for performing specific image processes. Also, depending on the type of image process, it is also possible for the CPU 101 to perform image processing according to a program without using the image processing unit 104.

[0016] Based on the control of the CPU 101, the display 105 displays images, GUI (Graphical User Interface) screens that make up the GUI, and the like. The CPU 101 generates a display control signal according to a program, generates a video signal for display on the display 105, and controls each part of the electronic device 100 to output it to the display 105. The display 105 displays an image based on the output video signal. Note that the configuration of the electronic device 100 itself includes up to an interface for outputting a video signal for display on the display 105, and the display 105 may be composed of an external monitor (such as a TV).

[0017] The operation unit 106 is an input device (reception device) for receiving user operations, including character information input devices such as keyboards, pointing devices such as mice and touch panels, buttons, dials, touch sensors, touch pads, and the like. Note that the touch panel is an input device configured flatly by overlapping it with the display 105 and outputting coordinate information corresponding to the touched position.

[0018] The recording medium I / F 107 enables a recording medium 108 such as a memory card, CD, or DVD to be mounted, and reads data from the mounted recording medium 108 and writes data to the recording medium 108 based on the control of the CPU 101. The external I / F 109 is an interface for connecting to an external device by a wired cable or wirelessly and performing input / output of video signals and audio signals. The communication I / F 110 is an interface for communicating with an external device, the Internet 111, and the like and performing transmission and reception of various data such as files and commands.

[0019] The camera unit 112 is a camera unit composed of an image sensor (imaging sensor) such as a CCD or CMOS element that converts optical images into electrical signals. The camera unit 112 includes a lens group (photographic lens) including a zoom lens and a focus lens, a shutter with an aperture function, an image sensor, an A / D converter that converts analog signals output from the image sensor into digital signals, and a barrier that covers the imaging system to prevent dirt and damage. The image processing unit 104 performs resizing processing such as predetermined pixel interpolation and reduction, as well as color conversion processing, on the data captured and acquired by the camera unit 112. Based on the calculation results obtained by the image processing unit 104, the CPU 101 performs exposure control, distance measurement control, and AWB (auto white balance) processing. The image data for display, captured by the camera unit 112 and processed by the image processing unit 104, is displayed on the display 105. The digital signal captured by the camera unit 112, converted to digital by an A / D converter and stored in the memory 102, is converted to analog by a D / A converter and sequentially transmitted to the display 105 for display, enabling live view display (LV display). Live view can be displayed in the still image shooting standby state, the video shooting standby state, and during video recording, and the captured subject image is displayed in near real time. The CPU 101 controls the camera unit 112 and the image processing unit 104 to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, and AWB processing in response to shooting preparation instructions based on user operations made on the operation unit 106. In response to a shooting instruction, the CPU 101 controls the start of a series of shooting operations (actual shooting), from actual exposure and reading the signal from the image sensor, to image processing by the image processing unit 104 to generate an image file, and finally to recording it on the recording medium 108. Shooting instructions can be made by user operations on the operation unit 106. The camera unit 112 is capable of shooting both still images and videos. Furthermore, the electronic device 100 itself may only have an interface for inputting images captured by a camera, and the camera unit 112 may be an external camera.

[0020] Figure 3 is a flowchart showing an example of the accounting process in the product recognition application on the electronic device 100. Each process in this flowchart is realized by the CPU 101 loading the program stored in the non-volatile memory 103 into memory 102 and executing it. When the application software (product recognition application) according to this embodiment, installed on the electronic device 100, is started, the approval recognition application process shown in Figure 3 begins.

[0021] In S301, the CPU 101 starts acquiring images from the camera unit 112, sequentially stores images for LV display (LV images) for a predetermined number of frames in the memory 102, and performs LV display on the display 105.

[0022] In S302, the CPU 101 determines, based on the image stored in memory 102, whether the tray 30 has been placed on the stand 20 (whether it is within the field of view of the camera unit 112). For example, the presence of a subject with many linear components, which is presumed to be the tray, within the field of view can be used to detect that the tray 30 has been newly placed. If it has been placed, the process proceeds to S303; otherwise, it proceeds to S304.

[0023] In S303, the CPU 101 determines whether the image has stabilized (come still) based on the image held in memory 102. For example, it calculates the change in brightness of the image in the preceding and succeeding frames to determine whether the image has stabilized (whether the tray 30 has completely stopped). Alternatively, it may detect that the image has stabilized (that the amount of motion is below a predetermined amount) based on conditions such as the motion vector detected from the preceding and succeeding frames being below a predetermined value. If the image is stable, the process proceeds to S304; otherwise, it proceeds to S308.

[0024] In S304, the CPU 101 inputs the LV image (image captured by the camera unit 112) stored in memory 102 into a trained model stored in non-volatile memory 103 and held in memory 102, and infers the type (item) of the product placed on the tray 30. The inference result is stored in memory 102. This process is an image recognition process that analyzes the image captured by the camera unit 112 to determine whether the captured image contains a registered object (product) that has been registered in the product registration process described later. In other words, it is a process that detects each of multiple registered objects (products) from the image captured by the camera unit 112. This makes it possible to detect items such as "roll bread" and "melon bread" from an image captured by the camera unit 112. Note that the recognition process in S304 itself may be performed by an external device (not shown, e.g., a server device) that can communicate with the electronic device 100. In this case, the CPU 101 controls the camera unit 112 to send the captured image to an external device for recognition processing. Then, it receives the recognition result (inference result) from the external device and records it in the memory 102.

[0025] In S305, the CPU 101 refers to the inference result of the product type held in memory 102 and determines whether or not it has detected any of the registered products (objects). If any of the registered products (objects) is detected, the process proceeds to S306; otherwise, if none of the registered products (objects) are detected, the process proceeds to S307.

[0026] In S308, the CPU 101 reads the product identification information (e.g., product code, product number) recorded in association with the product item detected in S306 from the non-volatile memory 103. Then, it generates a barcode based on the read identification information and displays the image of the barcode on the display 105. The identification information is information corresponding to the product item (type of object) and is necessary information for generating a barcode. Specifically, it reads the first two digits of the in-store code stored in association with the product, the item code, and the check digit from the registration information table shown in Figure 8 (described later), and generates a barcode representing a 13-digit number (product code) by combining these. Note that the check digit may be calculated based on the preceding 12-digit number sequence instead of being read from the registration information table. An example of the display in S306 is shown in Figure 5.

[0027] Figure 5 shows an example of the display of product recognition results on the display 105. The captured image 500 is an image taken by the camera unit 112 and is a still image that was the target of detection in S304. Alternatively, an LV image taken by the camera unit 112 may be displayed instead of a still image. In the illustrated example, the captured image 500 includes products 501 to 504, which have been detected. In the parts of the image 500 where products have been detected, a product label indicating the product name and a product detection frame indicating the detected object are displayed. For example, a product detection frame 508 is displayed around product 501 to indicate that product 501 has been detected. Also, a product label 507 displaying the product name of product 501 is displayed at the position corresponding to the position where product 501 was detected (above the product detection frame 508 in this embodiment). In the illustrated product label 507, for convenience, it is labeled "Product Name 1," but in reality, the name of product 501 would be displayed as, for example, "Roll Bread." Similarly, product labels and product detection frames are displayed for products 502 to 504.

[0028] The barcode display area 510 is an area that displays the names and barcodes of products detected (recognized) from the captured image 500 in a single list. The barcode display area 510 displays the names and barcodes of all recognized products in a list. Barcodes 511 to 514 correspond to the detected products 501 to 504, respectively. By reading these barcodes 511 to 514 with the barcode reader 201, product identification information is entered into the register 200, and accounting information such as the product name and price can be obtained. In other words, accounting processing similar to that of a normal register becomes possible.

[0029] Product names 515-518 are the names of the detected products 501-504, respectively. For convenience, they are shown as "Product Name 1" and "Product Name 2," but in reality, product names (information identifying the item) such as "Roll Bread" and "Melon Bread" are displayed.

[0030] By displaying the barcodes of multiple detected products together in the barcode display area 510, the barcode reading operation by the barcode reader 201 is made easier. Barcodes 511 to 514 could each be displayed near the corresponding product on the captured image 500 (for example, at the location of each product label), but this would require the barcode reader 201 to move to the location of each barcode to read it. This would not only increase the amount of movement required for the barcode reader 201, but it would also require different movements depending on the arrangement of the bread, rather than uniform movements, making quick reading impossible. This could lead to an increase in the workload of the worker 10 and the time required for accounting processing. In contrast, in this embodiment, instead of displaying the barcodes near the detected products on the captured image 500, multiple barcodes corresponding to multiple detected products are displayed together in the barcode display area 510. By displaying them in this way, all barcodes can be read by moving the barcode reader 201 within the limited range of the barcode display area 510, thus shortening the movement distance of the barcode reader 201. Furthermore, multiple barcodes are arranged in a line (in this embodiment, they are displayed in a single vertical column). Therefore, by moving the barcode reader 201 with a uniform motion (in this embodiment, by moving it once vertically), all barcodes can be read. As a result, the operator 10 can quickly read the barcodes with a familiar and simple motion, reducing the workload and the time required for accounting processing.

[0031] Furthermore, if multiple identical products (products of the same item, products of the same type) are detected from the captured image 500, the barcode display area 510 will display multiple barcodes corresponding to the number of detected items. In the illustrated example, two products of product name 2 (for example, melon bread) are detected (products 502 and 504 are products of the same item). Therefore, the barcode display area 510 displays two barcodes for product name 2 (barcodes 512 and 513). Consequently, barcodes 512 and 513 are the same barcode (identical barcode). By displaying them in this way, all detected products and the correct number of products can be entered into the register 200 simply by reading the displayed barcodes in order with the barcode reader 201, without having to input the number of products. In other words, the numerical input operation of entering the number of products at the register 200 can be eliminated. Therefore, the operation can be simplified and work efficiency can be improved.

[0032] In S307, the CPU 101 determines whether the image is stable, similar to S303. If the image is stable, the process in S307 is repeated. If the image becomes unstable (if a change in the image is detected), the process proceeds to S308. Changes in the image may occur when tray 30 is removed, when additional products (bread) are placed on tray 30, or when products (bread) are removed from tray 30. In such cases, the image recognition process in S304 must be performed again. Conversely, image recognition processing is not performed until a change in the image is detected, and the display of the barcode that was displayed in S306 is continued. This prevents display fluctuations caused by image recognition processing, allowing the operator 10 to reliably have the barcode read by the barcode reader 201.

[0033] In S308, the CPU 101 determines, based on the image stored in memory 102, whether the tray 30 has been removed (moved) from the stand 20. For example, if a subject with many linear components, presumably the tray 30, moves out of the field of view, it can be detected that the tray 30 has been removed. If it has been removed, the process proceeds to S309; ​​otherwise, it proceeds to S303. If an item has been added to or removed from the tray 30, the process is determined as "No" in S308, and the image recognition process in S304 is performed after waiting for the image to stabilize in S303. This allows for the detection of items added to or removed from the tray 30 while it remains in place, and the screen described in Figure 5 can be updated. Alternatively, in response to the detection in S308 that the tray 30 has been removed, the screen may be updated to hide all barcodes displayed in the barcode display area 510, as well as the product labels and product detection frames. This allows the operator 10 to recognize that normal image recognition processing is being performed. Furthermore, it prevents errors such as accidentally accounting for items that have already been processed (double accounting).

[0034] In S309, the CPU 101 determines whether or not an instruction operation to switch to product registration mode has been performed. For example, if the icon for switching to registration mode (not shown) displayed on the display 105 is operated, it is determined that an instruction operation to switch to product registration mode has been performed. If an instruction operation to switch to product registration mode has been performed, the process proceeds to S310; otherwise, it proceeds to S311.

[0035] In S310, CPU101 performs the product registration process. Details of the product registration process will be described later using Figure 4.

[0036] In S311, the CPU 101 determines whether there has been a termination event, such as closing the application software according to this embodiment or turning off the power to the electronic device 100. If there is no termination event, the process proceeds to S302; if there is a termination event, the process shown in Figure 3 is terminated.

[0037] Figure 8 shows an example of a registration information table (also simply called registration information) recorded in the non-volatile memory 103, which contains the registration status (registration details) of a product. The product is registered in the registration information table by the product registration process in S310. The image recognition process in S304 and the barcode display in S306 are performed using the information registered in the registration information table. The registration information table stores the following associated information: the first two digits of the in-store code, the item code (a 10-digit number), the check digit (a single digit), whether the product is registered or not, the product name, and the recognition conditions. In the figure, "-" represents no data (NULL). In this embodiment, the registration information is described as being recorded in the non-volatile memory 103, but it may also be recorded in a server device that can communicate with the electronic device 100 or in a removable recording medium 108. In this way, one set of registration information can be used in common by multiple electronic devices, and can be used by multiple electronic devices corresponding to multiple cash registers.

[0038] An in-store code is a code that can be arbitrarily assigned by at least one of the following: a store, an organization, or a business. In this embodiment, when registering a product in the product registration process, the in-store code, which is a 13-digit JAN code starting with 20 to 29, is stored in association with the product being registered. By using an in-store code for registration, it is possible to prevent duplication with the distribution JAN code.

[0039] In the registration information table in Figure 8, the first two digits of the in-store code are a sequence of numbers from 20 to 29, indicating that it is an in-store code. The item code is a number used to identify a product. The check digit is a verification number added to detect errors in the digit sequence, and is a value determined based on a 12-digit sequence consisting of two digits from 20 to 29 and the 10-digit item code. The 13-digit sequence, which combines the first two digits of the in-store code, the item code (10-digit sequence), and the check digit (1-digit number) for each row, constitutes the product code (in-store code). In other words, each row in the registration information table corresponds to one product code (in-store code). In S306 mentioned above, if a registered product is detected, a barcode representing this 13-digit product code associated with that product is generated.

[0040] In the registration information table in Figure 8, the column indicating whether or not a product is registered (registered information) shows whether or not a product is registered in association with the product code in each row.

[0041] In the registration information table in Figure 8, the information in the "Product Name" column represents the product name of the product registered in association with the product code of each row.

[0042] In the registration information table in Figure 8, the information in the recognition conditions column is necessary to detect products registered in association with the product code of each row using image recognition. This information includes information about the trained model and product features. Because this section contains a large amount of information, it is not limited to directly recording the condition data in the registration information table; it may also include link information to access the recognition condition data or information that identifies (specifies) the recognition condition data. In the image recognition process S304 in Figure 3 mentioned above, detection processing based on these recognition conditions is performed for multiple products registered in the registration information table.

[0043] Figure 4 shows a flowchart of the product registration process. This process is a detailed version of S310 in Figure 3. Each process in this flowchart is implemented by the CPU 101 loading a program stored in non-volatile memory 103 into memory 102 and executing it. This process registers products that will be detected (more specifically, recognized in S304) in the accounting process. When a user adds a new product to the management (detection target), this process records recognition information (recognition conditions) for image recognition of the product, information for displaying the corresponding barcode, and the product name in association with each other in the electronic device 100. At the same time, the user registers information about the new product in the cash register 200 so that when the barcode displayed on the electronic device 100 is read, the product can be identified in the cash register 200.

[0044] In S401, the CPU 101 displays a product name input screen on the display 105. Figure 6(a) shows an example of the product name input screen displayed on the display 105. The text box 601 is an area where the user enters the name of the product to be newly registered.

[0045] In S402, the CPU 101 accepts product name input from the user. The user can select the text box 601 by touch and input any product name using a software keyboard or other means. In the illustrated example, "Crispy Curry Bread" is entered in response to the user's input. The entered product name is stored in memory 102.

[0046] In S403, the CPU 101 determines whether the next button 602 has been pressed. If the next button 602 has been pressed, the process proceeds to S404; otherwise, it proceeds to S402.

[0047] In S404, the CPU 101 compares the entered product name (the product name entered in S402) held in memory 102 with the registered product name registered in the registration information table to determine if it is the same as an already registered product name (a duplicate). If it is a duplicate, the process proceeds to S405; otherwise, it proceeds to S404.

[0048] In S405, CPU101 displays an error on display105 indicating that the product name is duplicated, and S401 displays the product name input screen again.

[0049] In S410, CPU 101 retrieves an available in-store code. Specifically, CPU 101 refers to the registration information table, searches for rows where the registered column is empty (i.e., unregistered), retrieves one in-store code from the unregistered row, and stores it in memory 102. The number assigned in S410 is obtained from in-store codes that begin with a specific two-digit number (29 in this embodiment) from the first two digits of the in-store code range 20 to 29. In this way, in-store codes starting with a specific two-digit number (29) can be used as numbers for product categories (bread in this embodiment) registered in this application software. Then, in-store codes starting with other two-digit numbers (e.g., 28) can be used as numbers for other categories (e.g., seafood), and so on, making product management easier.

[0050] In S411, the CPU 101 generates a barcode from the unregistered in-store code acquired in S410 and displays it on the display 105. Figure 6(b) shows an example of the display on the display 105 at this time. The registration target number 611 is the in-store code acquired in S410. Barcode 612 is the barcode image generated from the in-store code acquired in S410. The user reads barcode 612 with the barcode reader 201 of the register 200 and checks whether another product has already been registered for this in-store code at the register 200. If it is found that the in-store code represented by barcode 612 is an available number at the register 200, it can be determined that this in-store code can be adopted as a target for new product registration. For example, if the in-store code represented by barcode 612 is already registered as the number of another product at the register 200, it can be determined that this in-store code should not be adopted as a target for new product registration and that it is better to acquire a different number. In this way, before photographing and learning the products to be registered, as described later, candidate registration numbers are displayed as barcodes, allowing the system to first check if the registration number is available in register 200 before proceeding with further processing. This reduces unnecessary rework. The subsequent processing referred to here includes the processing in S434-S438 in response to the photography instruction received in S432, which is one of the registration operations (details will be described later). In this embodiment, the product name is entered before the barcode display in S411, but the input of the product name as related information to the registration number (processing in S401-S405) may also be performed after S411.

[0051] In S412, the CPU 101 determines whether or not button 614 has been touched (operated). For example, if the barcode reader 201 reads the in-store code represented by barcode 612 and finds that it is not an available number, the user touches button 614, which is the operation item, in order to assign a different in-store code. If button 614 is touched, the process proceeds to 413; otherwise, it proceeds to S414.

[0052] In S413, the in-store code obtained in S410de and displayed as registration target number 611 is recorded as already registered in the registered column of the registration information table, and the process proceeds to S410 to obtain another available in-store code number. In this way, numbers that were deemed not to be used are recorded as registered and are no longer subject to acquisition in S410, so they will not be automatically presented in the next time.

[0053] In S414, the CPU 101 determines whether the advanced settings button 613 has been touched. The advanced settings button 613 is an operation item that displays a screen where the user can enter a number of their choice as the registration target number, rather than the presented registration target number 611. If the advanced settings button 613 is touched, the process proceeds to S420; otherwise, it proceeds to S415.

[0054] In S415, the CPU 101 determines whether the forward button 615 has been touched. If it has been touched, proceed to S401; otherwise, proceed to S416.

[0055] In S416, the CPU 101 determines whether the Next button 616 has been touched. The Next button 616 is an operation item that accepts an instruction to confirm the displayed registration target number 611 as the registration target number to be associated with the product to be registered. If the Next button 616 is touched, the process proceeds to S417; otherwise, it proceeds to S412.

[0056] In S417, the CPU 101 confirms the in-store code held in memory 102 as the number to be registered. Specifically, it continues to hold it as the confirmed number.

[0057] In the S420, the CPU 101 displays the detailed settings screen on the display 105. Figure 6(c) shows an example of the detailed settings screen displayed on the display 105.

[0058] Figure 6(c) shows the screen for the user to set the registration target number to an arbitrary value. The registration target number 611 is a label displaying the registration target number stored in memory 102, and the barcode 612 is the registration target number converted into a barcode image and displayed. Input field 621 is a field for entering a two-digit number sequence indicating that it is an in-store code. The user can select input field 621 and enter any number between 20 and 29 using the software keyboard or a pull-down menu. Input field 622 is a field for entering an item code. The user can select input field 621 and enter any 10-digit number sequence using the software keyboard. When numbers are entered into input fields 621 and 622, the CPU 101 searches the registration information table to check if an in-store code containing that combination is already registered. If it is already registered (indicated in the registered column), it displays message 623 to notify the user that this number is already registered and cannot be used. The registration target number 624 is the in-store code consisting of the numbers entered in input fields 621 and 622. Barcode 625 displays the registration target number 624 as a barcode. When the operation item button 626 is touched, the in-store code containing the numbers entered in input fields 621 and 622 is discarded, the screen returns to the screen shown in Figure 6(b), and a screen is displayed with the in-store code obtained in S410 as the registration target number. When the operation item button 627 is touched, the screen returns to S401 and the screen shown in Figure 6(a) is displayed. When the operation item button 628 is touched, the process proceeds to S417, where the in-store code consisting of the combination of numbers and check digits entered in input fields 621 and 622 is confirmed as the registration target number and stored in memory 102. If the in-store code containing the combination of numbers entered in input fields 621 and 622 has already been registered and message 623 is displayed, the next button 628 will be displayed in an inactive state and will not accept touch operation. Touching the inactive next button 628 will not proceed to S417.

[0059] In S421, the CPU 101 accepts input of a two-digit number indicating an in-store code and an item code. That is, it accepts input operations for input fields 621 and 622. The entered numbers are stored in memory 102.

[0060] In S422, the CPU 101 determines whether the in-store code containing the number entered in S421 is already recorded as registered in the registration information table. If it is already registered, the process proceeds to S423; otherwise, if it is an available number, the process proceeds to S424.

[0061] In S423, CPU 101 displays message 623 on display 105 as an error indication.

[0062] In S424, the CPU 101 displays the registration target number 624 and barcode 625 on the display 105 based on the in-store code which includes the number entered in S421.

[0063] In S425, the CPU 101 determines whether the next button 628 has been touched. If the next button 628 has been touched, the process proceeds to S417; otherwise, it proceeds to S421.

[0064] In S430, CPU 101 initializes the variable n, which is stored in memory 102, to 1. Variable n is used to count the number of training images (number of times taken) for the product to be registered.

[0065] In S431, the CPU 101 displays a shooting screen on the display 105 for acquiring the nth image for training purposes of the product to be registered. An example of this display is shown in Figure 6(d).

[0066] Figure 6(d) shows an example of the display of the shooting screen for photographing the product to be registered. Image 631 is a live view image (LV image) captured by the camera unit 112. Product 632 is the product to be registered that is shown as the subject in image 631. Guide lines 633 are auxiliary lines that indicate the position where the product should be placed. The user (operator 10) frames the electronic device 100 so that the product to be registered is in the position indicated by the guide lines, or places the product to be registered at the position indicated by the guide lines 633 within the fixed field of view of the fixedly installed electronic device 100. Display item 635 shows the number of images required to register the product (number of shots) in the denominator and the number of images to be obtained in the next shot (which number of shots the next shoot will be) in the numerator. In the example shown, a total of 10 shots are required, and the current screen is for the first shot (the next shot will be the first shot). The shooting button 634, previous button 636, and next button 637 are operation items that the user can touch to operate them. The "Next" button 637 becomes touchable once the required number of photos (10 photos in the illustrated example) have been taken for product registration. Until then, it is displayed in an inactive state and does not respond to touch.

[0067] In S432, the CPU 101 determines whether the shutter button 634 has been touched. If the shutter button 634 is touched, the process proceeds to S434; otherwise, it proceeds to S433.

[0068] In S433, the CPU 101 determines whether the forward button 636 has been touched. If it has been touched, the process proceeds to S411; otherwise, it proceeds to S432.

[0069] In S434, the CPU 101 causes the camera unit 112 to perform the actual shooting; that is, it performs shooting control. In this embodiment, the actual shooting is assumed to be still image shooting, but it may also be video shooting.

[0070] In S435, the CPU 101 uses the image processing unit 104 to perform learning processing based on the images captured in the actual shooting in S434. Specifically, it extracts (crops) the portion of the captured image indicated by the guide line 633 and inputs it into the learning model, generating a trained model as a result of the learning process. Alternatively, it extracts features from the image from which the portion indicated by the guide line 633 has been extracted (cropped). The CPU 101 then records the generated trained model and the extracted features in the non-volatile memory 103 as recognition conditions for the row corresponding to the in-store code of the confirmed registration target number held in memory 102 in the registration information table mentioned above in Figure 8. Since more information is expected to lead to more accurate image recognition (improved recognition accuracy), learning is performed not only on a single image but on multiple images (10 in this embodiment). Through this recording control, the recognition conditions as related information acquired in response to the shooting instruction, which is a type of registration operation, are recorded in association with the in-store code of the registration target number.

[0071] In S436, the CPU 101 determines whether the variable n is 1 or not. If n=1, proceed to S437; otherwise, proceed to S438. It then determines whether the value of the number of photos taken at the time of product registration, which is held in memory 102, is 1 or not. If it is 1, proceed to S437; otherwise, proceed to S438.

[0072] In S437, the CPU 101 records the confirmed registration target number held in memory 102 as registered and associates it with the product name in non-volatile memory 103. Specifically, in the row corresponding to the in-store code of the confirmed registration target number held in memory 102 in the registration information table, it records information indicating "registered" in the "registered" column, and records the product name entered in S402, held in memory 102, in the "product name" column. In this way, the product name, in-store code, and recognition conditions of the new product are registered. Note that the product name and the information indicating that it is registered are not recorded until the processing in S435 is performed at least once. This is to prevent the in-store code from being marked as registered even though there are no recognition conditions and the product information is effectively not registered, if the process is interrupted after the input of the product name and confirmation of the registration target number but without performing the learning process even once.

[0073] In S438, the CPU 101 determines whether the variable n is equal to Nmax. Nmax is the denominator displayed in display item 635 and is a predetermined number representing the number of images required for product registration. In this embodiment, it is 10. If n is not equal to Nmax, the process proceeds to S439, incrementing the variable n by one, and then proceeding to S431 to process the next image capture. If n is equal to Nmax, the process proceeds to S450.

[0074] In the S450, the CPU 101 initializes the variable m, which is stored in memory 102, to 1. Variable m is used to count the number of times the system checks whether the registered product has been correctly detected.

[0075] In S451, the CPU 101 displays a confirmation screen on the display 105 to acquire the mth image, which is used to verify that the registered product is correctly recognized. An example of this display is shown in Figure 7(a).

[0076] Figure 7(a) shows an example of a confirmation screen display for acquiring the mth image to verify whether the registered product is correctly recognized. Area 708 is a display area that shows whether the recognition result is correct and the recognized product name. In the example shown, the recognition result and product name are not displayed because it is before the first confirmation. Image 701 is an LV image taken by the camera unit 112. Product 702 is the product to be confirmed that is shown as the subject in image 701 (the same individual product as the product photographed in S434, or a different individual of the same item). Guide line 703 is an auxiliary line that indicates the position where the product should be placed. The user (operator 10) frames the electronic device 100 so that the product to be confirmed is in the position indicated by the guide line, or places the product to be confirmed at the position indicated by the guide line 703 within the fixed field of view of the fixedly installed electronic device 100. Display item 705 shows a predetermined number in the denominator, which represents the number of images to be taken to verify the product (number of shots), and the numerator, which number image will be obtained in the next shot (which number of verification shots the next shot will be). In the example shown, a total of 5 verifications are planned, and the current screen is for taking the first shot (the next shot will be the first shot). The check button 704, previous button 706, and next button 707 are all user-touchable operation items.

[0077] In S452, the CPU 101 determines whether the forward button 706 has been touched. If it has been touched, the program proceeds to S430; otherwise, it proceeds to S453.

[0078] In S453, the CPU 101 determines whether or not the check button 704 has been touched. If it has been touched, the process proceeds to S454; otherwise, it returns to S452.

[0079] In S454, the CPU 101 inputs the LV image stored in memory 102 into the trained model stored in non-volatile memory 103 and held in memory 102, and infers the type (item) of the product placed on tray 30. This process is the same as the process in S304 described above and is performed using the recognition conditions registered in the training process in S435. The image used for inference (a still image of one frame of the LV image when the check button 704 is touched) is also stored in memory 102.

[0080] In S456, the CPU 101 refers to the inference result of the product type held in memory 102 and determines whether it has detected a product with the registration target number recorded in S437 from among the multiple registered products (objects). If a product with the registration target number is detected, the process proceeds to S461. Otherwise, that is, if no registered product is detected, or if a registered product is detected but does not match the product with the registration target number, the process proceeds to S457.

[0081] In S457, the CPU 101 displays an NG screen on the display 105. An example of this display is shown in Figure 7(b).

[0082] Figure 7(b) shows an example of the NG screen displayed on display 105. The NG screen is displayed when the product recognition result differs from the registered content. The same display elements as in Figure 7(a) are denoted by the same symbols as in Figure 7(a). Area 708 displays "Recognition Result: NG," indicating that the product recognition result differs from the registered content. It also displays "Product Name: Spicy Curry Bread," indicating that the product was detected as something different from the "Crispy Juicy Curry Bread" registered by the user. Image 711 is the image used for image recognition processing in S454, and displays a still image of one frame of the LV image when the check button 704 was touched. Product 712 is the product to be confirmed that is pictured in image 711 (the same individual product as the one photographed in S434, or a different individual of the same item). Guidance 710 is a message that informs the user that using image 711 as a correct image for the registration target number (in this example, an image of a crispy, juicy curry bun) for training (feature extraction) can be expected to improve recognition accuracy. Guidance 710 indicates that pressing (touching) the add registration button 714 will instruct the system to perform the training process (feature extraction process) using image 711.

[0083] In S458, the CPU 101 determines whether the add registration button 714 has been touched. If the add registration button 714 has been touched, the process proceeds to S459; otherwise, it proceeds to S460.

[0084] In S459, the CPU 101 extracts (crops) the portion indicated by the guide line 703 from a still image of one frame of the LV image (i.e., the image judged as NG) when the check button 704 held in memory 102 is touched, and inputs it into the learning model. In this way, a trained model is generated. Alternatively, features are extracted from the image from which the portion indicated by the guide line 703 has been extracted (cropped). Then, the CPU 101 records the generated trained model or extracted features in the non-volatile memory 103 as recognition conditions for the row corresponding to the in-store code of the confirmed registration target number held in memory 102 in the registration information table mentioned above in Figure 8. This process is the same as in S435, except for the input image.

[0085] In S460, CPU 101 determines whether the forward button 706 has been touched. If it has been touched, it proceeds to S451; otherwise, it proceeds to S458.

[0086] In S461, the CPU 101 displays an OK screen on the display 105 for a predetermined time (for example, 3 seconds) to indicate that the product was recognized correctly. On the OK screen, "Recognition result: OK" is displayed in area 708, and the recognized product name (in this example, "Crispy Juicy Curry Bread") is displayed.

[0087] In S462, the CPU 101 determines whether the variable m is equal to Mmax. Mmax is the denominator displayed in display item 705 and is a predetermined number representing the number of images (number of shots) to be taken to verify the product. In this embodiment, it is 5. If m is not equal to Mmax, the process proceeds to S463, incrementing the variable m by 1, and then proceeding to S451 to process the next shot. If m is equal to Mmax, the process proceeds to S464.

[0088] In S464, the registration completion screen is displayed on CPU 101 and display 464. An example of this display is shown in Figure 7(c).

[0089] Figure 7(c) shows an example of the registration completion screen. Product name 721 is the product name registered in association with the registration target number, and is the product name entered in S402. Product image 722 is an image of the registered product, taken in S434, and is one of the images used for the learning process in S435. This product image 722 was stored in memory 102, but it may also be recorded in non-volatile memory 103 in association with the in-store code of the registration target number in the registration information table. Barcode 723 is a barcode that contains information about the registration target number, generated based on the registration target number confirmed in S417. The previous button 724 and the complete button 725 are operation items, respectively. The user (worker 10) reads the barcode 723 displayed on the display 105 with the barcode reader 201 and registers the product with the same in-store code in the register 200. For example, after registering a new in-store code at register 200, the barcode 723 is read, and the in-store code to be registered is entered into register 200. Since the barcode 723 also displays the numerical sequence of the in-store code, instead of reading the barcode with the barcode reader 201, the operator can look at the displayed numerical sequence and use the number keys on the register to enter the in-store code to be registered. Then, the operator 10 operates register 200 to register various information such as product name, product category, and price, associating it with the entered in-store code. In this way, when the same barcode as barcode 723 is read at checkout, payment can be made at the register. Thus, according to the product registration process in Figure 4, when registering code information (in-store code) associated with an object using a device (electronic device 100) separate from the barcode reading device (register 200), the barcode to be registered is displayed. Therefore, it can be easily linked and registered with the barcode reading device (register 200).

[0090] In S465, the CPU 101 determines whether the complete button 725 has been touched. If the complete button 725 has not been touched, the system waits in S465. If the complete button 725 is touched, the product registration process shown in Figure 4 is terminated.

[0091] According to the product registration process described above, recognition conditions for detecting an object (product) can be learned in association with the code information (in-store code) to be registered, and these recognition conditions can be recorded in association with the code information.

[0092] Furthermore, in addition to registering the object and its code information in association with each other in the electronic device 100, it is also necessary to register information about the object (such as price and name) and its code information in association with each other in the external device (cash register 200) that reads the code information. In contrast, in this embodiment, the barcode or number sequence of the code information to be registered is displayed on the display 105 of the electronic device 100 at an appropriate timing during the product registration process, making it easy to perform registration work on the external device as well. Even in a configuration where the electronic device 100 and the cash register 200 cannot communicate and cannot share the registration information table, this embodiment allows each device to easily perform coordinated registration work.

[0093] According to the embodiment described above, objects (e.g., products) can be managed more easily by using the detection results of objects (e.g., products) from captured images. In the above embodiment, products that are checked out at a store were used as an example of objects, but the embodiment is not limited to this. Even objects other than products can be detected from captured images, and the embodiment is applicable as long as they can be managed by displaying code images such as barcodes (1D codes) or QR codes (trademarks) (2D codes) and reading the displayed code images with existing systems. For example, the objects can be delivery goods and mail in logistics, or raw materials, supplies, and parts in factories.

[0094] Furthermore, the various controls described above, which are performed by the CPU 101, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0095] Furthermore, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0096] Furthermore, although the above-described embodiments explained the application of the present invention to an electronic device 100 as an example, the invention is not limited to this example and can be applied to any device capable of acquiring captured images. In other words, the present invention can be applied to personal computers, PDAs, mobile phone terminals, portable image viewers, game consoles, e-book readers, and the like.

[0097] (Other embodiments) The present invention can also be realized by performing the following process: supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program code. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0098] 100...Electronic devices 200...Cash registers

Claims

1. A control means that controls a detection process to detect two or more objects from the items contained in the captured image, A display control means that controls the display of two or more code pieces for purchasing two or more objects, corresponding to two or more objects detected in the detection process that do not have code pieces for purchasing the items attached, in a predetermined area. An electronic device characterized by having the following features.

2. The electronic device according to claim 1, characterized in that the display control means controls the two or more code pieces to be displayed together in an area for displaying the code piece of the object to be purchased.

3. The electronic device according to claim 2, characterized in that the display control means displays an image including the object detected in the detection process as the subject, and controls the code information to be displayed side by side in the area where the code information of the object to be purchased is displayed.

4. The electronic device according to any one of claims 1 to 3, characterized in that the display control means changes the number of code information displayed for purchasing the detected objects of the same type according to the number of objects of the same type detected by the control means.

5. The electronic device according to claim 4, characterized in that the display control means controls the display of the code information of the same type of object to be displayed for the number of times the same type of object has been detected.

6. The electronic device according to claim 5, characterized in that the two or more code information for two or more objects of the same type is the same code information.

7. A label display means that displays a label that can identify the product name of each object at the position on the image where each of the two or more objects is detected by the control means, Equipped with, The electronic device according to any one of claims 1 to 6, characterized in that the display control means controls the display of two or more code pieces of information for purchasing the two or more objects detected in the detection process in association with the product name displayed on the label.

8. A means for identifying that an object in which the object is located has been removed from a predetermined area to be photographed, Equipped with, The electronic device according to any one of claims 1 to 7, characterized in that the display control means hides the code information when the object is removed.

9. The electronic device according to any one of claims 1 to 8, characterized in that the code information is a code image of a one-dimensional code or a two-dimensional code.

10. The electronic device according to any one of claims 1 to 9, characterized in that the code information is code information that can be arbitrarily assigned by at least one of a store, organization, or business operator.

11. The electronic device according to any one of claims 1 to 10, characterized in that the code information displayed by the display control means can be read by a reading unit of an external device and used for processing by the external device.

12. The electronic device according to claim 11, characterized in that the external device is a cash register and the process is accounting processing.

13. The electronic device according to claims 1 to 12, further comprising processing means for performing a registration process to register information for recognizing an object in association with code information.

14. The electronic device according to claim 13, characterized in that the information for recognizing the object is a feature quantity obtained from an image relating to the object.

15. The system further includes a receiving means for receiving registration operations for related information of an object corresponding to the code information to be registered, The electronic device according to claim 13 or 14, characterized in that the processing means performs a registration process that registers the code information to be registered in association with the related information received by the receiving means.

16. The electronic device according to claim 15, characterized in that the aforementioned related information is the product name of the object.

17. A control step in which the control means of the electronic device is controlled to perform a detection process to detect two or more objects from the articles contained in the captured image, A display control step in which the display control means of an electronic device controls the display to display two or more code pieces for purchasing two or more objects that were detected in the detection process and for which no code piece for purchasing the item has been attached, all together in a predetermined area. A method for controlling electronic equipment, characterized by including the following:

18. A program for causing a computer to function as one of the electronic devices described in any one of claims 1 to 16.

Images