Item recognition device and information processing program

Abstract

This reduces the effort required for the operator to select one of several candidates. [Solution] The object recognition device of the embodiment comprises an imaging device, measuring means, recognition means, selection means, display means, setting means, and determination means. The measuring means measures the direction and amount of movement of the imaging device. The recognition means recognizes an object captured in an image based on the image obtained by the imaging device. The selection means selects a candidate for the object recognized by the recognition means. The display means causes the display device to display a screen that includes, as a part, an area associated with the candidate selected by the selection means. The setting means sets a cursor position within the screen displayed by the display means based on the direction and amount of movement measured by the measuring means. The determination means determines that the candidate associated with the area is an object captured in the image when the cursor position set by the setting means is within the area.

Description

Technical Field

[0001] Embodiments of the present invention relate to an article recognition device and an information processing program.

Background Art

[0002] Techniques for image recognition of an article (hereinafter referred to as a target article) captured in an image obtained by a camera are known. However, when there is another article similar to the target article, it may not be possible to determine which of a plurality of candidates the target article is by image recognition.

[0003] Therefore, a plurality of candidates are presented to an operator, and one candidate specified by the operator's operation of an input device is determined as a recognition result. For this reason, the operator has to operate an input device to specify one of the plurality of candidates. Under such circumstances, it has been desired to reduce the labor of the operator required to specify one of the plurality of candidates.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the present invention is to provide an article recognition device and an information processing program that can reduce the labor of the operator required to specify one of a plurality of candidates.

Means for Solving the Problems

[0006] The object recognition device of the embodiment comprises an imaging device, measuring means, recognition means, selection means, display means, setting means, and determination means. The measuring means measures the direction and amount of movement of the imaging device. The recognition means recognizes an object captured in an image based on the image obtained by the imaging device. The selection means selects a candidate for the object recognized by the recognition means. The display means causes the display device to display a screen that includes, as a part, an area associated with the candidate selected by the selection means. The setting means sets a cursor position within the screen displayed by the display means based on the direction and amount of movement measured by the measuring means. The determination means determines that the candidate associated with the area is an object captured in the image, in response to the cursor position set by the setting means being within the area. [Brief explanation of the drawing]

[0007] [Figure 1] A block diagram showing the schematic configuration of a transaction processing system according to one embodiment, and the main circuit configuration of a user terminal included in the transaction processing system. [Figure 2] A flowchart illustrating the information processing performed by the processor based on a smartphone POS application. [Figure 3] A flowchart illustrating the information processing performed by the processor based on a smartphone POS application. [Figure 4] A diagram showing an example of a registration screen. [Figure 5] A diagram illustrating an example of a selection screen. [Figure 6] A diagram illustrating an example of a selection screen. [Figure 7] A diagram illustrating an example of a selection screen. [Modes for carrying out the invention]

[0008] An example of an embodiment will be described below with reference to the drawings. Figure 1 is a block diagram showing the schematic configuration of the transaction processing system 100 according to this embodiment, and the main circuit configuration of the user terminal 1 included in the transaction processing system 100. The transaction processing system 100 is configured to enable communication between the user terminal 1 and the transaction processing device 2 via a communication network 200. While any number of user terminals 1 and transaction processing devices 2 may be included in the transaction processing system 100, only one of each is shown in Figure 1.

[0009] User terminal 1 is an information processing terminal owned by the customer. Typically, user terminal 1 is owned by the customer and brought into the store for use by the customer. User terminal 1 is a terminal device that accepts customer operations for transaction processing on transaction processing device 2. Thus, the operator of user terminal 1 is primarily the customer. The transaction processing device 2 uses the user terminal 1 as a user interface and processes information to handle the buying and selling transaction of goods between the customer and the store, in accordance with the operations performed by the customer at the store. The transaction processing device 2 may be implemented as, for example, a cloud server and handle transactions at multiple stores. Alternatively, the transaction processing device 2 may be implemented as, for example, a local server and handle transactions at only one store.

[0010] The communication network 200 can use the internet, VPN (virtual private network), LAN, public communication network, mobile communication network, etc., individually or in appropriate combinations. As an example, the communication network 200 can be used in combination with the internet and a mobile communication network.

[0011] User terminal 1 includes a processor 11, main memory 12, auxiliary storage unit 13, touch panel 14, camera 15, motion sensor 16, mobile communication unit 17, and transmission line 18, etc. The processor 11 and the main memory 12, auxiliary storage unit 13, touch panel 14, camera 15, motion sensor 16, and mobile communication unit 17 are able to communicate with each other via the transmission line 18. The processor 11, main memory 12, and auxiliary storage unit 13 are connected by the transmission line 18, thereby forming a computer for controlling user terminal 1.

[0012] The processor 11 corresponds to the central part of the computer described above. The processor 11 performs information processing to control each part in order to realize various functions as a user terminal 1, in accordance with information processing programs such as the operating system and application programs.

[0013] Main memory 12 corresponds to the main memory portion of the computer described above. Main memory 12 includes a read-only memory area and a rewritable memory area. In the read-only memory area, main memory 12 stores a portion of the information processing program described above. Main memory 12 may also store data necessary for the processor 11 to perform processing to control each part in the read-only memory area or the rewritable memory area. Main memory 12 uses the rewritable memory area as a work area for the processor 11.

[0014] The auxiliary storage unit 13 corresponds to the auxiliary storage portion of the computer described above. The auxiliary storage unit 13 can utilize, for example, an EEPROM (electric erasable programmable read-only memory), an HDD (hard disk drive), an SSD (solid state drive), or various other well-known storage devices. The auxiliary storage unit 13 stores data used by the processor 11 in performing various processes and data generated by the processing performed by the processor 11. The auxiliary storage unit 13 may also store the information processing program described above. In this embodiment, the auxiliary storage unit 13 stores the smartphone POS application APA, which is one of the information processing programs. The smartphone POS application APA is an application program that describes the information processing procedures described later for enabling an information communication terminal such as a smartphone to function as a user terminal 1 of the transaction processing system 100.

[0015] The touch panel 14 displays an arbitrary screen and inputs operations by touching the screen. That is, the touch panel 14 functions as a display device and an input device of the user terminal 1. The touch panel 14 has a function of detecting the touch position on the screen in order to input operations by touching the screen, and is an example of a detection means. The camera 15 includes an optical system and an image sensor, and generates image data representing an image within the field of view formed by the optical system by the image sensor. The camera 15 is fixed to, for example, the main housing of the user terminal 1 and moves in the same manner as the user terminal 1 as the user terminal 1 moves. The camera 15 is an example of an imaging device.

[0016] The motion sensor 16 detects the movement of the user terminal 1. The motion sensor 16 physically detects the movement of the user terminal 1. As the motion sensor 16, for example, a well-known sensor device such as an acceleration sensor can be used. The mobile communication unit 17 is an interface for data communication via the communication network 200. As the mobile communication unit 17, for example, a well-known communication device for performing data communication via a mobile communication network can be used. The transmission path 18 includes an address bus, a data bus, control signal lines, etc., and transmits data and control signals exchanged between the connected components.

[0017] Next, the operation of the transaction processing system 100 configured as described above will be described. Note that the contents of the various processes described below are examples, and changes in the order of some processes, omission of some processes, or addition of other processes are possible as appropriate. For example, in the following description, in order to explain the characteristic operations of the present embodiment in an easy-to-understand manner, the description of some processes is omitted. For example, when some error occurs, a process for dealing with the error may be performed, but the description of such a process is omitted. In the following description, the service provided to the customer by the transaction processing system 100 is referred to as the "smartphone POS service".

[0018] In order for a customer to use the smartphone POS service, the customer registers as a member with the provider of the smartphone POS service. Through this member registration, an identifier (hereinafter referred to as a member code) for identifying the customer is determined, and the member database managed by the transaction processing device 2 is updated so as to associate various types of member information necessary for the customer to use the smartphone POS service with the above member code. In addition, in order for a customer to use the smartphone POS service, the customer installs the smartphone POS app APA on an information communication terminal such as a smartphone owned by the customer and makes it available as the user terminal 1. Then, the customer enters a store targeted for the smartphone POS service with the user terminal 1 in a state where information processing based on the smartphone POS app APA is activated.

[0019] Now, in the user terminal 1, the processor 11 executes information processing based on the smartphone POS app APA. FIG. 2 and FIG. 3 are flowcharts of information processing based on the smartphone POS app APA by the processor 11.

[0020] As ACT1 in FIG. 2, the processor 11 waits for a check-in to be instructed. Then, when a predetermined event occurs, such as a two-dimensional code for check-in prepared in the store being photographed by the camera 15, the processor 11 determines that the check-in has been instructed as YES and proceeds to ACT2. As ACT2, the processor 11 executes a check-in process for starting a transaction related to the purchase of goods in the store where the customer has entered. In this check-in process, the processor 11 starts communication via the communication network 200 with the transaction processing device 2, requests the execution of the transaction processing, and exchanges various types of information for the transaction processing. When the transaction processing device 2 is requested to start the transaction processing from the user terminal 1 as described above, the transaction processing device 2 starts the transaction processing. When the transaction processing device 2 is already executing a transaction processing for another user terminal 1, it starts a new transaction processing in parallel with that transaction processing.

[0021] As ACT3, the processor 11 displays the registration screen on the touch panel 14. Figure 4 shows an example of a registration screen SCA. The registration screen SCA consists of an image IMA and buttons BUA, BUB, BUC, and BUD. The image IMA shows the details of a registered transaction. In the example in Figure 4, the image IMA shows that the transaction list contains a total of 6 items worth 3,439 yen: one each of the product names "AAAAA," "CCCCC," "DDDDD," and "EEEEEE," and two of the product name "BBBBB." The image IMA also shows, for example, that the unit price of the product name "AAAAA" is 380 yen. Button BUA is a soft key that prompts the user to proceed to registering a product by scanning a barcode (hereinafter referred to as barcode registration). Button BUB is a soft key that prompts the user to proceed to registering a product by pressing a button (hereinafter referred to as button registration). Button BUC is a soft key that prompts the user to proceed to registering a product by image recognition (hereinafter referred to as image recognition registration). Button BUD is a soft key that prompts the user to proceed to checkout. Note that the details represented by the image IMA change sequentially depending on the status of the transaction list. In ACT3, the transaction list does not yet contain any items, so the image IMA displayed on the registration screen SCA here does not represent information about each item, but rather an image that shows a total of 0 items and 0 yen.

[0022] In ACT4, processor 11 checks whether barcode registration has been specified. If processor 11 cannot confirm the relevant event, it determines NO and proceeds to ACT5. In ACT5, processor 11 checks whether button registration has been specified. If processor 11 cannot confirm the relevant event, it determines NO and proceeds to ACT6. In ACT6, processor 11 checks whether image recognition registration has been specified. If processor 11 cannot confirm the relevant event, it determines NO and proceeds to ACT7. As ACT7, processor 11 checks whether or not a change to a registered transaction has been instructed. If processor 11 cannot confirm the event, it determines NO and proceeds to ACT8. As ACT8, processor 11 checks whether an accounting instruction has been issued. If processor 11 cannot confirm the relevant event, it determines NO and returns to ACT4. Thus, the processor 11 awaits either a barcode registration, button registration, or image recognition registration being specified as ACT4 to ACT8, or a change instruction or accounting instruction being given.

[0023] If a customer wishes to register a barcode, they specify barcode registration by performing a predetermined operation, such as tapping button BUA. Upon receiving such a specification, for example via touch panel 14, processor 11 determines YES in ACT4 and proceeds to ACT9. As ACT9, processor 11 executes the process for barcode registration. For example, processor 11 activates camera 15 and waits for the barcode to appear in the image captured by camera 15. The customer positions the barcode of the product they wish to register within the field of view of camera 15. Once the product's barcode has been captured by camera 15, processor 11 requests transaction processing device 2 to add the product represented by that barcode as the subject of the transaction. Transaction processing device 2 updates the transaction list in response to the request from user terminal 1. Transaction processing device 2 then instructs user terminal 1 to update the registration screen SCA to reflect the updated transaction list. As ACT10, processor 11 updates the registration screen SCA in response to the above instructions from transaction processing unit 2. After this, processor 11 returns to the waiting state for ACT4 to ACT8.

[0024] If a customer wishes to register a button, they specify the button registration by performing a predetermined operation, such as tapping button BUB. Upon receiving such a specification, for example via the touch panel 14, the processor 11 determines YES in ACT5 and proceeds to ACT11. As ACT11, processor 11 executes the process for button registration. For example, processor 11 displays a screen on touch panel 14 with buttons associated with products. The customer specifies a product by performing a predetermined operation, such as tapping the button associated with the product they wish to register. When processor 11 receives the operation to specify a product, for example via touch panel 14, it requests transaction processing unit 2 to add the product to the transaction. Transaction processing unit 2 updates the transaction list in response to the request from user terminal 1. Transaction processing unit 2 then instructs user terminal 1 to update the registration screen SCA to reflect the updated transaction list. After this, processor 11 executes ACT10 in the same manner as described above, and then returns to the waiting state for ACT4 to ACT8.

[0025] If a customer wishes to register using image recognition, they specify the registration by performing a predetermined operation, such as tapping button BUC. Upon receiving such a specification, for example via touch panel 14, processor 11 determines YES in ACT6 and proceeds to ACT12 in Figure 3. As ACT12, processor 11 acquires an image to be processed for image recognition. For example, processor 11 starts video recording or repeated still image capture by camera 15 and waits for the product to be captured by camera 15. The customer positions the product to be registered within the field of view of camera 15. Once the product's barcode has been captured by camera 15, processor 11 acquires the image generated by camera 15.

[0026] As ACT13, processor 11 calculates the similarity between the product shown in the acquired image and multiple other products. For example, processor 11 identifies the region in the acquired image where the product is shown. It then analyzes the image within that region and extracts feature quantities for each of several predetermined items. Processor 11 then calculates the similarity between the extracted feature quantities and the various products (hereinafter referred to as "target products") predetermined for image recognition by matching them. Products with a higher similarity calculated here are more likely to be the products shown in the image. Therefore, processor 11 recognizes the products shown in the image as products with a high similarity calculated here. Thus, by processor 11 executing information processing based on the smartphone POS application APA, the computer with processor 11 as its central component functions as a recognition means.

[0027] As ACT14, processor 11 selects candidate products from the target products for the product to be registered. For example, processor 11 selects a predetermined number of target products with high similarity as candidate products. In this embodiment, processor 11 selects target products for which the top three similarity scores have been calculated as candidate products. Processor 11 may choose not to select target products with a similarity score lower than the standard value as candidate products. In other words, if there are multiple products for which a high similarity score has been calculated, i.e., if multiple products are recognized, processor 11 selects a predetermined number of these multiple products as candidate products. Thus, by having processor 11 execute information processing based on the smartphone POS application APA, the computer with processor 11 as its central component functions as a control means. Furthermore, ACT12 to ACT14 can utilize well-known processes, such as those already performed in existing equipment.

[0028] As ACT15, the processor 11 displays a selection screen on the touch panel 14. The selection screen is a user interface screen that allows the customer to select one of the candidate products. Figure 5 shows an example of a selection screen (SCB). The selection screen SCB includes areas ARA, ARB, ARC, ARD, and ARE. Area ARA is the area located in the center of the selection screen SCB. In the example in Figure 5, area ARA is rectangular, but its shape is arbitrary and can be determined as appropriate by the creator of the smartphone POS application APA. Areas ARB, ARC, ARD, and ARE are rectangular regions positioned opposite each of the four sides of area ARA.

[0029] Processor 11 displays the video obtained by camera 15 in area ARA. Processor 11 makes areas ARB, ARC, ARD, and ARE visible by using different display colors from the surrounding areas. Processor 11 may also make areas ARB, ARC, ARD, and ARE visible by other arbitrary methods such as drawing borders. Processor 11 associates three candidate products with areas ARB, ARC, and ARD, and displays the string names of the associated candidate products inside areas ARB, ARC, and ARD. In this embodiment, Processor 11 associates the candidate product with the highest similarity with area ARB. In this embodiment, Processor 11 associates the candidate product with the second highest similarity with area ARC. In this embodiment, Processor 11 associates the candidate product with the third highest similarity with area ARD. In other words, Figure 5 is an example where "Jonagold," "Fuji," and "Tomato" are selected as candidate products in order of decreasing similarity. Processor 11 displays the string "Cancel" in area ARD. Areas ARB, ARC, ARD, and ARE are also GUI (graphical user interface) buttons. Therefore, in the following, areas ARB, ARC, ARD, and ARE will be referred to as button areas when there is no need to distinguish between them individually. Each button area only needs to be adjacent to area ARA, and its shape, number, and arrangement are arbitrary and may be determined as appropriate by the creator of the smartphone POS application APA, etc. Thus, by having the processor 11 perform information processing based on the smartphone POS application APA, the computer with the processor 11 as its central component functions as a display means that displays the selection screen SCB on the touch panel 14, which acts as a display device.

[0030] As ACT16, processor 11 measures the direction and amount of movement of user terminal 1. For example, processor 11 calculates the direction and amount of movement of user terminal 1 using optical flow based on the time change of video captured by camera 15. For optical flow, for example, the process described in Patent Document 1 can be used. The direction and amount of movement determined by optical flow are actually the direction and amount of movement of camera 15. Thus, by processor 11 executing information processing based on the smartphone POS application APA, the computer with processor 11 as its central component functions as a measurement means. Note that since camera 15 moves in the same way as user terminal 1 along with the movement of user terminal 1, the direction and amount of movement of camera 15 are also the direction and amount of movement of user terminal 1.

[0031] As ACT17, processor 11 sets the cursor position. The cursor position is any position within the two-dimensional coordinate system defined within the selection screen SCB. Processor 11 uses a predetermined reference position within area ARA as the initial cursor position. The reference position is typically the center of area ARA, but can be arbitrarily determined by the creator of the smartphone POS application APA or the user of user terminal 1. Then, when processor 11 has progressed from ACT15 to ACT16 and then to ACT17, it sets a new cursor position at a position separated from the reference position by a distance corresponding to the amount of movement determined in ACT16, in the direction of movement determined in ACT16. Thus, by processor 11 executing information processing based on the smartphone POS application APA, the computer with processor 11 as its central component functions as a setting means.

[0032] As ACT18, processor 11 updates the selection screen SCB. For example, processor 11 updates the selection screen SCB to display an arrow with the reference position as its base and the newly set cursor position as its tip. Figure 6 shows an example of the updated selection screen SCB. The selection screen SCB in Figure 6 is an example of what happens when user terminal 1, which was displaying the selection screen SCB shown in Figure 5 on the touch panel 14, moves toward area ARC. For this reason, the selection screen SCB shown in Figure 6 has an arrow ASA added to it that extends from the center of area ARA toward area ARC. The selection screen (SCB) only needs to allow the customer to recognize the cursor position; for example, it may be possible to move the display position of a point or a fixed-shape arrow according to the cursor position.

[0033] As ACT19, processor 11 checks whether any of the button areas have been tapped. If the touch panel 14 does not detect that any of the areas ARB, ARC, ARD, or ARE have been tapped, processor 11 determines NO and proceeds to ACT20. As ACT20, processor 11 checks whether the cursor position newly set in ACT17 is within the button area. If the cursor position set in ACT17 is not in any of the areas ARB, ARC, ARD, or ARE, processor 11 determines NO and returns to ACT16. Processor 11 then repeats ACT16 and subsequent steps as described above. However, when processor 11 executes ACT17 for the second time or later, it sets a new cursor position in the direction of movement newly determined in ACT16, at a distance corresponding to the amount of movement newly determined in ACT16 from the cursor position set in the previous ACT17. When processor 11 executes ACT18 for the second time or later, the base position of the arrow remains at the reference position, and only the tip position is set to the newly set cursor position.

[0034] Figure 7 shows an example of the updated selection screen SCB. The selection screen SCB in Figure 7 is an example of what happens when user terminal 1, which was displaying the selection screen SCB shown in Figure 6 on the touch panel 14, moves further toward area ARC. Therefore, in the selection screen SCB shown in Figure 7, the arrow ASA in Figure 6 is replaced with a longer arrow ASB.

[0035] When the selection screen SCB is displayed, the customer determines which of the candidate products represented in areas ARB, ARC, and ARD corresponds to the product they intend to register, and moves user terminal 1 in the direction of the corresponding area. For example, if the selection screen SCB shown in Figure 5 is displayed and the product to be registered is "Fuji," the customer moves user terminal 1 in the direction of area ARC, i.e., to the right. The customer then adjusts the movement of user terminal 1 so that the tip of the arrow represented on the selection screen SCB reaches the button area of ​​the determined candidate product. If the product to be registered is not one of the candidate products represented in areas ARB, ARC, and ARD, or if the customer decides to cancel the registration of the recognized product for any reason, the customer moves user terminal in the direction of area ARE.

[0036] When the selection screen SCB is in the state shown in Figure 7, the cursor position newly set in ACT17 is within area ARC. If the cursor position newly set in ACT17 is in any of areas ARB, ARC, ARD, or ARE, it is considered to be within the button area, and ACT20 determines YES, proceeding to ACT21. As ACT21, processor 11 checks whether it is a cancel instruction or not. Then, if the cursor position newly set in ACT17 is not in area ARE, processor 11 determines NO and proceeds to ACT22.

[0037] As ACT22, processor 11 requests transaction processing device 2 to add candidate products associated with the button area containing the cursor position as items for transaction. At this point, processor 11 has determined that the candidate products associated with the button area containing the cursor position are products that are visible in the image. Thus, by processor 11 executing information processing based on the smartphone POS application APA, the computer with processor 11 as its central component functions as a decision-making mechanism. In addition, in ACT14, processor 11 may not select target products with a similarity score lower than the threshold value as candidate products, and if only one candidate product is selected, it may skip ACT15 to ACT21 and execute ACT22 as if only one candidate product has been selected.

[0038] The transaction processing unit 2 updates the transaction list in response to a request from the user terminal 1. The transaction processing unit 2 then instructs the user terminal 1 to update the registration screen SCA to reflect the updated transaction list. The processor 11 then proceeds to ACT10 in Figure 2, and in response to the instruction from the transaction processing unit 2, returns the display on the touch panel 14 to the updated registration screen SCA, and then returns to the waiting state of ACT4 to ACT8.

[0039] If the cursor position newly set in ACT17 is in area ARE, processor 11 determines YES in ACT21, returns to ACT12, and repeats the processing from ACT12 onward as described above. In other words, processor 11 discards the result of the current image recognition and performs image recognition again.

[0040] The customer may tap one of the areas ARB, ARC, ARD, or ARE instead of moving the user terminal 1. If the touch panel 14 detects that one of the areas ARB, ARC, ARD, or ARE has been tapped, the processor 11 determines YES in ACT 19 and proceeds to ACT 23.

[0041] As ACT23, processor 11 checks whether it is a cancel instruction or not. If the tapped area is not area ARE, processor 11 determines it is NO and proceeds to ACT24. As ACT24, processor 11 requests transaction processing unit 2 to add candidate products associated with the tapped button area as items for trading. Transaction processing unit 2 updates the trading list in response to the request from user terminal 1. Transaction processing unit 2 then instructs user terminal 1 to update the registration screen SCA to reflect the updated trading list. Processor 11 then proceeds to ACT10 in Figure 2, and in response to instructions from transaction processing unit 2, returns the display on touch panel 14 to the updated registration screen SCA, and then returns to the waiting state of ACT4 to ACT8.

[0042] If the tapped area is ARE, processor 11 determines YES in ACT23, returns to ACT12, and repeats the processing from ACT12 onward as described above. In other words, processor 11 discards the result of the current image recognition and performs image recognition again. As described above, customers can register the products they wish to trade by using barcode scanning, button operation, or image recognition as appropriate.

[0043] If a customer wishes to change the quantity of an item already being traded, they tap the area on the registration screen SCA that displays the quantity of the item and then select the new quantity. Once the predetermined operation to instruct such a change in transaction details has been performed, processor 11 determines YES in ACT7 in Figure 2 and proceeds to ACT25. As ACT25, processor 11 executes a change process to modify the transaction details in response to the change instruction. For example, processor 11 requests transaction processing device 2 to change the quantity or delete an item. In response to this request, transaction processing device 2 updates the transaction list. Transaction processing device 2 then instructs user terminal 1 to update the registration screen SCA to reflect the updated transaction list. After this, processor 11 executes ACT10 in the same manner as described above, and then returns to the waiting state for ACT4 to ACT8.

[0044] If the customer wishes to proceed to checkout, they will indicate this by performing a predetermined action, such as tapping the BUD button on the registration screen SCA. In response, processor 11 determines YES in ACT8 and proceeds to ACT26. As ACT26, processor 11 executes accounting processing. For example, processor 11 requests accounting from transaction processing unit 2 and then, in cooperation with transaction processing unit 2, executes predetermined processing for accounting the transaction.

[0045] As ACT27, processor 11, in cooperation with transaction processing device 2, executes a predetermined checkout process to terminate the use of the smartphone POS service. Furthermore, the accounting and checkout processes may be the same as those used in other existing smartphone POS services. Then, once the checkout process is complete, processor 11 returns to the waiting state for ACT1.

[0046] As described above, the user terminal 1 changes the cursor position set in the selection screen SCB based on the movement direction and amount of the camera 15, and when the cursor position falls inside one of the areas ARB, ARC, or ARD associated with the candidate product, it determines that the product associated with that area is the recognition result. Thus, the customer only needs to move the user terminal 1 so that the cursor position of the camera 15, which is pointed at the product in question to photograph the product to be registered, moves to the area associated with the candidate product they wish to specify, and this operation can be completed with minimal effort.

[0047] Furthermore, since the user terminal 1 determines the direction and amount of movement of the camera 15 based on the optical flow obtained from the camera 15, there is no need to provide a sensor for determining the direction and amount of movement of the camera 15.

[0048] Furthermore, user terminal 1 cancels recognition when the cursor position is inside area ARE. Thus, if the customer decides not to register one of the candidate products, they only need to move user terminal 1 so that the camera 15, which is pointed at the product to be registered, moves to area ARE, thus minimizing the effort required for this operation.

[0049] Furthermore, if the user terminal 1 taps inside any of the areas ARB, ARC, or ARD, it will also determine the product associated with that area as the recognition result. Thus, if the customer can easily tap the selection screen SCB, they can also specify candidate products by tapping.

[0050] This embodiment can be modified in various ways as follows: The direction and amount of movement of the user terminal 1 may be measured using the motion sensor 16.

[0051] The device may be implemented as a separate entity consisting of a first unit, which is movable by an operator and equipped with a camera 15, and another unit separate from the first unit. In this case, the direction and amount of movement of the first unit are measured. For example, a handheld scanner for a POS terminal may be equipped with a camera 15, and the direction and amount of movement of the handheld scanner may be measured.

[0052] It may be implemented as an item recognition device that recognizes goods for purposes other than the recognition of goods to be registered as the subject of a transaction.

[0053] It may be implemented as an item recognition device that recognizes items that are not products.

[0054] The device may be implemented as an item recognition device that outputs the item recognition results for use in any other device, such as a POS terminal.

[0055] The transaction list may be created and updated by the processor 11 on the user terminal 1 and stored in the auxiliary storage unit 13. Alternatively, a copy of the transaction list equivalent to that stored in the transaction processing unit 2 may be stored in the auxiliary storage unit 13 of the user terminal 1.

[0056] Each function realized by the processor 11 through information processing can also be partially or entirely realized by hardware that performs non-program-based information processing, such as logic circuits. Furthermore, each of the above functions can also be realized by combining the above-mentioned hardware, such as logic circuits, with software control.

[0057] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0058] 1...User terminal, 2...Transaction processing unit, 11...Processor, 12...Main memory, 13...Auxiliary storage unit, 14...Touch panel, 15...Camera, 16...Motion sensor, 17...Mobile communication unit, 18...Transmission line, 100...Transaction processing system, 200...Communication network.

Claims

1. Imaging device and A measuring means for measuring the direction and amount of movement of the imaging device, A recognition means for recognizing an object captured in an image based on an image obtained by the aforementioned imaging device, A selection means for selecting candidates for articles recognized by the recognition means, A display means that displays a screen on a display device that includes, as part of the area associated with the candidate selected by the selection means, Within the screen displayed by the display means, there is a setting means for setting the cursor position based on the direction and amount of movement measured by the measuring means, A determination means that determines that, in response to the cursor position set by the setting means being within the area, the candidate associated with the area is an item captured in the image, An item recognition device equipped with the following:

2. The measurement means measures the direction and amount of movement of the imaging device by optical flow to a plurality of images obtained by the imaging device at different timings. The article recognition device according to claim 1.

3. The measurement means includes a sensor that physically detects the movement of the imaging device, and measures the direction and amount of movement of the imaging device based on the detection result of the sensor. The article recognition device according to claim 1.

4. A detection means for detecting the touch position made by the operator on the screen displayed by the aforementioned display device, Furthermore, it is equipped with, The determination means determines that, in response to the detection means detecting multiple locations within the areas as touch locations, the candidate associated with the area is an item captured in the image. The article recognition device according to any one of claims 1 to 3.

5. The display means causes the display device to display a screen that includes a cancellation area in addition to the area, The determination means, in response to the cursor position being within the cancellation area, refrains from determining the item based on the recognition result by the recognition means. The article recognition device according to any one of claims 1 to 4.

6. A computer installed in an object recognition device equipped with an imaging device, A measuring means for measuring the direction and amount of movement of the imaging device, A recognition means for recognizing an object captured in an image based on an image obtained by the aforementioned imaging device, A selection means for selecting candidates for articles recognized by the recognition means, A display means that displays a screen on a display device that includes, as part of the area associated with the candidate selected by the selection means, Within the screen displayed by the display means, there is a setting means for setting the cursor position based on the direction and amount of movement measured by the measuring means, A determination means that determines that, in response to the cursor position set by the setting means being within the area, the candidate associated with the area is an item captured in the image, An information processing program that enables a function to work.

Images